Mapping bass frequencies to visual motion

Chapter 2 of 7. Prerequisite: Remotion audio fundamentals, where you loaded a track with useAudioData and pulled a frequency array per frame. Now you turn one slice of that array into something on screen that visibly moves with the kick.

Every tutorial gets you to "the left of the array is bass" and stops. That sentence is true and useless. It does not tell you which indices, whether to use the smoothed curve or the raw one, how hard to gain the result, or what to attach it to so the picture reads as responding to the low end rather than jittering. That decision chain is the whole job, and this chapter is the chain the captain-random-visualiser build actually walked to get a usable bass signal.

Isolating the bass band in Remotion

Start from what visualizeAudio hands you. Called with numberOfSamples: 32, it returns a 32-element array for the current frame, and every value sits between 0 and 1. The Remotion docs are explicit: "An array of values describing the amplitude of each frequency range. Each value is between 0 and 1." Index 0 is the lowest frequency range, index 31 the highest. The bass lives in the first few bins.

The build ran visualizeAudio twice on every frame off the same decoded data:

// TrackVisualiser.tsx:53-54
const smooth = visualizeAudio({ fps, frame, audioData, numberOfSamples: 32, smoothing: true });
const raw    = visualizeAudio({ fps, frame, audioData, numberOfSamples: 32, smoothing: false });

Both calls read the same audioData object that useAudioData(staticFile(audio)) decoded once and holds in memory. The hook keeps that object stable across frames. The decode already happened, so the two per-frame samples are cheap. smoothing: true applies an energy-decay envelope, so the value glides. smoothing: false gives you the raw transient with all its spikes intact. You want both, because bass has two components you will map differently.

To pull a band out of the 32 bins, average a contiguous slice:

// TrackVisualiser.tsx:35-40
const avg = (a: number[], lo: number, hi: number) => {
  let s = 0;
  for (let i = lo; i < hi; i++) s += a[i] ?? 0;
  return s / (hi - lo);
};
const clamp = (v: number, min = 0, max = 1) => Math.max(min, Math.min(max, v));

That is the entire band-isolation primitive. No FFT of your own, no windowing, no library. avg(array, 0, 3) is "the mean of bins 0, 1, 2", which is the sub-bass region of a 32-bin spectrum.

The kick is the raw transient, not the smoothed envelope

Here is the distinction the "left = bass" advice never makes. Sub-bass and the kick drum live in nearly the same bins, but they are different signals and want different smoothing.

// TrackVisualiser.tsx:55-58
const sub  = clamp(avg(smooth, 0, 3) * 3.2);  // bins 0-2  sub-bass envelope -> the breath
const kick = clamp(avg(raw,    0, 2) * 3.6);  // bins 0-1  raw transient      -> the punch

sub comes off the smoothed array. It is the slow, continuous weight of the low end, the thing that swells under a phrase and sags between them. Drive a bloom or a slow zoom off it and the picture breathes.

kick comes off the raw array and only the two lowest bins. It keeps the sharp attack that smoothing would sand away. Drive a fast, short response off it and each hit reads as a punch. If you had used the smoothed curve for the kick, the transient would arrive rounded and late, and the picture would feel like it was reacting a beat behind the music. Using the raw array for the kick is the fix for the single most common "why does my visualiser feel sluggish" complaint.

Why there is no log-scaling in this build

Human pitch perception is logarithmic. An octave is a doubling of frequency, so the ear treats the gap from 50 Hz to 100 Hz as the same musical distance as 5 kHz to 10 kHz, even though the second gap is a hundred times wider in raw Hz. A common instinct, and something plenty of visualiser tutorials reach for, is to remap the linear FFT bins onto a logarithmic axis before you read a band off them, so the low end gets the perceptual weight your ear gives it.

This build did not do that. Here is why. When you only care about the bottom of the spectrum, the linear layout is already on your side. The lowest musical energy is packed into the first handful of bins, so avg(raw, 0, 2) captures the kick without any remap. Log-scaling earns its cost when you are rendering the whole spectrum as bars and want the low octaves to occupy a fair share of the width. For pulling one bass band out to drive one visual response, it is machinery you do not need. The band isolation here is fixed integer index slices with a linear gain, nothing more.

That gain is the constant sitting after each average. * 3.2 on the sub, * 3.6 on the kick. Averaged bass bins rarely climb near 1.0 on their own, so without a multiplier the mapped response barely twitches. The gains were hand-tuned by eye against the real track until the motion read at the right strength, then clamp caps each band at 1.0 so a loud passage cannot blow the value past the range the downstream maths expects.

Mapping the band to a visual response

You now have two normalised 0-to-1 numbers. Attaching them to the picture is deliberately plain arithmetic, not a spring or a tween:

// TrackVisualiser.tsx:62-68
const kb = interpolate(frame, [0, durationInFrames], [1.06, 1.16], { extrapolateRight: 'clamp' });
const scale      = kb + sub * 0.02 + kick * 0.015;
const brightness = 1 + sub * 0.22 + kick * 0.12;

Read scale as two stacked contributions. kb is a slow Ken Burns push, a continuous zoom from 1.06 to 1.16 across the whole clip, driven by frame alone through interpolate, whose job the docs describe as mapping "a range of values to another using a concise syntax." That is the time-driven motion, and it has nothing to do with audio. On top of it, sub * 0.02 and kick * 0.015 add a small audio-reactive nudge. The scene is always drifting in, and the bass gives it a subtle extra push on the low-end weight and a tighter one on each hit.

Note how small those coefficients are. A full-scale kick adds 0.015 to the zoom. That restraint is the point. The reactive term is a seasoning on a motion that already exists, not the whole motion. brightness follows the identical pattern: a base of 1, plus a larger sub term for the breathing lift and a smaller kick term for the flash. Sub gets the bigger coefficient on both because it is the sustained layer you want to feel underneath, and the kick gets the smaller, sharper one because it is the accent.

The mapping decision underneath all of this is which visual property each band steers. Sub-bass steers the slow properties, brightness and scale drift. The kick steers the fast accent on those same properties. You do not need a separate effect per band. You need one signal split into a slow envelope and a raw transient, then each pointed at the timescale it suits.

Give the mapped curve enough frames to read

The coefficient is only half of a good mapping. The other half is whether the response has room to play out.

An earlier sibling build, the tasogare visualiser, ran on the same two tools: Remotion and visualizeAudio, with the bass as its nervous system. It drove a perpendicular wobble off an energy-decay envelope on the low end. That wobble survives in this build, re-aimed as water ripple on the sea's reflection. The mechanic is the same one smoothing: true gives you here: an envelope that rises on energy and decays back over frames.

An energy-decay envelope only reads if the decay is slow enough to see. Map a punchy coefficient onto a response that resolves in three frames and it reads as a flicker, not a hit. This is why the kick term is the small, sharp one: it rides on top of a base that is already moving. The Ken Burns push runs the whole clip, so the transient nudge has a slow arc underneath it to register against. The arc needs time to read.

Where this leaves you

You now have the full chain the "left = bass" line skips. Take the 32-bin array from visualizeAudio. Run it twice, smoothed and raw. Average the low bins for a band, using the smoothed slice for the sustained sub and the raw slice for the transient kick. Gain each with a hand-tuned constant and clamp it to 0-to-1. Then add small reactive terms on top of a time-driven base rather than driving the property from audio alone. Skip log-scaling unless you are drawing the whole spectrum. And give the response enough frames to be seen.

The next chapter takes the same two signals into a genuinely different question: when a value should ease with spring physics and when a plain interpolate is the right tool, and why in this build the kick drives neither directly.