Chapter 1 of 7. Before a single pixel reacts to a beat you have to get the audio into the composition and read the frequency numbers correctly. This chapter loads the file, picks the right hook, and pulls a per-frame frequency array out of it.
A music visualiser is a function from sound to picture. Every effect in the rest of this course reads from the same thing: a small array of numbers describing the audio at the current frame. The scale push, the neon bloom, the light rays all pull from that one array. If it is wrong, nothing downstream can fix it, so this chapter is short and it is load-bearing.
The build this course is drawn from runs on Remotion 4.0.485, with @remotion/media-utils at the same version. The package overview describes it as "a package providing utility functions for getting information about video and audio, and for visualizing audio." Two functions from it do all the work in this chapter: useAudioData and visualizeAudio.
Load the whole file into memory with useAudioData()
The first decision is how the audio gets into the composition. In the real code it is one line, in all four audio-reactive compositions:
// TrackVisualiser.tsx:6
import { useAudioData, visualizeAudio } from '@remotion/media-utils';
// TrackVisualiser.tsx:48
const audioData = useAudioData(staticFile(audio));
useAudioData fetches the file, decodes it, and holds the samples in React state. The hook's docs spell out what it does for you: it "keeps the audio data in a state, wraps the function in a delayRender() / continueRender() pattern, and handles the case where the component gets unmounted while the fetching is in progress." That last part matters more than it looks. During render Remotion needs the frame to block until the audio is decoded, otherwise the first frames paint against no data. The hook does that blocking for you. You do not call delayRender() by hand.
There is one catch that bites every beginner, and it is visible in the guard the real code puts around the value:
// every band read is inside this guard
if (audioData) {
const smooth = visualizeAudio({ fps, frame, audioData, numberOfSamples: 32, smoothing: true });
// ...
}
useAudioData returns null on the first render while the fetch is still in flight. Read a frequency band off null and the composition throws. Every one of the four compositions guards the value with if (audioData) before touching it.
Two ways to load audio: useAudioData vs useWindowedAudioData
Remotion ships two hooks for getting audio data, and the difference is memory, not format. useAudioData decodes the entire file into memory at once. For a three-minute track that is fine. For a two-hour podcast it is not, because you are holding every decoded sample in RAM for a render that only ever looks at one frame at a time. For that case Remotion offers useWindowedAudioData. The windowed hook's docs put the difference plainly: "Unlike useAudioData(), which keeps all of the audio data in memory, this function makes HTTP Range requests to only load the audio data around the current frame."
A Range request is a partial fetch: the hook asks the server for only the bytes near the current frame rather than the whole file, so memory stays bounded no matter how long the track is. That is the entire trade the windowed hook makes. It is not a format restriction. Older Remotion versions did restrict it to WAV, but that limitation was lifted in 4.0.383, and on this build's 4.0.485 the windowed hook loads mp3 and the other formats its media layer supports, same as useAudioData. If you read an old thread claiming "useWindowedAudioData is WAV-only," it is describing a version that predates the fix.
Why useAudioData, not useWindowedAudioData?
The call
The visualiser behind this course loads the whole track with useAudioData() and reads it everywhere. One fetch, one array, no per-frame windowing.
Rejected
What it costs
The full decoded track sits in memory. Fine at music-video length; the rendering chapter covers where that ceiling actually bites.
Revisit when
A long-form or Lambda-chunked render is the trigger to move to the windowed hook, and to accept its WAV constraint.
Read the frequency array with visualizeAudio()
Loading the file gives you decoded samples. It does not give you "how much bass is there at frame 240." That is what visualizeAudio is for. It takes the decoded audioData plus the current frame and returns a frequency spectrum for that instant. The docs describe the return value as "an array of values describing the amplitude of each frequency range. Each value is between 0 and 1."
That 0-to-1 normalisation is a convenience Remotion adds on top of the raw web platform. Underneath, an AnalyserNode's byte frequency data is "composed of integers on a scale from 0 to 255." Remotion hands you the same shape rescaled to 0-to-1, so you never divide by 255 yourself. Low index is low frequency, high index is high frequency. Multiply by a gain, clamp, done.
The real code calls it twice per frame, with 32 bins, once smoothed and once raw:
// TrackVisualiser.tsx:53-54
const smooth = visualizeAudio({ fps, frame, audioData, numberOfSamples: 32, smoothing: true });
const raw = visualizeAudio({ fps, frame, audioData, numberOfSamples: 32, smoothing: false });
numberOfSamples: 32 gives a 32-bin array. smoothing: true applies an energy-decay envelope, so the value bleeds down gradually after a transient. That is the version you want for anything that should breathe: a bloom, a slow glow. smoothing: false is the raw transient, sharp on the hit and gone. That is the version you want for a punch or a flicker. Same audio, same frame, two temperaments. Chapter 3 is entirely about when to reach for each, so for now just note that you can have both from one call each.
Turning the array into bands you can name
A 32-number array is not yet "bass." You have to decide which indices are the low end and average them. The real code has a three-line helper and four named bands:
// TrackVisualiser.tsx:55-58
sub = clamp(avg(smooth, 0, 3) * 3.2); // bins 0-2 sub-bass envelope
kick = clamp(avg(raw, 0, 2) * 3.6); // bins 0-1 raw transient
hats = clamp(avg(smooth, 22, 32) * 4.5); // bins 22-31 highs
hatsRaw = clamp(avg(raw, 22, 32) * 5.0); // bins 22-31 raw highs
avg(array, lo, hi) averages a contiguous slice of the 32 bins. clamp bounds the result to 0-to-1. Two things are worth naming now, because beginners expect otherwise. There is no log-frequency mapping here. The bins are used as flat integer index slices, because visualizeAudio already returns them the way Remotion computes them. And the gain constants (3.2, 3.6, 4.5, 5.0) are hand-tuned to the track, not derived from anything. You watch the render against the real audio and turn the knob until the band reads right. Chapter 2 does exactly that for the bass path.
Extract the signal once, read it everywhere
Here is the architecture lesson the fundamentals chapter exists to plant, and it comes from a real moment on this build.
The neon bloom's opacity is one line: clamp(0.28 + sub * 0.4 + hatsRaw * 0.6) at TrackVisualiser.tsx:106. The 0.28 is a resting floor that keeps the bloom faintly present even in silence. sub * 0.4 is the sub-bass breath underneath, a slow lift on the low end. hatsRaw * 0.6 is the raw-highs flicker, sharp on the shimmer. Neither of those two drivers, sub and hatsRaw, was extracted for the bloom. They already existed for other layers, and the bloom just borrowed them.
That is the pattern the whole build fell into. The sub envelope alone drives the Ken Burns scale nudge (scale = kb + sub*0.02 + kick*0.015), the brightness and contrast grade, part of the neon opacity above, the celestial-bloom glow, the light-ray intensity, and the vignette that opens on the low end. Nobody planned that up front. Each new effect reached for the signal that was already being extracted, and reusing it kept everything in rhythmic lock-step for free. The coupling is read-only and one-way. Every effect reads the envelope and none of them writes it, so a new surface can lean on sub without any risk of disturbing the surfaces already using it. The WebGL depth-parallax layer (ParallaxPhoto) was already running when the bloom went in, and it too reads the same bands rather than computing its own.
That is the shape to aim for from chapter 1. Compute the frequency array once per frame with visualizeAudio, reduce it to a few named, normalised values (sub, kick, hats), and let each effect read the value it needs without mutating it. When the signal lives in one place and callers only read it, a new effect is a few lines that borrow an existing number, and every effect stays in sync because they are all reading the same beat.
The trade-off is real. Because every effect reads one shared sub envelope, you cannot make the bloom respond to bass differently from the vignette without splitting the signal into two paths. On this build that capability is absent by choice, not by oversight. When you do need it, the fix is to run visualizeAudio with a different smoothing flag or average a different band, which is the same "extract once, read many" pattern applied twice.
What to carry into chapter 2
You now have the whole front of the pipeline. useAudioData(staticFile(audio)) loads the file and returns null until it is decoded, so you guard it. useWindowedAudioData is the windowed alternative for very long files, where it makes HTTP Range requests to bound memory rather than decoding the whole track; the difference from useAudioData is memory, not format. visualizeAudio({ numberOfSamples: 32 }) turns decoded audio into a 32-bin, 0-to-1 frequency array for the current frame, and calling it with smoothing: true and smoothing: false gives you a breathing signal and a punching one from the same source.
Chapter 2 takes the sub and kick bands from that array and drives real motion with them: a Ken Burns push that leans harder on the beat, and the gain-tuning method that decides how hard.
Wire your own track into a guarded band extractor
Load a track of your own with useAudioData(staticFile(...)) in a fresh composition, sample it with visualizeAudio at 32 bins, smoothed and raw, and reduce the array to three named bands: a smoothed low band, a raw low transient, and a smoothed high band. Render each band as an on-screen numeric readout or simple meter so you can watch the values move against playback, then hand-tune a gain per band until each reads across most of the 0 to 1 range.
Expected behaviour
- The composition survives the first frame of a clean render because every band read sits behind an if (audioData) guard with a zeroed fallback
- Three named bands derived from fixed index slices of the 32-bin array are visible on screen and move with the music
- The smoothed low band visibly bleeds down gradually after a hit while the raw low transient spikes and vanishes
- Each band has its own hand-tuned gain constant and a clamp so no value exceeds 1
PROVE IT Play 20 seconds in Remotion Studio and point at one moment where the raw transient spikes on a hit while the smoothed band is still bleeding down from the previous one.
What does useAudioData return on the very first render, while the file is still being fetched and decoded?
Which Remotion version lifted the WAV-only restriction on useWindowedAudioData?
Why does this build compute the frequency bands once per frame and let every effect read the same values, rather than each effect extracting its own?
Show answer
Extracting once and sharing keeps every effect in rhythmic lock-step for free, because they are all reading the same beat. The coupling is read-only and one-way, so a new effect is a few lines that borrow an existing number without any risk of disturbing the surfaces already using it.
↺ re-read: “Extract the signal once, read it everywhere”