Audio sync drift: WAV, MP3, and Lambda

Chapter 4 of 7. Prerequisite: Chapter 1, where you loaded a track with useAudioData() and sampled it per frame with visualizeAudio(). This chapter is about the failure that follows: the bloom fires a beat late, or a click appears in the exported audio that was never in the studio preview.

The visualiser looks locked in the Remotion Studio. You render it, watch the MP4, and the kick flash lands just after the kick. Not by much. Enough to feel wrong and not enough to point at. Nothing in your code changed between the preview and the render, so the instinct is to blame the analysis. The analysis is almost never the problem. The problem is that the audio your code analysed and the audio the render played back are not the same timeline.

There are three distinct ways that happens. Each has a different cause and a different fix, and they get confused because they all present as the same symptom: sound and picture out of step. This chapter names all three and gives the codec workaround for each. Two of them this build hit and mitigated. The third is documented Remotion behaviour on a render path this build deliberately did not take.

Why sync depends on the decoded timeline, not the file

Start with what visualizeAudio() actually consumes. It takes fps, the current frame, and a decoded audioData object, and it returns the frequency spectrum for that instant. The visualizeAudio() docs describe the output as an array of amplitudes, each value between 0 and 1. To produce it, the function has to convert a frame number into a position in the audio. That conversion is frame / fps seconds, mapped onto the sample grid of the decoded file.

That sample grid comes from useAudioData(). The useAudioData() docs are explicit that the hook keeps all of the audio in memory and wraps the fetch in a delayRender() / continueRender() pattern. The whole file is decoded once, up front, and every frame reads from that one in-memory copy. In this build the call is a single line:

// TrackVisualiser.tsx:48
const audioData = useAudioData(staticFile(audio));

audioData is null on the first render while the decode runs, which is why every comp guards it with if (audioData) {...}. Once it resolves, the sample grid is fixed for the entire render.

Here is the load-bearing part. The frame-to-time math is only correct if the decoded duration matches the real playback duration. It is the same idea the browser exposes through the MDN AnalyserNode: analysis reads a slice of a timeline, and it is only useful when that timeline is the one you hear. visualizeAudio() runs its own FFT over the decoded data rather than routing through AnalyserNode, but the constraint is identical. When the decoded grid and the playback clock agree, frame 750 at 25 fps analyses exactly the audio you hear at 30 seconds. When they disagree, the visual for frame 750 is computed from the wrong slice of sound, and the offset is baked into the file. Sync drift is a timeline mismatch. The three classes below are three different reasons the two timelines stop agreeing.

Class one: VBR MP3 drift, and the WAV fix

This is the one people search for. Remotion audio sync drift on an MP3 source, worst on longer tracks, and the fix is to stop using a variable-bitrate MP3.

A constant-bitrate file spends the same number of bytes on every unit of time, so byte offset maps linearly to time and a decoder can compute any position by arithmetic. A variable-bitrate MP3 spends bytes according to how complex the audio is, so a quiet passage is smaller on disk than a loud one of the same length. There is no linear relationship between byte offset and time. Players rely on a header table (the Xing or VBRI header) to estimate where a given timestamp lives, and the estimate is approximate. Different decoders round it differently. The header can also misreport total duration outright.

Feed that into useAudioData() and the decoded grid inherits the error. If the header says the track is 195.0 seconds when the audio is really 195.4, the sample grid is stretched to fit the wrong length, and every frame after the first samples slightly early. The error is not constant. It accumulates. Frame 100 is off by a few milliseconds, frame 4000 is off by enough to see. That is why VBR drift reads as "gets worse toward the end" rather than a fixed offset you could nudge out with a delay.

The fix is to remove the variable from the equation. This build ships every track as a 48 kHz WAV. The design note is one line: run visualizeAudio() on the 48 kHz WAV, which is the format useAudioData() wants, with no transcode. The manifest confirms it. All ten tracks are tracks/<slug>/audio.wav. WAV is linear PCM. Byte offset maps to time by a fixed formula, decoded duration equals real duration to the sample, and the grid useAudioData() builds is exact. There is nothing for a header to lie about.

Choosing WAV costs disk and load time. A 48 kHz WAV is large next to an MP3, and useAudioData() pulls the whole thing into memory. On this build that trade is worth it because the tracks are a few minutes long and the render is local. If you were streaming a one-hour file to a browser you would reach for useWindowedAudioData() instead. It makes HTTP Range requests to load only the audio around the current frame rather than pulling everything into memory. This build does not use it. The files are short enough that whole-file decode is simpler and the sync guarantee is the same.

Class two: two audio sources fighting

The second class only appears when there is a video plate. The visualiser has comps that composite a music-video clip behind the reactive layer, and a video file carries its own audio track. Now there are two candidate timelines: the audio muxed into the video, and the standalone WAV. If the video plays its own audio and your analysis reads the WAV, the two can drift apart even when each is individually fine, because they were encoded separately and their zero points need not align.

The fix is to have one source and one source only. The video is muted, and the WAV drives both playback and analysis:

// ReactiveFinish.tsx:74
<OffthreadVideo src={staticFile(video)} muted ... />
// ReactiveFinish.tsx:67
<Audio src={staticFile(audio)} />

The comment in that file states the rule plainly: the clean WAV is the single source of both sound and analysis. <OffthreadVideo> is muted everywhere it appears, including the second blurred bloom copy layered for the screen-blend glow. The <Audio> component, always staticFile(audio), is the only thing that produces sound, and it is the same file useAudioData() decoded for visualizeAudio(). One timeline feeds the ears and the analysis both. There is nothing for the two to disagree about.

Class three: AAC chunking pops on a Lambda render

The third class does not show up locally at all. It appears when you distribute the render, and it presents differently: not a gradual drift but a click or pop at regular intervals in the exported audio.

Remotion Lambda splits a render into chunks and encodes them in parallel across many functions. Each chunk is a slice of the timeline encoded independently, then the slices are stitched into the final file. Encode a lossy codec like AAC per chunk and the boundaries do not always line up sample-for-sample, and the discontinuity is audible as a tick. Remotion knows this. The renderMediaOnLambda() docs describe the audio-codec option and note that each chunk might use an uncompressed codec and convert it in the final encoding stage, specifically to prevent audio artifacts. The two-stage encode exists because the naive per-chunk lossy encode pops.

This visualiser has never rendered on Lambda. There is no @remotion/lambda dependency, no renderMediaOnLambda, no deploySite anywhere in the repo. Lambda appears only in the design doc as an open decision, a "move batch to Lambda if needed" that was never needed. Every render this build has produced is local, through scripts/render-all.mjs, which shells out to npx remotion render with --gl=angle and --log=error. Local rendering encodes the audio in one pass, so the chunk-boundary class cannot arise. That is a reason WAV plus a single local encode was the right default, not a claim about a Lambda pipeline I ran.

Diagnosing drift with a frame slice, and the perceptual class

You cannot fix a timeline you cannot see, and re-rendering three minutes to check one bass hit is a waste. render-all.mjs takes a --frames 3000-3450 flag that renders only a slice of the composition, so you can pull the exact 18 seconds around a suspect moment in a fraction of the time. Alongside it the build keeps QC stills at fixed timestamps, out/qc_2s.png, qc_20s.png, qc_32s.png, so a known beat can be checked frame-by-frame against where the visual actually lands. Render the slice, step through it, and read whether the flash is on the transient or behind it. That tells you drift from not-drift before you touch a codec.

There is a fourth thing that reads like sync drift and is not, and it is worth naming so you stop hunting a bug that is not there. On a real track the onset the ear calls "the beat" and the sample where energy actually peaks are not always the same instant. Soft-attack sounds ramp up, so a detector fires slightly late, or lands on a secondary peak the ear ignores. This build runs visualizeAudio() twice per frame: once with smoothing: true for the energy envelope that drives the bloom and breath, once with smoothing: false for the raw transient that drives the punch and flicker. The smoothed envelope has a longer arc, so a value that peaks a frame early or late still rises and falls across several frames. Anything driven by it absorbs a frame of slop. The raw transient does not. It reads every frame exactly, which is why the punch is the layer where a genuine mistiming shows first.

The rule

Sync drift is a timeline mismatch, and there are three timelines that can drift. Fix each at its source. VBR MP3 drift is a lying decoded grid, and the fix is a WAV or a constant-bitrate source before the file ever reaches useAudioData(). Two-source drift is a muted-video-plus-separate-audio problem, and the fix is one WAV driving both playback and visualizeAudio(). AAC chunking pops are a distributed-encode problem, and the fix is a deliberate audioCodec and the two-stage encode Lambda already documents, or a local render that never chunks at all.

Diagnose before you patch. Render a frame slice, check it against a QC still, and decide whether the offset is systematic drift or a single perceptual miss. A systematic offset that worsens toward the end is almost always the VBR grid. A one-off that looks wrong alone and reads fine in motion is the envelope, not the codec. The getByteFrequencyData() grid is only as trustworthy as the decoded duration it sits on. Get the source clean and keep to one timeline. Then the analysis you wrote in Chapter 1 lands where the sound does.

Chapter 5 leaves the timeline and takes the spectrum you now trust into a radial layout: turning the 32-bin frequency array into a circular visualiser.