Rendering: memory, longer tracks, and Lambda quirks

Chapter 7 of 7. Prerequisite: Chapter 6, where the video plate goes muted and the clean WAV becomes the single source of both sound and analysis. That decision was a bet on a specific render path. This chapter cashes it out: how the memory behaves on longer tracks, why the WAV-only streaming hook traps MP3 users, and why a Remotion render goes out of sync the moment you move it to Lambda.

The visualiser renders locally. Every finished file in this project came off the Remotion CLI on my own machine, never a cloud queue. The batch driver scripts/render-all.mjs walks the manifest and shells out to that CLI; the polished masters like out/lyrical-assassin__final-master__v02.mp4 (about 329 MB, a three-minute track at full resolution) were driven straight from the CLI too. No Lambda render ever shipped from this repo. The chapters that mention Lambda mention it as a documented Remotion capability the build deliberately did not reach for. This one explains what would have broken if it had, so the target query gets a straight answer instead of a war story I never lived.

How the audio actually loads, and what that costs

Every audio-reactive composition in the build loads its track the same way, wrapping the whole file in a React hook:

// TrackVisualiser.tsx:48
const audioData = useAudioData(staticFile(audio));

audioData is guarded with if (audioData) because the hook returns null on the first render while the file decodes. The useAudioData docs describe that guard machinery precisely: the hook "wraps the function in a delayRender() / continueRender() pattern, and handles the case where the component gets unmounted while the fetching is in progress." Once it resolves, the whole decoded waveform sits in a buffer for the entire render and visualizeAudio() samples it per frame. For a three-minute WAV that buffer is large but survivable, so nothing in this project needed a second strategy.

The cost scales with track length, not with what you do per frame. A 48 kHz stereo WAV, held as float32 for the analysis, costs about 0.4 MB per second: 48000 samples times 2 channels times 4 bytes lands near 384 KB. Three minutes is roughly 70 MB, which is fine. A forty-minute DJ set or a podcast episode is a different question, because the buffer grows linearly with track length and every parallel render worker holds its own copy.

The WAV-only windowed hook, and why it traps MP3 users

Remotion ships an answer for the long-track memory problem. useWindowedAudioData() never loads the whole file. The docs draw the contrast directly: "Unlike useAudioData(), which keeps all of the audio data in memory, this function makes HTTP Range requests to only load the audio data around the current frame." You get a sliding window of samples near the playhead instead of the entire waveform, and memory stays flat no matter how long the track runs.

The pain point sits in the fine print. useWindowedAudioData is WAV-only. Range-requesting arbitrary byte offsets into an audio file only yields decodable samples when the container has a fixed, seekable byte-to-time relationship, and a compressed MP3 does not. So the moment you reach for the windowed hook to survive a long track, you are forced to have a WAV. Anyone who arrived with an MP3, which covers most people with a music library, is stuck: the hook that solves the memory problem refuses the format they have.

If you are staring at a long MP3 and the windowed hook, the honest path is to transcode once to WAV up front and feed that WAV to the windowed hook, or stay on useAudioData and give the render more memory. Transcoding once is the same no-transcode-at-render-time discipline this build already runs, moved to a one-off preprocessing step so the render stays predictable.

Managing frame memory on the heavy compositions

The frame side is where this project actually spent its memory budget. The storyboard compositions lay down full-resolution photographic plates and cross-dissolve between them, and the naive version mounts every plate for the whole timeline. The code flags the waste in a comment before it fixes it:

// StoryboardVideo.tsx:99
// Only mount plates near the playhead — 15 full-res PNGs at once is wasteful.

The composition only mounts the plates near the playhead. A plate that will not appear for another twenty seconds is not in the tree yet, so its decoded image is not in memory yet. This is the frame-image analogue of what useWindowedAudioData does for audio, done by hand at the component level because the plates are images this build controls directly.

The heavier compositions push further into GPU territory. The parallax layer runs a WebGL displacement shader through @remotion/three, and the reactive-finish composition composites an <OffthreadVideo> plate underneath the visualiser, plus a second blurred screen-blended copy of the same video for bloom. <OffthreadVideo> extracts the exact frame with FFmpeg and hands it back as an image instead of driving a live <video> element, which is what keeps it frame-accurate under a headless render. It is also memory the render pays per frame. The design notes flag the WebGL-plus-particles path as the expensive one and float splitting long renders into segments/ as the release valve, an open decision because three-minute tracks never demanded it.

The render command that ships the file

The local render shells out to the CLI, and the flags matter more than the length of the command. The entry-point argument is mandatory: you point the CLI at the compositions bundle, then name the composition and the output.

npx remotion render src/index.ts <composition-id> <out>.mp4 \
  --gl=angle \
  --log=error

--gl=angle selects the ANGLE OpenGL backend, the one that renders the WebGL parallax shader reliably in a headless context. --log=error keeps the batch output readable when render-all.mjs walks the whole manifest. The rest of the render policy lives in remotion.config.ts so it applies to every composition without repeating flags:

Config.setVideoImageFormat('jpeg');
Config.setOverwriteOutput(true);
Config.setConcurrency(4);

concurrency 4 is the memory dial. Remotion renders frames in parallel across workers, and every worker holds its own copy of the decoded-audio buffer and its own frame plates. Four workers on a machine with headroom is a sensible default. On a long track with the whole-file audio hook, four copies of a large buffer is exactly where a render dies with an out-of-memory kill, and dropping concurrency is the first lever to pull before anything more exotic. jpeg frames are cheaper to hold and encode than PNG, at the cost of lossless intermediates, the right trade for a photographic visualiser where no frame is a flat UI screenshot.

Why a Remotion render goes out of sync on Lambda

Here is the direct answer to the query, and it is a real property of the platform, not a bug in your composition. Lambda does not render your video in one pass. It splits the timeline into chunks, renders them on separate function invocations in parallel, and stitches the results. Video frames stitch cleanly because a frame is a frame. Audio is where the seams show, because encoding audio independently on each chunk and then concatenating the encoded pieces produces gaps or pops at every boundary, and those boundaries read as drift against the picture.

Remotion knows this and handles it in the encode, which is why the fix lives in a codec choice and not in your composition. The renderMediaOnLambda docs describe the mechanism on the audioCodec option: "Each Lambda chunk might actually choose an uncompressed codec and convert it in the final encoding stage to prevent audio artifacts." Each chunk renders its audio uncompressed, the pieces concatenate without a lossy boundary artifact, and a single final pass encodes the whole track to the codec you asked for. That two-stage encode is the thing standing between you and out-of-sync audio on a chunked render.

So when a local render is perfectly in sync and the Lambda render of the same composition drifts or pops, the first suspect is the audio codec forcing a per-chunk compressed encode instead of letting Remotion do the uncompressed-then-final dance. Do not go hunting in your visualizeAudio maths. The composition is identical. The chunk boundaries are the new variable.

Reuse a value you already compute

The performance lesson worth carrying out of this project is smaller than Lambda and more useful day to day. Reuse a value you already compute per frame instead of adding a fresh pass for every visual response.

The visualiser isolates a sub-bass envelope once per frame, then spends it three times over. The same sub value pushes the Ken Burns scale, lifts the brightness, and drives the opacity of the celestial bloom:

// TrackVisualiser.tsx:63-68 (abbreviated)
const scale = kb + sub * 0.02 + kick * 0.015;
const brightness = 1 + sub * 0.22 + kick * 0.12;
// ...and the sub-driven celestial bloom reads the same envelope for its glow

One visualizeAudio read feeds three visual channels. There is no second analysis pass for brightness and no third for the bloom. The cheapest reactive channel is the one that piggybacks on a number already sitting in scope.

The static mask over the light-ray fan is the same instinct applied to layout. The volumetric rays are a rotating conic-gradient, and a fixed radial mask-image fades that full-frame fan to transparent at its edges so it reads as a soft radial burst:

// TrackVisualiser.tsx:127-128
maskImage: 'radial-gradient(closest-side, black 0%, rgba(0,0,0,0.5) 28%, transparent 66%)',

The mask never changes. It is computed once and reused on every frame while the rays underneath rotate. A constant mask costs nothing after the first paint, and the motion comes entirely from the cheap layer beneath it. When a per-frame value or a static asset can carry a second job, hand it that job before you reach for a new render pass.

Where this leaves you

Take the chunk-boundary rule with you, because it is the answer people search for and rarely find stated plainly. A Remotion composition that is in sync locally and drifts on Lambda is almost never a composition bug. It is the audio codec being encoded per chunk instead of uncompressed-then-final. Fix the codec, not the maths.

The rest is memory discipline you can apply on any machine today. useAudioData holds the whole waveform, comfortable at three minutes and a wall at forty. useWindowedAudioData stays flat on memory at the price of a hard WAV-only requirement that a one-off transcode answers. Frame memory comes down by mounting only the plates near the playhead and by turning concurrency down when a longer track runs out of headroom. Every one of those is a legible lever, and knowing which to pull first is the whole skill.