Bilingual captions look like a UI feature: translation on top, original at the bottom. The hard part is the word doing all the work in that sentence — accurate. Every translated line has to be the line being spoken, for the whole video, no matter how you seek.
Shortly after we shipped bilingual captions, we got a peculiar bug report: the Chinese line was perfectly good Chinese — for a scene three minutes away. Worse, one phrase would sometimes repeat eight times and stack into a wall of text. This note covers what we found, and the three design decisions that ended it.
The free lunch is poisoned
Video platforms ship machine-translated caption tracks. Free, instant, one request away — every caption product reaches for them first, and so did we. Then we measured them against the original track. Verdict: the text is passable, the timing is not trustworthy. The offset between translation and speech isn't constant — it accumulates as the video plays. Accumulating error means there is no calibration to apply: the track isn't misaligned by some amount, it was never aligned at all.
A free lunch that charges your product's reliability is not free.
From fragments to sentences
What actually comes back from the platform when you ask for the original track is not sentences. It's a few hundred fragments, cut by speech pauses, each with millisecond timing:
RAW FRAGMENTS (split by speech pauses)
382.1s "I guess heterosexual women"
383.3s "aren't allowed to have hair anymore."
385.7s "They have to shave their heads instead..."
|
| deterministic sentence merge
v
SPINE CUES (split by sentences)
#153 382.1s - 385.0s "I guess heterosexual women
aren't allowed to have hair anymore."
#154 385.7s - ... "They have to shave their heads..."
timing = exact union of the fragments, never invented
The first two fragments are one sentence. Auto-generated captions are like this everywhere — a thought split across two or three cues is the norm, not the exception.
You can't feed these to a translator as-is: half a thought translates into anything — "I guess heterosexual women", alone, can go anywhere. And you can't display them as-is either — fragments flash by too fast to read. So the first pass is a deterministic merge that sews fragments into sentence-shaped cues: keep merging until a sentence ends, with safety valves that force a cut when a cue grows too long, would sit on screen too long, or straddles a clear silence (a pause is a natural break). The merge is script-aware — CJK text joins without spaces and gets a tighter length budget than Latin text.
One rule in that pass matters more than the rest: merging only ever happens at fragment boundaries. A merged cue's start and end are the exact union of its fragments' timing — no cue ever wears an invented timestamp.
And because the merge is deterministic — same input, same cues, same numbering, every time — two things become possible downstream: cue ids are stable enough to serve as join keys, and a finished translation can be cached against the exact shape of the track it translated. Both are about to matter.
Decision 1: one timeline
We threw the translated track away and rebuilt around a single rule: there is exactly one timeline in the system. The native caption track — fetched once, merged into sentences — becomes what we internally call the spine. Every cue has an id, a start time, a duration, and the original text:
YouTube native caption track <-- the only source of time
|
| parse -> merge into sentences
v
SPINE
#0 0.0s - 4.5s "Apple is releasing eight new..."
#1 4.5s - 8.2s "Here to comment are the most..."
#2 8.2s - 11.9s "Red Heart and Aerial Tramway."
...
|
v pushed to screen immediately (native text)
translations attach later, keyed by id -- never by time
The rule's teeth: translations carry no time. A translation is an annotation on an id, never a second track. What appears on screen at any given second is decided by the spine alone; a translation only ever answers "what is cue #21 in Chinese" — never "what should be on screen now".
Decision 2: ids out, ids back
Once the spine exists, we batch its sentences to an LLM with their ids attached, plus a few neighbouring lines as context (marked do-not-translate) so idioms and running jokes survive:
SEND RECEIVE
context #18, #19 (do not translate)
translate #20 "They have to..." #20 -> translated
#21 "Under what..." #21 -> translated
#22 "Let me think..." #23 -> translated
#23 "[Laughs] I..."
set check: #22 missing
|
next round: resend #22 ONLY
LLM output is not to be trusted structurally: it may skip a line, merge two, or invent ids. So every batch is set-validated on return — the id set coming back must equal the id set sent. Extras are discarded. Missing ids go out again in a smaller batch; if one keeps failing, we give up on that id and that single line shows the original. Failure is contained by construction.
Scheduling follows the playhead: the batch you are watching translates first, the next one is prefetched, and the rest backfill in the background. Seeking just changes which batch goes next — finished work is never redone, and a finished video is cached on your Mac permanently.
Decision 3: why it cannot drift
JOIN BY POSITION (fragile) JOIN BY ID (cannot slip)
native translated native translations
[0] ------- [0] #0 --lookup-- {#0: "..."}
[1] ------- [1] #1 --lookup-- {#1: "..."}
[2] --+ (blank -- dropped) #2 --lookup-- missing -> native
[3] +---- [2] <- off by one #3 --lookup-- {#3: "..."}
[4] --+
[5] +---- [3] <- drift grows
one hole shifts every line a hole stays a hole:
after it, forever one native line, nothing else
The left side is every ordering- or alignment-based join: one hole shifts everything after it, and the error compounds — exactly the behaviour we measured on the platform's translated track. On the right, the join key is the id itself, written into the request and required back in the response. A wrong id doesn't shift anything; it fails validation and gets retried. There is no state in which a misaligned translation is accepted.
We didn't fix drift. We removed the assumption it needs. Drift requires translations that carry their own time. Ours carry none.
![DeepSRT playing KARINA live at WATERBOMB Seoul: rapid live-stage lyrics with inline [music] markers — the translated line and the original land on the same second](/notes/accurate-bilingual-captions/live-stage-aligned.jpg)
What you get
Native captions appear the moment the video does. Translations catch up within seconds and the whole video finishes inside a minute, cached forever after. Because the LLM sees surrounding dialogue, colloquialisms and jokes come out as language rather than word-by-word MT output. And as always: no servers — translation runs on your own Gemini key, straight from your Mac. There is nowhere for us to see what you watch.
Caption products learn one lesson twice: with platform data, verify the clock before you trust the words. We've paid this tuition; this note is the receipt.