Netflix's lip-sync judge gets 81% of its AUC gain from scrambled video, and gains 0.056 on the wrong line
On 30 September 2026 a University of Maryland and Netflix team posted Align Then Reason (ATR), a judge that scores a candidate dub line against silent video, and reported mean AUC gains of 59.4%, 50.2% and 50.8% over fine-tuned Qwen3.5 baselines. We recomputed every row of the paper's Table 1: the numbers are exact, but 81% of the 9B gain comes from four synthetic timing corruptions where the baselines sit at chance, the wrong-line (mismatch) gain is +0.056 AUC, and on 4 October 2026 no code, weights or benchmark were public.
ATR's 50.8% gain is mostly baselines learning to see scrambled frames, not better line matching
Table 1 of arXiv:2610.00825 scores 15 systems on seven corruption axes over 2,100 test clips in seven languages. ATR with a Qwen3.5-9B reasoner reaches mean AUC 0.920 against 0.610 for Qwen3.5-9B fine-tuned (SFT) to read lips, which is the paper's 50.8% (0.920 / 0.610 = 1.508, our division). Splitting the same rows by axis, the three content axes go from a mean of 0.737 to 0.874 (+0.137) and the four temporal axes from 0.516 to 0.954 (+0.439). Summing the per-axis gains, temporal corruptions supply 1.755 of 2.167 AUC points, or 81.0%.
The suspicion that a relative gain over an AUC near 0.5 makes a small change look large does not hold here. The absolute gain is +0.310 AUC, and measured as distance above chance ATR-9B sits 3.8 times further out than its baseline (0.420 against 0.110). The 2B result is stronger still on that view: 0.394 against 0.061, 6.5 times. The issue is composition, not arithmetic. Our judgement: a dubbing lead reading "50.8% better" would assume the judge is much better at telling a right line from a wrong one, and on the mismatch axis it moves from 0.730 to 0.786.
The same split holds at every scale and backbone. Content-only relative gains over the matching baseline are 33.8% (2B), 18.3% (4B), 18.6% (9B), 15.8% (LLaMA-3.1-8B) and 16.4% (Mistral-7B); temporal-only gains are 78.1% to 85.1%. The mismatch axis gains +0.150, +0.088, +0.056, +0.041 and +0.037 respectively, shrinking as the baseline grows.
What Netflix and Maryland published on 30 September 2026
arXiv:2610.00825v1, Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing, by Rui Liu (University of Maryland, College Park, work done as a Netflix intern), Bhavin Jawade, Haoqi Li, Shivam Mehta, Karan Saxena, Yinghong Lan and Cameron R. Wolfe (Netflix). Primary category cs.CV, 15 pages, single version as of 4 October 2026. The first author claimed the paper on Hugging Face on 3 October 2026, under the Netflix organisation.
The task is reference-free: given a silent clip and a text line, score whether the line fits the mouth, before any dubbed audio exists. Stage one encodes frames with Auto-AVSR (768-d, adapted to 256-d) and the line's phonemes with XPhoneBERT, and runs CTC whose labels are the positions 1 to N of the line's phonetic units, so only an in-order path scores well. Stage two hands a LoRA-tuned LLM (rank 16) one soft token per phonetic unit plus a calibrated alignment score written as two decimals in the prompt; the Yes minus No logit is the final score.
Training uses about 50K multilingual samples from "on-screen dialogue segments"; the test set holds 2,100 clips balanced across English, Spanish, French, Japanese, Korean, Brazilian Portuguese and Turkish. Negatives are three content corruptions (mismatch: another clip's line of similar length; shuffle: the true line's words permuted; dub: the professional dub translation) and four temporal ones (reverse, circular shift, freeze, swap halves). Source, licence and consent terms for the clips are not stated.
The headline numbers recomputed against the paper's own tables
| Quantity | Value | Source |
|---|---|---|
| Mean AUC gain, ATR over same-size Qwen3.5 SFT | 59.4% / 50.2% / 50.8% (2B / 4B / 9B); recomputed exactly | Abstract, Table 1; our division |
| Share of 9B summed gain from temporal axes | 81.0% (1.755 of 2.167 AUC points) | Table 1; our arithmetic |
| Mismatch AUC, Qwen3.5-9B SFT to ATR-9B | 0.730 to 0.786 (+0.056) | Table 1 |
| Same gain with 3-seed means instead of Table 1 | 57.6% / 48.7% / 49.2% (ATR-9B 0.910 ± 0.015, not 0.920) | Table 9; our division |
| English-only mismatch, Auto-AVSR vs ATR-9B | 0.784 vs 0.765 (baseline wins) | Table 7 |
| Paired accuracy, ATR-9B vs best baseline | 0.939 vs 0.763 Auto-AVSR (+23.1%); Claude Opus 5 0.737 | Table 2; our division |
| Shift size, freeze span, frame rate | not reported | Sections 2.2, 3.1, Appendix A |
| Per-language results for full ATR | not reported; scorer-only, 3 axes | Table 8 |
| Dub-line reranking Top-1, ATR-9B vs LLaMA-AVSR | 0.479 vs 0.316 = +51.6% (abstract says 52.0%) | Table 5; our division |
| Script-to-clip per-clip accuracy, ATR-9B vs length-fit heuristic | 0.746 vs 0.656 = +13.7%; heuristic beats every lip reader | Table 6; our division |
| MuAViC zero-shot vs recalibrated mean AUC | 0.751 vs 0.875 (recalibration uses 300 genuine target-language clips) | Table 3, Section 3.3 |
| Code, weights, benchmark | none found; HF paper page 0 models, 0 datasets, 0 spaces | HF and GitHub APIs, 4 Oct 2026, our check |
Paper values from arXiv:2610.00825v1. Divisions, splits, shares and release checks are BLOMEGA computations, 4 October 2026.
Every Table 1 mean we recomputed from its seven axis values matches the printed mean to three decimals, for all 15 rows. The tables also agree with each other: the seven-language averages of the scorer rows in Table 8 give 0.696 (mismatch), 0.970 (reverse) and 0.971 (freeze), against 0.695, 0.970 and 0.971 for "w/o LLM reasoner" in Table 4, and Auto-AVSR's per-language averages give 0.601, 0.670 and 0.609 against 0.602, 0.670 and 0.609 in Table 1. Our reading: the per-language table is the scorer without the LLM, which the Table 8 caption says, and the pooled numbers are a straight average across balanced languages.
Three smaller discrepancies. First, Table 1 reports 0.894, 0.904 and 0.920 for the three Qwen3.5 reasoners, while the three-seed means in Table 9 are 0.884, 0.895 and 0.910, each about 0.01 lower; LLaMA and Mistral go the other way (0.890 to 0.895, 0.894 to 0.900). The paper does not say which seed Table 1 shows. Second, the dub-line reranking gain is 0.479 / 0.316 = 1.516, printed as 52.0% in the abstract and conclusion and as 52% in the body. Third, the Table 5 caption credits the 0.479 to "the alignment scorer", while the text and the table row credit ATR-9B.
On script-to-clip assignment the paper compares against "the best lip-reading baseline" (Auto-AVSR, 0.634 per-clip). Its own Table 6 lists an isochrony length-fit heuristic at 0.656 per-clip and 0.397 exact-block, ahead of all four lip readers. Against that heuristic ATR-9B's margin is +0.090 per-clip (+13.7%) and +0.125 exact-block, which the text mentions in passing; the 17.7% in the abstract is the lip-reader comparison.
How a single calibrated number carries all of ATR's timing sensitivity
The ablation in Table 4 locates the timing signal precisely. Removing the calibrated scalar drops mean AUC from 0.920 to 0.672, and reverse, shift and swap fall to 0.525, 0.534 and 0.517, chance level. The authors explain why: soft tokens pool over frames, so they "do not explicitly preserve temporal order". Freeze survives at 0.805 because a frozen frame changes what the mouth looks like, not only the order.
The LLM does the opposite job. Without it the scorer already reaches 0.970 reverse and 0.971 freeze, higher than full ATR-9B on reverse (0.964), and the LLM adds content discrimination: mismatch 0.695 to 0.786, shuffle 0.757 to 0.974, dub 0.701 to 0.862. Our judgement: the shuffle jump is the LLM reading word order in text it can see, since a permuted sentence is usually ungrammatical, so it measures language sense more than lip reading.
The temporal tests corrupt the video, never the text, and the text has no timestamps. That matters for what "timing" means here. Reverse plays the clip backwards, swap exchanges its halves, circular shift wraps the end to the front by an unreported amount, and freeze holds one frame for an unreported span. These are gross order errors. A real dub that starts 80 ms late, or a line whose stressed syllable lands two frames off a bilabial closure, is a different and much smaller perturbation, and arXiv:2610.00825v1 contains no offset sweep in frames or milliseconds and no frame-rate figure (searches of the full text for "fps", "ms", "offset" and "frame rate" return nothing).
Per language, the timing side is flat and the content side is not. In Table 8 the scorer's reverse AUC sits between 0.96 and 0.98 in all seven languages (spread 0.02) and freeze between 0.96 and 0.99. Mismatch spans 0.63 (English) to 0.78 (Korean), a spread of 0.15. English is the scorer's weakest mismatch language and Auto-AVSR's strongest (0.78); Auto-AVSR is at 0.50 on Japanese and Korean, which the authors attribute to tokenizer limits on non-Latin scripts.
What this means for dubbing QC and lip-sync data teams
- Read ATR as an order checker that also has content sense, not a phonetic matcher. Its strongest measured skill is noticing that frames are reversed, shifted, frozen or swapped (0.931 to 0.970 AUC). Its weakest is the case closest to a real QC call, a plausible line that is not the one being mouthed (0.786 pooled, 0.765 in English, where Auto-AVSR scores 0.784).
- The dub-line reranking result is the one to watch. Picking the professional line out of four meaning-preserving paraphrases within ±15% length, ATR-9B gets 0.479 Top-1 against 0.250 chance and 0.237 for length fit (Table 5). That is the selection step a TTS dubbing pipeline would run, and the gain over the best lip reader is 0.163 Top-1.
- Keep the length heuristic in the baseline set. For assigning lines to shots, isochrony length fit (0.656 per-clip) beats every lip-reading model in Table 6. Our judgement: any vendor pitching a lip-sync scorer for script alignment should be benchmarked against it first.
- Budget calibration data per language. On three unseen MuAViC languages (German, Arabic, Russian) ATR-9B drops to 0.751 mean AUC and freeze to 0.572; refitting two constants on 300 genuine target-language clips restores 0.875. So "zero-shot" in practice means 300 verified clips per new language, which a production team must source and clear.
- Frontier chat models are not a substitute today. On paired accuracy, Claude Opus 5 averages 0.737, Gemini 3.1 Pro 0.690 and GPT-5.6 variants 0.656 to 0.725 against 0.939 for ATR-9B (Table 2), and Claude Opus 5 scores 0.485 on shift, below chance.
- If you build training data for a judge like this, the gap is small timing errors. The paper's negatives are all synthetic and coarse. Human-rated clips with measured offsets of one to five frames, per language, are what would show whether the 0.93 to 0.97 temporal AUCs carry over to the errors dubbing reviewers actually flag.
Check it yourself
Get the text (we used pypdf because pdftotext was not installed; pdftotext -layout works as well):
curl -sL https://arxiv.org/pdf/2610.00825 -o atr.pdf
python3 -c "import pypdf;r=pypdf.PdfReader('atr.pdf');open('atr.txt','w').write('\n'.join(p.extract_text(extraction_mode='layout') for p in r.pages))"
grep -niE "fps|millisec| ms |frame rate|offset|github|huggingface" atr.txt # 4 Oct 2026: no matchesRecompute the headline and the axis split from Table 1 (columns: mismatch, shuffle, dub, reverse, shift, freeze, swap, printed mean):
python3 - <<'PY'
sft={'2B':[.632,.604,.685,.502,.501,.502,.503,.561],'4B':[.696,.739,.750,.510,.505,.505,.510,.602],
'9B':[.730,.737,.743,.520,.513,.510,.519,.610]}
atr={'2B':[.782,.967,.821,.945,.909,.905,.932,.894],'4B':[.784,.960,.841,.955,.926,.916,.944,.904],
'9B':[.786,.974,.862,.964,.931,.970,.952,.920]}
for k in sft:
a,b=atr[k],sft[k]; g=[x-y for x,y in zip(a[:7],b[:7])]
print(k,'rel %.1f%%'%(100*(a[7]/b[7]-1)),'abs +%.3f'%(a[7]-b[7]),
'temporal share %.3f'%(sum(g[3:])/sum(g)),'mismatch +%.3f'%g[0],
'above-chance ratio %.1f'%((a[7]-.5)/(b[7]-.5)))
for k,s in {'2B':.884,'4B':.895,'9B':.910}.items(): print(k,'seed-mean gain %.1f%%'%(100*(s/sft[k][7]-1)))
print('rerank',0.479/0.316-1,'s2c vs isochrony',0.746/0.656-1)
PYExpected: 2B 59.4%, 4B 50.2%, 9B 50.8%; temporal share 0.810 at 9B; mismatch +0.056 at 9B; seed-mean gains 57.6%, 48.7%, 49.2%; rerank 0.516; script-to-clip 0.137. Table 1 is on page 6 of the PDF, Tables 2 and 3 on page 7, Table 4 on page 8, Tables 7 to 9 on page 15.
Release state, run on 4 October 2026 at 19:05 UTC:
curl -s https://huggingface.co/api/papers/2610.00825 | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['numTotalModels'],d['numTotalDatasets'],d['numTotalSpaces'],d['organization']['name'])"
# 0 0 0 netflix
curl -s https://huggingface.co/api/arxiv/2610.00825/repos # {"models":[],"datasets":[],"spaces":[]}
curl -s "https://huggingface.co/api/models?author=lr10260" # 19 models, newest createdAt 2026-05-21, none ATR
for p in 1 2 3; do curl -s "https://api.github.com/users/Netflix/repos?per_page=100&page=$p"; done \
| grep '"name"' | grep -iE 'lip|sync|dub|judge|align|reason' # 239 repos, none relevantWhat would prove this wrong
Our claim is that ATR's reported advantage is concentrated in coarse synthetic timing corruptions, and that its edge on telling a right line from a plausible wrong one is small (+0.056 AUC on mismatch at 9B, behind Auto-AVSR in English).
It would be wrong if the authors, or anyone with released weights, publish an offset sweep showing ATR at 0.85 AUC or above on shifts of three frames or fewer (about 120 ms at 25 fps), or a human-rated dubbing test where ATR's mismatch-type AUC beats a fine-tuned lip reader by 0.10 or more. We predict that by 31 March 2027 neither will be public, and that if weights are released, an independent re-run of the mismatch axis lands within 0.03 of 0.786.
Sources
- Rui Liu, Bhavin Jawade, Haoqi Li, Shivam Mehta, Karan Saxena, Yinghong Lan, Cameron R. Wolfe (University of Maryland, College Park; Netflix). Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing. arXiv:2610.00825v1, submitted 30 September 2026. Sections 2 to 3, Tables 1 to 9, Appendices A to D.
- Hugging Face paper page for 2610.00825, Netflix organisation, claimed by the first author 3 October 2026; API checked 4 October 2026: 0 linked models, datasets or spaces.
- github.com/Netflix, 239 public repositories listed via api.github.com on 4 October 2026; none matches the paper.
- Anwar et al. MuAViC: A Multilingual Audio-Visual Corpus. arXiv:2303.00628, 2023. Source of the German, Arabic and Russian transfer test.
- Ma et al. Auto-AVSR, ICASSP 2023; Nguyen et al. XPhoneBERT, Interspeech 2023. The public encoders ATR builds on.
- BLOMEGA computations, 4 October 2026: Table 1 means and per-axis splits, seed-mean gains from Table 9, Table 8 to Table 4 cross-check, downstream ratios from Tables 5 and 6, release checks.
FAQ
Is the 50.8% AUC improvement in Align Then Reason inflated by a baseline near 0.5?
No, the absolute gain is large: ATR-9B mean AUC 0.920 against 0.610 for Qwen3.5-9B SFT, +0.310 AUC, or 3.8 times the baseline's distance above chance. What the single number hides is where the gain sits. Recomputed from Table 1, the four synthetic temporal axes (reverse, shift, freeze, swap) contribute 81% of the summed per-axis gain, while the mismatch axis, a wrong line of similar length, moves only from 0.730 to 0.786.
Does ATR test small lip-sync offsets of a few frames?
The paper does not say. Temporal negatives are full reversal, a circular shift, a frozen frame held for a span, and swapped halves. Neither the shift size in frames or milliseconds, the freeze span, nor the video frame rate is reported in arXiv:2610.00825v1, so there is no evidence either way on offsets of one to three frames.
Can a dubbing team download and run the ATR lip-sync judge?
Not as of 4 October 2026. The paper contains no code or data link, the Hugging Face paper page lists 0 linked models, 0 datasets and 0 spaces, none of Netflix's 239 public GitHub repositories matches, and the first author's 19 Hugging Face models predate the paper (newest 21 May 2026). Its parts are public: Auto-AVSR, XPhoneBERT and the Qwen3.5, LLaMA-3.1 and Mistral backbones.