A "state of the art" AI dubbing model's lip-sync score collapses 34 points in the one setting real dubbing uses
CoSync-DiT, a flow-matching movie-dubbing model described in an arXiv paper submitted 14 April 2026 (arXiv:2604.12292), reports state-of-the-art lip-sync and voice-identity scores against prior methods on the CelebV-Dub benchmark. Its own ablation tables, not a competitor's critique, show audio-visual sync falling from 65.94 to 31.79 (a 34-point, 52% relative drop) and speaker similarity from 65.21 to 47.24 once the test moves to an out-of-domain movie with a non-parallel reference clip, the exact mismatch every real dubbing job has: there is never a recording of the target actor saying the target-language line.
The SOTA comparison runs on the one condition a real dubbing job never gets
CoSync-DiT's headline comparison against prior methods (AlignDiT, InstructDubber, EmoDubber) uses what the paper calls Setting 1 on the CelebV-Dub benchmark: the reference audio is matched, ground-truth speech, the actor's own recording of that exact line. In that setting CoSync-DiT reports AVSync 65.94%, speaker similarity (SPKSIM) 65.21%, word error rate (WER) 4.29%, and emotion similarity (EMOSIM) 84.61%, and the paper presents this as beating the prior state of the art.
No dubbing job works that way. A localization team dubbing a film into another language has a voice actor's existing recordings, never a recording of that actor speaking the new target-language line in sync with the new video. CoSync-DiT's own Setting 2 tests exactly this mismatch: the reference is a different, unaligned clip of the same speaker. On CelebV-Dub, that alone drops AVSync to 52.12 and SPKSIM to 53.44. Push the same mismatch onto an unseen, out-of-domain movie (the CinePile-Dub zero-shot benchmark) and the numbers fall further: AVSync to 31.79, SPKSIM to 47.24, WER up to 7.53%, EMOSIM down to 70.12%. The condition that most resembles production dubbing work is the condition the abstract's "state of the art" framing is least informative about.
What the paper is, and when it was released
"CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing" was submitted to arXiv on 14 April 2026 (arXiv:2604.12292v1, cs.SD) by Gaoxiang Cong, Liang Li, Jiaxin Ye, Zhedong Zhang, Hongming Shan, Yuankai Qi and Qingming Huang. It proposes a flow-matching framework that guides speech generation through what the authors call acoustic style adapting, fine-grained visual calibrating and time-aware context aligning, plus a Joint Semantic and Alignment Regularization term enforcing frame-level consistency. The paper reports results on the CelebV-Dub benchmark (in-domain) and CinePile-Dub (an out-of-domain movie set, used for zero-shot evaluation), each tested under two reference-audio conditions the authors call Setting 1 (ground-truth, matched reference) and Setting 2 (a different, unaligned clip of the same speaker).
The numbers, by setting
| Setting | AVSync | SPKSIM | WER | EMOSIM | Source table |
|---|---|---|---|---|---|
| CelebV-Dub, Setting 1 (matched reference, in-domain) | 65.94 | 65.21 | 4.29% | 84.61 | Tab. 8 |
| CelebV-Dub, Setting 2 (non-parallel reference, in-domain) | 52.12 | 53.44 | 6.39% | 80.29 | Tab. 9 |
| CinePile-Dub, zero-shot, Setting 1 reference | 45.24 | 60.04 | 5.59% | 77.41 | Tab. 10 / 13 |
| CinePile-Dub, zero-shot, Setting 2 reference (non-parallel) | 31.79 | 47.24 | 7.53% | 70.12 | Tab. 12 / 14 |
| Point change, best to worst setting | -34.15 (-52%) | -17.97 (-28%) | +3.24 pts (+75%) | -14.49 (-17%) | computed from above |
All values are for CoSync-DiT's own proposed method ("Ours"), read directly from the arXiv:2604.12292v1 full-text HTML (fetched 17 October 2026). The paper does not present a single combined table across all four conditions; we assembled this one from its separate tables, and the bottom row's percentage changes are our own calculation, not the paper's.
Why implicit alignment breaks when the reference audio doesn't match the target line
Diffusion-based dubbing models that use "implicit alignment," including CoSync-DiT, learn to copy vocal identity and prosodic timing jointly from a reference clip rather than through an explicit duration model. When the reference clip is the exact matched line (Setting 1), the model has almost nothing to infer: it can copy timing and timbre from the same recording. When the reference is a different, unaligned clip of the same speaker (Setting 2), the model has to separate "what this voice sounds like" from "how this specific old recording happened to be timed and phrased," and the paper's abstract itself flags that implicit-alignment methods "can be disrupted by the reference audio" in exactly this way. The zero-shot condition compounds the problem: CinePile-Dub is an out-of-domain movie the model was not trained on, so neither the visual lip-motion statistics nor the acoustic environment match training data either.
What this means for anyone evaluating an AI dubbing vendor's benchmark claims
A vendor or paper citing a dubbing benchmark's top-line number is very likely citing the matched-reference, in-domain condition, because that is the condition every published baseline in this space is compared under. That number tells you almost nothing about the condition a real localization job is in: an existing voice actor, a new movie, no matched recording of the new line. The gap between CoSync-DiT's Setting 1 and its zero-shot Setting 2 result, 34 points of AVSync and 18 points of speaker similarity, is the right order of magnitude to ask any AI dubbing vendor to disclose for their own system, specifically under non-parallel reference audio and an unseen title, before trusting a benchmark number quoted from a matched, in-domain test.
For a buyer, the practical question to ask a vendor is not "what is your SOTA score" but "what is your score when the reference audio is not the target line," because that is the condition every commercial dub actually runs in. A model's own published ablation is a source for that question even when the vendor demoing it only shows the best-case result.
Check it yourself
- Paper: arxiv.org/abs/2604.12292, "CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing," submitted 14 April 2026. The full-text HTML version at arxiv.org/html/2604.12292v1 carries Tables 8 through 14 with the Setting 1 / Setting 2 breakdowns cited here.
- Compare Table 8 (Setting 1, CelebV-Dub) against Table 9 (Setting 2, CelebV-Dub) directly to see the in-domain drop before zero-shot is even introduced.
- CinePile-Dub and CelebV-Dub are both referenced in the paper as benchmark datasets; check the paper's dataset section for access details if you want to reproduce the evaluation rather than read the reported numbers.
What would prove this wrong
If a revised version of this paper, or an independent reproduction, reports Setting 2 / zero-shot numbers for CoSync-DiT that are substantially closer to its Setting 1 numbers than what v1 shows (for example, an AVSync score above 50 rather than 31.79 under the same non-parallel, zero-shot condition), that would mean the collapse we describe here was specific to this version's evaluation run rather than a structural property of implicit-alignment dubbing models. We have verified these numbers directly against the v1 full-text tables as of 17 October 2026; a later arXiv revision superseding v1 before publication would need to be checked against the same table numbers before this framing is repeated.
Sources
- Gaoxiang Cong, Liang Li, Jiaxin Ye, Zhedong Zhang, Hongming Shan, Yuankai Qi, Qingming Huang, "CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing," arXiv:2604.12292, 14 April 2026. arxiv.org/abs/2604.12292 (full text: arxiv.org/html/2604.12292v1)