Sony and CMU Benchmark Finds MOS Scores Fail to Predict Human Judgment of AI-Dubbed Speech

What a 46-metric benchmark actually tested

Vendors selling AI dubbing routinely cite a single mean opinion score, or a MOS-prediction model such as UTMOS or NISQA-MOS, as evidence that a dub sounds close to human. The paper COMPASS, arXiv:2606.03241, submitted 2 June 2026 by Alkis Koudounas, Hayato Futami, Quentin Jodelet, Osamu Take and Emiru Tsunoo of Sony Group Corporation together with Shinji Watanabe of Carnegie Mellon University, set out to check whether that one number tracks what human listeners actually prefer.

The authors combined 46 automatic metrics across eight evaluation dimensions: translation quality measured on the text output, translation quality measured on an ASR transcript of the synthesized audio, audio naturalness, speaker consistency, prosody and emotion, isochrony (temporal alignment), isometry (length and character-ratio compression) and lipsync. They ran those 46 metrics against 1,248 model-language configurations built from ten language pairs in the FLEURS and CVSS datasets, covering three system architectures: four end-to-end models (SeamlessM4T-Medium, SeamlessM4T-Large-v2, Qwen2.5-Omni, Qwen3-Omni), two-stage speech-translation-plus-TTS cascades built from Whisper, SeamlessM4T, Gemma or Qwen paired with CosyVoice3 or Chatterbox-TTS, and three-stage ASR-plus-MT-plus-TTS pipelines adding NLLB-200 as the text translator.

Then the authors ran a separate human-rating study across three domains, chosen to stress different things: dubbing (the MELD-ST corpus, English-Japanese, short, emotionally dense dialogue), podcasts (the EuroParl Multimedia Centre corpus, English into German, Spanish, French and Italian, long-form conversational speech), and medical consultations (MultiMed-ST, English-Chinese, where exact fidelity matters most). For each domain, they checked which of the 46 automatic metrics actually agreed with what the human raters said.

NISQA-MOS, a generic speech-quality predictor with no knowledge of dubbing-specific goals, moved in the opposite direction from how humans rated emotional preservation on the same clips: rho = -0.90, with a 95 percent confidence interval of -1.00 to -0.69. The paper reports that UTMOS shows the same near-zero-or-negative pattern across all three domains, not just dubbing. Metrics built for prosody and timing (AutoPCP, CPS Ratio, Delta Duration) reached rho = 0.91 (CI 0.78 to 0.99) against the same human panel, and timing alone (Delta Duration plus Speech Overlap) reached rho >= 0.94, the paper's highest reported correlation anywhere, on exactly the domain where the authors say human raters agreed with each other least.

What agreed with humans, domain by domain

The dubbing result is not a one-off. COMPASS ran the same check in podcasts and medical consultations and found a different metric family winning in each, which is the paper's central argument: a single leaderboard metric cannot serve every use case.

Note the difference in confidence interval width. Dubbing's -0.90 and 0.94 come with wider bands than medical's 1.00 [0.72, 1.00], which the paper attributes partly to smaller, noisier samples and partly to the dubbing domain having the lowest inter-rater agreement of the three.

How 46 metrics turn into 10, and why architecture choice matters

COMPASS is not only a scorecard, it is also a method for cutting evaluation cost. After computing all 46 metrics on all 1,248 configurations, the authors applied correlation filtering across directions and kept 10 metrics per translation direction. That reduced set preserves system rankings at Spearman's rho greater than 0.80 against the full 46-metric set, while cutting the measured evaluation time by about 2.5 times.

The same 1,248-configuration run exposed how differently cascaded and end-to-end systems trade off. Best-versus-worst gaps between architectures exceed 30 percent on naturalness and speaker preservation, but stay within a few points on translation quality. In plain terms, nearly every architecture gets the words right to a similar degree, but they diverge sharply on whether the output sounds natural and keeps the original speaker's voice, which is the exact axis a single translation-quality metric would never catch.

One confound the paper's own limitations section flags: cascades built on Whisper-Lv3 for transcription are evaluated partly using Whisper-based metrics, which risks a self-evaluation loop where a system and its scorer share the same backbone. The authors list it as a limitation rather than a result, and it is one more reason a single automatic score, cascaded or end to end, deserves a second look before a vendor cites it as proof of parity with human dubbing.

What this means for someone doing this work

A single MOS number cited as proof of dubbing quality should be read skeptically. On COMPASS's dubbing test set, NISQA-MOS moved opposite to human ratings of emotional preservation (rho = -0.90). A vendor, among ElevenLabs, Deepdub, Papercup or the dubbing units of Iyuno, TransPerfect, RWS and Lionbridge/TELUS International, citing one aggregate MOS score for a dub is citing the exact metric type this paper's own human study found least reliable for that domain.

Evaluation budgets should follow the domain, not a fixed metric list. A buyer evaluating a podcast-localization vendor gets more signal from translation-quality and isometry metrics (rho up to 1.00 in COMPASS's medical and podcast tests); a buyer evaluating a dubbing vendor gets more signal from prosody and timing metrics (rho 0.91 to 0.94), because that is what tracked human judgment in each respective domain.

Architecture choice hides inside a single leaderboard number. A greater than 30 percent best-versus-worst spread on naturalness and speaker preservation, against only a few points of spread on translation quality, means two systems that look nearly tied on translation accuracy can sound completely different to a listener, and a leaderboard that reports only one blended score will not show that.

Cutting metrics for speed is defensible, cutting them to one is not. COMPASS's reduction from 46 to 10 metrics per direction preserved rankings at rho greater than 0.80 while running about 2.5 times faster, which is a workable shortcut for a team scoring thousands of dubbed minutes. Reducing further, to the single MOS number vendors tend to publish, is the step the paper's human-validation results argue against specifically for dubbing.

Check it yourself

# Paper abstract and metadata
curl -s https://arxiv.org/abs/2606.03241

# Full text, including the human-evaluation tables and limitations section
curl -s https://arxiv.org/html/2606.03241v1

# PDF
curl -s -o compass.pdf https://arxiv.org/pdf/2606.03241

# Datasets used for the 1,248-configuration benchmark
# FLEURS: https://huggingface.co/datasets/google/fleurs
# CVSS:   https://github.com/google-research-datasets/cvss

# Metrics named in the human-validation results, to reproduce the correlations
# NISQA-MOS, UTMOS, AutoPCP, CPS Ratio, Delta Duration, Speech Overlap,
# COMET-DA, Semantic Score, ChrF++, Chars Compliance, Chars LR

The paper's own limitations section is worth reading before citing any of its numbers as final: it flags read-speech bias in FLEURS and CVSS, a possible Whisper self-evaluation loop, sentence-level human evaluation that misses long-form effects, and no latency measurements at all, since the benchmark is offline only.

What would prove this wrong

The claim here is narrow: on MELD-ST's English-Japanese dubbed dialogue, NISQA-MOS moves opposite to human ratings of emotional preservation (rho = -0.90), and timing-only metrics alone reach rho >= 0.94 on the same panel. That would not hold if a replication on a different dubbing corpus, for example a cinematic, film-grade dataset in a different language pair, finds NISQA-MOS or UTMOS correlating positively, above roughly 0.3, with human emotional-preservation ratings, or finds the prosody and timing metrics scoring meaningfully below 0.7. Watch for a camera-ready version of arXiv:2606.03241, a competing replication from another lab, or a dubbing vendor publishing its own human-correlation numbers for a production pipeline, through the end of 2027.

FAQ

What is the best way to measure AI dubbing quality in 2026?

Not a single generic mean opinion score. COMPASS (arXiv:2606.03241) found that on dubbed clips, prosody and timing metrics such as AutoPCP, CPS Ratio and Delta Duration correlated with human ratings at rho 0.91 to 0.94, while the standalone MOS predictor NISQA-MOS correlated at rho -0.90 with human ratings of emotional preservation on the same clips.

Is AI dubbing better than human dubbing, or the other way around?

COMPASS does not run that comparison directly, since it benchmarks evaluation metrics rather than AI dubs against human dubs. It does report that human raters agreed with each other least in the dubbing domain of the three domains tested, which means judging dub quality, by machine or by a human panel, is a less stable exercise in dubbing than it is in podcast or medical speech translation.

How can a localization vendor evaluate AI dubbing at scale without a human panel scoring every clip?

COMPASS's own metric-reduction result offers one answer: its 46 metrics can be filtered to 10 per translation direction while preserving system rankings (Spearman's rho greater than 0.80) and cutting evaluation time by about 2.5 times. That keeps programmatic QA workable at scale, provided the retained metrics are the domain-appropriate ones, prosody and timing for dubbing, rather than a single generic MOS score.

Sources

1. Alkis Koudounas, Hayato Futami, Quentin Jodelet, Osamu Take, Shinji Watanabe, Emiru Tsunoo, "Benchmarking Speech-to-Speech Translation Models" (COMPASS), arXiv:2606.03241, Sony Group Corporation and Carnegie Mellon University, submitted 2 June 2026. arxiv.org/abs/2606.03241
2. Full text (HTML): arxiv.org/html/2606.03241v1
3. PDF: arxiv.org/pdf/2606.03241
4. FLEURS dataset (Google): huggingface.co/datasets/google/fleurs
5. CVSS dataset (Google Research): github.com/google-research-datasets/cvss