PACT beats 9 speech-to-speech systems on voice-preserving dubbing, and its key step only runs one way

What the paper claims, and what changed

End-to-end speech-to-speech translation (S2ST) systems, speech-native multimodal LLMs, and cascade pipelines (transcribe, translate, resynthesize) each solve part of dubbing: some preserve what was said, some preserve who said it. The paper's starting claim is that none of the systems it tested reliably preserve how something was said once it crosses languages: emotion, tone, rhythm, pauses, and pragmatic attitude.

PACT (the paper's own name for its framework) is a prompting layer, not a new speech model. It sits in front of a frozen CosyVoice 3 decoder and feeds it continuous prompt tokens derived from the source speaker's pragmatic cues, remapped into the target language's own expression space rather than copied directly. The target-language mapping is the actual contribution: without it, a dubbing system either keeps the source language's prosody (which can sound wrong in the target language) or discards expression entirely (which is what most cascade pipelines do today).

Evaluation uses TOPI, a bidirectional Spanish-English benchmark: 30 evaluation pairs with target-language transcriptions, drawn from a larger 2,577-pair DRAL training set. The English-direction decoder was pretrained on a 400-hour subset of the Emilia speech corpus; no equivalent Spanish pretraining corpus was used.

The numbers, against nine other systems

Table I from the paper, reproduced in full. Scores are Spanish-to-English || English-to-Spanish. ASR-BLEU measures content fidelity via re-transcription; Prag Sim and Spk Sim are the paper's own learned similarity metrics for pragmatic attitude and speaker identity, both 0 to 100, higher is better.

PACT leads on all three metrics in both directions. The closest competitor on pragmatic similarity is SeamlessExpressive (72.0 Es→En, 81.4 En→Es), beaten by 7.3 and 1.5 points respectively; the closest on speaker similarity is also SeamlessExpressive (66.3 || 70.6), beaten by 6.6 and 1.9 points. The three closed-source systems (Qwen3.5-LiveTranslate, Qwen3.5-Omni-Plus, GPT-Audio-1.5) all trail PACT on every metric in both directions, including GPT-Audio-1.5's pragmatic similarity of just 40.0 on Spanish-to-English, the lowest score any system posts on that column.

PACT does not win everywhere. On a human evaluation of 10 utterances (Spanish-to-English only, five raters, 1-5 MOS scale), naturalness went to Qwen3.5-Omni-Plus at 4.78, against PACT's 3.64. PACT's human scores were 3.42 for pragmatic similarity and 4.08 for speaker similarity. The paper reports a Pearson correlation of r = 0.413 (p = 0.0029) between its own automatic Prag Sim metric and these human ratings, which it calls moderate, not strong.

How the prompt actually gets built

Six steps, four of them frozen pretrained models and two of them the paper's own small trained components.

The asymmetry has a stated cause, not an unstated one. The Spanish-to-English direction benefits from the 400-hour Emilia pretraining subset for English targets; no equivalent large pretraining corpus exists for the Spanish-target adapter, which trains only on the 2,577-pair DRAL set. The paper reports that enabling the same calibration step for English-to-Spanish hurt performance, so the authors turned it off for that direction rather than force a step that a smaller adapter could not support. The result, per Table I, is that both directions still beat every baseline, but through different internal configurations of the same architecture.

What this means for anyone shipping AI dubbing

The headline number that matters for a localization buyer is not the aggregate win, it is the gap between Prag Sim and ASR-BLEU improvements. PACT's ASR-BLEU lead over the next-best system (Fish-Speech at 26.6 Es→En) is 4.1 points, a translation-fidelity gain any cascade system could plausibly close. Its Prag Sim lead over the next-best system on that direction (Fish-Speech at 73.8) is 5.5 points, on a metric no other tested system was explicitly designed to optimize. The expression-preservation gain is the differentiator, not the translation quality.

The direction-dependent calibration step is the operational catch for anyone evaluating a system like this on a new language pair: a method's published aggregate score can hide the fact that its best component was quietly disabled for the direction with less training data. Any buyer piloting a pragmatic-prompting dubbing system on a new language pair should ask specifically which internal steps were enabled for which direction, not just the final blended score.

Check it yourself

• The paper: arXiv:2610.18247, "PACT: A Pragmatic Prompting Framework for Expressive and Voice-Preserving Speech-to-Speech Translation," Yuchen Song and Satoshi Nakamura, posted 13 October 2026.
• Full text with Table I and Table II: arxiv.org/html/2610.18247v1.
• Recompute the Prag Sim gap yourself: PACT 79.3 (Es→En) minus SeamlessExpressive 72.0 = 7.3 points. PACT 82.9 (En→Es) minus SeamlessExpressive 81.4 = 1.5 points. The gap is far wider on the direction where calibration (step 5) is active.
• CosyVoice 3, the frozen decoder PACT prompts: check its own documentation for what "speaker conditioning" means at the implementation level, since PACT inherits that component unchanged.

What would prove this wrong

If a follow-up evaluation on a language pair with balanced pretraining data in both directions (unlike this paper's Spanish-English asymmetry) shows the calibration step improving both directions equally, the English-to-Spanish disabling here was a data artifact specific to this paper's setup, not a structural limit of the method. If PACT's Prag Sim advantage narrows to within 2 points of SeamlessExpressive on a replication by a different lab before the end of 2027, the gain reported here should be treated as partly specific to this paper's own TOPI test split.