In IWSLT 2026's first voice-cloning track, the best clone of your speaker got the script most wrong

The task ran for the first time in 2026, with 12 real speakers per language

July 2026, San Diego. The 23rd IWSLT evaluation campaign added a Cross-Lingual Voice Cloning track. Five teams entered: HW-TSC, IIT-Patna, KIT, Langswap and SIT-TCD. The setup is deliberately narrow, which is what makes the results readable. Systems get English reference audio from a source speaker and target-language text in Arabic, Chinese or French, and must synthesise that text in that speaker's voice. No translation step, no source transcript, no ground-truth target audio. Results are in Speech Translation and Metrics in 2026, Track VIII, Tables 13 and 14.

The reference material is unusually generous. Twelve reference audio files per target language, each a different source speaker, extracted from 12 ACL 2023 presentations in English with diverse speaker accents, most of them around five minutes long. The organisers note they deliberately did not trim them, since voice cloning normally needs a few seconds, and say some submissions benefited from the extra length. Target text came from bilingual journals: arXiv (Chinese, French), TAL (French), Journal of Software (Chinese), and An-Najah, Palestine Ahliya and Princess Sumaya University journals (Arabic), filtered by LaBSE semantic similarity. The reference text is 49 lines in Arabic, 112 in Chinese and 99 in French, which is exactly the 588, 1,344 and 1,188 evaluation samples the paper reports, at 12 speakers each.

Metrics, all automatic. Content consistency: WER and CER from faster-whisper large-v3 with beam size 5 and voice-activity filtering, after NFKC normalisation, lowercasing, punctuation removal and jieba segmentation for Chinese. Speaker similarity: cosine similarity of ECAPA-TDNN embeddings in SpeechBrain. Prosody similarity: cosine similarity over mean and standard deviation of fundamental frequency plus mean energy, extracted with Praat-Parselmouth. No human listening test was reported.

One thing the findings paper does not describe is a consent or release process for the 12 source speakers whose voices were cloned into three languages by five research teams. The presentations are public; that is not the same fact. For anyone building a commercial dubbing pipeline, the gap between "the recording is public" and "the speaker agreed to this use of their voice" is the entire compliance surface.

All 14 published results, both columns, side by side

The findings paper splits content consistency (Table 13) from speaker and prosody similarity (Table 14). They belong next to each other, because neither is interpretable alone.

Three readings, all from that table.

Chinese is the clean case. Five systems, and the WER order and the speaker-similarity order match exactly. Spearman rank correlation 1.0. That is not a trend, it is a line.

Arabic breaks the pattern, and is also the hardest. SIT-TCD leads Arabic on all four columns at once: 0.132 WER, 0.050 CER, 0.786 speaker, 0.997 prosody. The Arabic rank correlation between WER and speaker similarity is -0.4 over four systems. Arabic is still where the errors live: the best Arabic WER, 0.132, is worse than the worst French WER except one, and three times the best Chinese result.

The backbone decides the content column. Every system that put Qwen3-TTS 1.7B behind Chinese or French landed under 0.052 WER. The 0.6B version of the same family, used by Langswap, landed at 0.189 and 0.133. Parameter count inside one model family moved WER by a factor of four.

The tradeoff is written into the selection objectives the teams chose

This is not an emergent property of neural speech. Three of the five teams published the exact rule they used to pick which of several generated takes to submit, and the rule predicts where they landed.

HW-TSC is the instructive failure. It generated many takes and kept the one with the highest timbre cosine similarity against three random segments of the reference. It optimised for exactly the quantity it was about to be graded on, and finished 4th of 5 in Chinese speaker similarity at 0.580. The selection ran against its own timbre feature vectors; the grading ran against ECAPA-TDNN embeddings in SpeechBrain. Optimising against your own embedder does not transfer to someone else's.

Prosody similarity carried no information. Across all 14 results it ran from 0.980 to 0.997, a total spread of 0.017. Speaker similarity over the same 14 ran 0.479 to 0.813, a spread of 0.334, roughly twenty times wider. The prosody feature vector is mean and standard deviation of fundamental frequency plus mean energy, three summary statistics of a five-minute recording, compared by cosine similarity. Any two speech signals of similar loudness and register will score above 0.98 on that. If you are writing an acceptance test for a dubbing vendor, do not put this metric in it.

What this changes for anyone buying or building AI dubbing

Specify both numbers or you will be sold one. A demo clip that sounds uncannily like the original actor is a speaker-similarity demo. A demo that reads a technical script flawlessly is a WER demo. On the 2026 evidence, in Chinese, those were opposite ends of the same ranking. Write both into acceptance criteria with thresholds, and state which one loses when they conflict.

Pick the language before you pick the vendor. Best Arabic WER in the track was 0.132. Best French was 0.050. Same teams, same models, same reference audio. An Arabic dubbing plan priced off a French pilot is priced wrong, and the error shows up as human correction hours.

Content errors in cloned speech are invisible to the listener who needs them most. A 0.191 WER in a cloned Chinese voice is not a robotic tone, it is words that were not in the script, delivered in a familiar voice at conversational confidence. That is the failure mode to review for, and it is the reason a native-speaker QA pass is not optional at these error rates.

Fine-tuning moves identity; the backbone moves accuracy. The two fine-tuned systems took the top speaker-similarity slot in all three languages between them. The lowest error rates came from zero-shot use of a larger backbone. If you want both, that suggests fine-tuning a large backbone rather than choosing between the two, which no 2026 submission did.

Get the release before you get the reference audio. Five minutes of a person's voice is now enough for five separate teams to produce their voice in three languages they may not speak. The consent artefact needs to name the languages, the uses and the retention period, because the technical constraint that used to limit the use has gone.

Check it yourself

The tables are in one open PDF, the models are on Hugging Face, and the correlations are four lines of Python.

# 1. the source tables (Track VIII, Tables 13 and 14)
curl -sL -o iwslt2026.pdf https://aclanthology.org/2026.iwslt-1.39.pdf
python3 -c "
import pypdf
t='\n'.join(p.extract_text() or '' for p in pypdf.PdfReader('iwslt2026.pdf').pages)
i=t.find('Track VIII'); print(t[i:i+14000])"

# 2. the correlations quoted above
python3 - <<'PY'
from scipy.stats import spearmanr
zh = [("IIT",0.042,0.499),("HW-TSC",0.043,0.580),("KIT",0.113,0.609),
      ("Langswap",0.189,0.686),("SIT-TCD",0.191,0.789)]
fr = [("HW-TSC",0.050,0.582),("IIT",0.051,0.479),("KIT",0.063,0.602),
      ("SIT-TCD",0.069,0.813),("Langswap",0.133,0.761)]
ar = [("SIT-TCD",0.132,0.786),("KIT",0.157,0.637),("Langswap",0.160,0.785),
      ("IIT",0.219,0.669)]
for name, rows in [("Chinese",zh),("French",fr),("Arabic",ar)]:
    r = spearmanr([x[1] for x in rows],[x[2] for x in rows])
    print(f"{name:8s} n={len(rows)}  rho={r.statistic:+.2f}")
allsim  = [x[2] for x in zh+fr+ar]
prosody = [.989,.989,.981,.989,.993,.987,.987,.980,.996,.991,.997,.982,.996,.990]
print("speaker range", round(max(allsim)-min(allsim),3),
      " prosody range", round(max(prosody)-min(prosody),3))
PY
# Chinese  n=5  rho=+1.00
# French   n=5  rho=+0.80
# Arabic   n=4  rho=-0.40
# speaker range 0.334  prosody range 0.017

# 3. the models, all public
# https://hf.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base   https://hf.co/k2-fsa/OmniVoice
# https://hf.co/openbmb/VoxCPM2  https://hf.co/fishaudio/s2-pro  https://hf.co/ResembleAI/chatterbox

# 4. the recommended in-domain data
# https://hf.co/datasets/ymoslem/acl-6060   (ACL 60/60, translations of ACL 2022 talks)

To replicate the evaluation on your own content: transcribe the generated audio with faster-whisper large-v3 at beam size 5, normalise with NFKC plus lowercasing plus punctuation removal (and jieba for Chinese), score WER and CER with Hugging Face evaluate, and take cosine similarity of SpeechBrain ECAPA-TDNN embeddings against the reference. That is the whole protocol.

What would prove this wrong

The claim under test is that in cross-lingual voice cloning, speaker similarity and content accuracy are currently in tension, and that the tension is a property of the systems rather than of one benchmark. It is wrong if, at IWSLT 2027 or in a peer-reviewed evaluation using the same protocol, a single system posts a Chinese WER at or below 0.060 together with an ECAPA-TDNN speaker similarity at or above 0.780. In 2026 the best on each axis separately were 0.042 and 0.789, and they were different systems at opposite ends of the ranking. SIT-TCD's Arabic result, best on all four columns, is the existence proof that the tension can break, so a second edition could settle this quickly.

A second prediction, marked as judgement: the 2027 edition will drop or replace the prosody similarity metric, because a metric with a 0.017 observed range across 14 systems cannot rank anything. The organisers' own future-work section names code-switching and terminology consistency as additions and does not mention prosody. If prosody similarity is reported unchanged in 2027, this reading was wrong.

FAQ

Is there really a tradeoff between clone accuracy and speaker identity?

In IWSLT 2026, yes. Ranking the five Chinese submissions by WER and by speaker similarity gives the identical order, a Spearman correlation of 1.0. French is 0.8. Arabic is the exception at -0.4, where SIT-TCD led on both.

Which target language is hardest?

Arabic. Best Arabic WER was 0.132, against 0.042 for Chinese and 0.050 for French. French was the most tractable, with most systems under 0.07 WER and 0.02 CER.

Does prosody similarity tell you anything?

Not as measured here. It ran 0.980 to 0.997 across all 14 results, a spread of 0.017, against 0.334 for speaker similarity. It is a cosine over three summary statistics and does not separate systems.

Does fine-tuning help cross-lingual voice cloning?

For identity, clearly. The two fine-tuned systems, SIT-TCD (LoRA) and KIT (GRPO), took the top speaker-similarity result in three and one of three languages. The lowest error rates came from zero-shot use of Qwen3-TTS 1.7B.

Sources

1. IWSLT 2026 organisers (60 authors), Speech Translation and Metrics in 2026: Findings of the IWSLT Campaign, Proceedings of the 23rd International Conference on Spoken Language Translation, San Diego, July 2026. Track VIII (Cross-Lingual Voice Cloning): task setup, evaluation data, metrics, system descriptions, Table 13 (WER and CER) and Table 14 (speaker and prosody similarity). PDF.
2. Elizabeth Salesky et al., ACL 60/60 evaluation set, ymoslem/acl-6060 on Hugging Face. The in-domain development data recommended by the organisers, built from ACL 2022 presentations.
3. Model cards for the backbones named in the system descriptions: Qwen3-TTS 1.7B, OmniVoice, VoxCPM2, Fish Audio S2 Pro, Chatterbox.
4. Evaluation components: faster-whisper (Whisper large-v3, beam 5), SpeechBrain ECAPA-TDNN speaker embeddings, Praat-Parselmouth for F0 and energy, Hugging Face evaluate for WER and CER.
5. Correlation and range calculations in this note computed by BLOMEGA on 16 September 2026 from Tables 13 and 14. Code in "Check it yourself".

Related BLOMEGA guides: EU AI Act Article 50 and AI dubbing disclosure · The cost of a dubbed minute in 2026 · Consented AI training data providers.