BabelFake: English-trained deepfake detectors score 5 to 6 AUC points higher on English, and the consented data has no download yet

English-trained detectors keep an English premium on BabelFake, even though no language is "hardest"

Table 4 of arXiv:2610.06339v1 gives AUC per language for 11 detector and training-set combinations. We subtracted the mean of German, Italian, French and Spanish from the English score in each row. For (B)RAVEn audio-visual linear probes trained on DeepSpeak, whose three Hugging Face releases are all tagged language:en, English scores 78.73 against a four-language mean of 72.49, a premium of +6.25. The video-only probe shows +4.90. AVH-Align supervised on FakeAVCeleb shows +5.22. Swap the training set for MAVOS-DD, which is multilingual, and the same (B)RAVEn audio-visual and video-only probes come out -1.26 and -1.55, slightly worse on English than elsewhere.

The authors' summary sentence is also true: the worst language differs by row (English twice, French three times, Spanish four times, German and Italian once each, by our count), so no single language is the hard one. Both readings come from the same table. For a team that will deploy a detector trained on mostly English data, the conditional reading is the one that matters, and the paper never prints it as a number. One row runs against the pattern: the audio-only (B)RAVEn probe trained on DeepSpeak is 5.05 points worse on English, but at 60.09 overall AUC it sits close to chance on this benchmark. The remaining rows: AVH-Align trained on AV-Deepfake-1M shows +3.60 (unsupervised) and +3.37 (supervised) at low absolute AUC, SpeechForensics, which has no deepfake training, +1.49, and the two other MAVOS-DD rows -0.94 (audio-only (B)RAVEn) and -3.71 (AVH-Align supervised).

A size caveat, which is our judgement: the split is identity-disjoint 70/10/20 over 496 people, so the test set holds about 99 identities, roughly 20 per language. Every per-language AUC therefore averages over about 20 faces and voices, and the paper gives no confidence intervals or per-identity bootstrap. A 5 to 6 point gap at that size is suggestive, not settled. The direction flipping with the training set is the more reliable part of the signal.

What Segna, Tschesche and Rohrbach released on 5 October 2026

arXiv:2610.06339, BabelFake: A Multilingual Audio-Visual DeepFake Benchmark, by Carlotta Segna, Joel Tschesche and Anna Rohrbach (TU Darmstadt and hessian.AI, Germany), v1 submitted Monday 5 October 2026 at 13:42 UTC, 25 pages, cs.CV, marked "under submission". The arXiv licence on the PDF is the default non-exclusive distribution licence, which covers the paper, not the data. Funding came from the hessian.AI Service Center (BMFTR grant 16IS22091) and the hessian.AI Innovation Lab.

Participants were recruited on Prolific, one study per language, paid at Prolific's recommended GBP 9 per hour for a session of about 40 minutes, and recorded on their own webcams at home. The protocol mixes "The North Wind and the Sun", Harvard-style phonetically balanced sentences per language, quick-fire and open-ended questions, emotional reading prompts from M-AILABS and scripted head and hand actions. Real clips are transcribed with Whisper Large and cleaned with Gemini 2.5 to drive four multilingual voice-cloning engines (Qwen3-TTS, FishAudio2, OmniVoice, ChatterBox). Those voices feed 3 lip-sync and 4 portrait-animation methods; 4 face-swap methods keep the authentic audio. The paper counts 15 generative models in all.

The cohort covers 45 countries, ages 18 to 78 (median 32), 49.6% female and 50.4% male, and 72.2% White, 11.1% Mixed, 8.7% Black, 3.6% Asian and 4.0% other. Detection is benchmarked cross-dataset with SpeechForensics, AVH-Align and (B)RAVEn linear probes, plus an in-domain run (Table 12) where AVH-Align reaches 97.37 AUC and (B)RAVEn-AV 95.52, so the test set is learnable once you can train on it.

What the tables, the consent text and the release check show

Paper values from arXiv:2610.06339v1 (5 October 2026). Differences, ratios, recomputed correlations, the k/n check and release checks are BLOMEGA computations, 6 October 2026.

The release finding first. The paper contains no URL for the dataset, code or an access form; the only links are to third-party tools. On 6 October 2026 the Hugging Face API returned empty lists for datasets, models and spaces matching "babelfake", GitHub repository search returned 0 results, and babelfake.github.io, github.com/babelfake and huggingface.co/datasets/babelfake all returned 404. Section 1 says "we release BabelFake"; Section 7 says it "will not be released publicly on the open web". As of the check, the release is a description of future access terms.

The consent wording does not line up across the paper. Section 3.1 says participants "explicitly agree to the collection, processing, and public release of their recordings for research purposes", and Section 3.2 says the form explains "public dataset release". The form reproduced in Appendix A says "the collected and generated data will only be made available to vetted researchers upon request", restricted to "academic, non-commercial research", and Appendix A.1 confirms controlled access. Our reading, a judgement: the form and A.1 are the binding text and the narrower one; the body text overstates what was consented. For anyone hoping to license this data commercially, the form is the answer, and it is no.

The balance claim needs care too. Across languages BabelFake is balanced: 98 to 100 speakers, 18.6% to 22.2% of clips and 19.0% to 21.3% of hours per language. Within a language it is not: 17 to 21 fake clips for every real one. That ratio is normal for detection benchmarks, but a model trained without reweighting will see 1 real clip in 19, and AUC hides it while accuracy and calibrated thresholds do not.

The human-evaluation tables carry small arithmetic faults. Every rate should be a whole number of "real" votes over the stated count. 28 of 37 rows pass. HunyuanPortrait reports 43.48% on 20 ratings, which is impossible (20 ratings allow 40% or 45%), and 43.48% is exactly 10/23, the AvatarForcing real-audio row above it, so we read it as a copied cell. The aggregated fake-audio rows in Table 5 do pass (AvatarForcing 57/118, KeySync 43/113, LatentSync 21/78), so the per-TTS breakdown in Table 15 is the weak layer. None of this moves the paper's main conclusions; it does mean per-TTS human numbers should not be quoted to two decimals.

How a 40-minute webcam session becomes about 765 fakes behind an access gate

Each participant contributes about 10 minutes of real footage (83.90 real hours over 496 people, our division) and ends up in 705 to 846 fake clips depending on language, because each real clip is re-driven by several engine and voice combinations. Pairing always keeps the language fixed and relaxes ethnicity, then age, then sex until a donor is found, so every fake is same-language: a German face with a German cloned voice, never an English voice dubbed onto a German face.

Where the authentic voice survives is where detectors fail. Face swaps always keep it; lip sync and portrait animation come in both versions. Table 3's average over the four audio-visual detector variants drops by 14.57 AUC points on LatentSync (73.14 to 58.57) and 9.34 on KeySync (62.28 to 52.94) when the real recording drives the mouth instead of a cloned voice. SpeechForensics, the best overall detector at 79.79, scores 43.75 on KeySync with real audio, below chance.

Humans react in the opposite direction. With the real voice kept, the share of clips rated "Real" or "Probably Real" rises from 26.92% to 66.67% for LatentSync and from 38.05% to 63.33% for KeySync (Table 5). Those real-audio cells rest on 30 ratings each, 20 and 19 votes. The voice-cloning engine matters as much as the lip-sync engine: LatentSync with OmniVoice fooled raters 17.40% of the time and with Qwen3-TTS 62.50%, the latter on just 8 ratings (Table 15).

What this changes for dubbing, localization and multilingual data teams

AI-dubbed content will be judged by detectors with a language bias. Same-language cloned voice plus lip sync is exactly the BabelFake "fake audio" condition, and it is also what an AI dub does. Detectors flag that condition more easily than real-audio manipulations (cloned-audio averages of 62.28 to 80.65 AUC across the five engines in Table 3). If a platform runs a detector trained on English-tagged data, our reading of Table 4 is that an English dub is likelier to be flagged than a German, French, Italian or Spanish one. Labelled, consented dubs should carry provenance metadata rather than rely on how a detector scores them; the paper itself cites EU AI Act Article 50 transparency obligations.

Evaluate per language with your own data. MAVOS-DD's Hugging Face card lists Arabic, Romanian, English, German, Hindi, Spanish and Russian, and no French or Italian, yet (B)RAVEn trained on it does best on Italian (77.41) and French (75.57). Training-language coverage alone does not predict transfer. The only safe check is a per-language held-out set with enough identities for a confidence interval; 20 per language is the floor BabelFake gives, not a target.

BabelFake is a design reference, not a training asset, for commercial teams. Its terms exclude commercial use, and its consent form does not let a company retrain production models on it. What transfers is the design: per-language studies on one platform, explicit consent to face and voice manipulation, a recipient log so withdrawals reach every copy, and versioned releases. Participant pay was about GBP 6 per person (GBP 9 per hour for about 40 minutes), roughly GBP 2,976 for 496 people before pilot sessions and platform fees, our arithmetic. That buys about 10 minutes of usable real footage per head, which is the budget line to plan around.

Withdrawal is the clause to copy. Appendix A.1 commits to deleting a withdrawn participant's originals and "all derived samples that use their face or voice", and to notifying past recipients, who must delete under the data-use agreement. For a dataset where one person's face and voice can sit in hundreds of fakes paired with other people, derived-sample deletion is the part most consent forms leave out.

Coverage stops at Latin script. The authors list Mandarin, Arabic and Hindi as out of scope. A team localizing into those languages gets no evidence from BabelFake about how detectors treat their content.

Check it yourself

The release state, as of 6 October 2026:

curl -s "https://huggingface.co/api/datasets?search=babelfake"   # []
curl -s "https://huggingface.co/api/models?search=babelfake"     # []
curl -s "https://api.github.com/search/repositories?q=babelfake" | grep total_count   # 0
for u in https://babelfake.github.io https://github.com/babelfake https://huggingface.co/datasets/babelfake; do
  curl -s -o /dev/null -w "$u %{http_code}\n" -L "$u"; done   # 404 x3
curl -s https://huggingface.co/api/datasets/faridlab/deepspeak_v2 | grep -o '"language:[a-z]*"'   # "language:en"
curl -s https://huggingface.co/api/datasets/unibuc-cs/MAVOS-DD | grep -o '"language:[a-z]*"'      # ar ro en de hi es ru

The paper: curl -sL https://arxiv.org/pdf/2610.06339 -o babelfake.pdf. Table 2 is on page 4, Tables 3 and 4 on pages 6 and 7, Table 5 on page 8, the consent form on pages 12 and 13, Tables 13 to 15 on pages 21 and 22. The English premium and the per-language ratios, with values typed from Tables 2 and 4:

python3 - <<'PY'
from statistics import mean
t4={'BRAVEn-AV DeepSpeak':[78.73,74.73,71.37,70.35,73.49],'BRAVEn-V DeepSpeak':[74.84,71.38,69.99,68.16,70.22],
    'BRAVEn-AV MAVOS-DD':[73.09,72.38,77.41,75.57,72.05],'BRAVEn-V MAVOS-DD':[71.50,70.63,75.83,74.64,71.09],
    'AVH-Align sup FakeAVCeleb':[75.55,76.37,68.40,69.47,67.09],'SpeechForensics':[80.99,78.91,77.54,77.73,83.83]}
for k,v in t4.items(): print(k, round(v[0]-mean(v[1:]),2))     # EN minus mean(DE,IT,FR,ES)
t2={'EN':(4097,70465),'DE':(4016,81211),'IT':(4047,84587),'FR':(3899,70430),'ES':(4058,72889)}
for k,(r,f) in t2.items(): print(k, round(f/r,1), round(100*r/(r+f),2))   # fakes per real, real %
for n,r in [(20,43.48),(26,43.75),(32,38.24),(20,50.0)]:              # k/n check, Table 15
    k=round(r*n/100); print(n, r, k, round(100*k/n,2), abs(100*k/n-r)<=0.06)
print(mean([72.05,73.88,71.37,79.79]), mean([73.88,71.37,79.79]))    # Avg. AV: 74.27 vs 75.01
PY

Expected: +6.25, +4.90, -1.26, -1.55, +5.22, +1.49; fakes per real 17.2, 20.2, 20.9, 18.1, 18.0; the HunyuanPortrait (n=20) and the two AvatarForcing rows print False; Avg. AV prints 74.27 and 75.01. The consent conflict: compare Section 3.1, line "explicitly agree to the collection, processing, and public release", with Appendix A, paragraph "Privacy and Data Collection".

What would prove this wrong

Our claim is that BabelFake's English-tagged training sets give detectors an English premium of several AUC points, and that multilingual training removes it. It would be wrong if the authors, or anyone with access, publish per-language AUC with identity-level bootstrap intervals and the DeepSpeak-trained (B)RAVEn-AV English premium falls inside an interval that includes zero, or if a v2 of the paper with a re-drawn identity split shows the premium below 2 points. We predict that by 30 June 2027, any per-language BabelFake result for a detector trained only on English-tagged deepfake data will again put English above the mean of German, Italian, French and Spanish.

On the release: we predict that by 31 March 2027 BabelFake will still have no unauthenticated download, and that access, if open, will be a request form limited to academic institutions. A Hugging Face dataset page for BabelFake with an open or click-through commercial licence before that date would prove us wrong.

Sources

1. Carlotta Segna, Joel Tschesche, Anna Rohrbach (TU Darmstadt and hessian.AI). BabelFake: A Multilingual Audio-Visual DeepFake Benchmark. arXiv:2610.06339v1, 5 October 2026. Sections 3 to 7, Tables 1 to 5 and 12 to 15, Appendices A, A.1, C, F, G.
2. Hugging Face API searches for "babelfake" (datasets, models, spaces) and GitHub repository search, run 6 October 2026: no results. babelfake.github.io, github.com/babelfake, huggingface.co/datasets/babelfake: HTTP 404 on 6 October 2026.
3. Barrington, Bohacek, Farid. DeepSpeak dataset, faridlab/deepspeak_v2 on Hugging Face (also v1 and v1_1), tagged language:en, gated, licence "other"; read 6 October 2026.
4. Croitoru et al. MAVOS-DD, unibuc-cs/MAVOS-DD on Hugging Face, language tags ar, ro, en, de, hi, es, ru, last modified 20 June 2026; paper arXiv:2505.11109.
5. BLOMEGA computations, 6 October 2026: English premium per Table 4 row, Table 2 sums and ratios, Avg. AV recomputation from Table 3, k/n consistency check over 37 rows of Table 15, Pearson r from Tables 5 and 15, participant pay from Section 3.2.

FAQ

Can a localization or dubbing company get the BabelFake data?

Not under the terms the paper describes. Section 7 and Appendix A.1 say BabelFake will be distributed through a controlled-access system to verified researchers affiliated with academic institutions, under a non-commercial data-use agreement, and the consent form participants signed restricts use to academic, non-commercial research. On 6 October 2026 we found no download link in the paper, no Hugging Face dataset, model or space, and no GitHub repository named BabelFake.

Do deepfake detectors trained on English transfer to other languages on BabelFake?

Partly, and with a measurable English premium. In Table 4, (B)RAVEn audio-visual trained on English-tagged DeepSpeak scores 78.73 AUC on English against a mean of 72.49 on German, Italian, French and Spanish (+6.25); the video-only probe shows +4.90 and AVH-Align supervised on FakeAVCeleb +5.22. Trained on multilingual MAVOS-DD, the same (B)RAVEn probes score 1.26 to 1.55 points lower on English than on the other four. Each language has roughly 20 test identities and the paper reports no confidence intervals, so treat the gap as a direction, not a precise size.

Why do detectors miss lip-sync fakes that keep the real voice?

In Table 3 the average audio-visual detector AUC on LatentSync falls from 73.14 with cloned audio to 58.57 with the authentic recording, and on KeySync from 62.28 to 52.94. The authors read this as over-reliance on audio-visual synchronisation and synthetic-speech cues. Humans move the other way: the share of LatentSync clips rated real rises from 26.92% to 66.67% when the real voice is kept (Table 5, 30 rated clips).