A 20-Language Benchmark Audit Finds Only 19% of Translated HellaSwag Items Are Error-Free

A structural check found broken items before translation quality was even scored

EU20 is a set of five widely used LLM evaluation benchmarks, ARC, GSM8K, HellaSwag, MMLU and TruthfulQA, machine-translated with DeepL into 20 European languages so researchers can report multilingual scores instead of English-only ones. Thellmann, Stadler and Färber built a three-step automated quality assurance pipeline to check what that translation actually produced, then compared EU20 against two sibling multilingual releases built with different engines: Okapi, translated with ChatGPT, and Global-MMLU, translated with Google Translate and partly corrected by human editors.

The first step was a structural audit, run before any translation quality metric touched the text. It found that 327 of the 10,042 English HellaSwag validation originals, 3.26%, produced at least one target-language translation with empty answer options, meaning the multiple-choice item had a blank where a choice should be. Of those 327 broken originals, 257 (78%) broke in two or more of the 20 languages at once, not just a single outlier language. German lost 63 HellaSwag validation items outright to missing content, Spanish, French and Italian lost 4 each. Separately, 6,420 non-English ARC train samples were flagged for split inconsistency, and 8 ARC train samples violated the audit's cross-language coverage rule.

This matters because every quality number published afterward, COMET scores, win rates, downstream LLM accuracy, inherits whatever the structural layer missed. A model benchmarked on a blank-option HellaSwag item in German is not being tested on commonsense reasoning. It is being tested on a data pipeline's willingness to translate text that should never have reached a model in the first place.

Clean rates fall from 72.6% to 19.3% depending on which benchmark gets translated

After the structural pass, the authors ran reference-free xCOMET-XXL scoring and a span-level error analysis judged by a majority vote of three LLM annotators (GPT-4o-mini, Llama-4 Scout and Mistral-Large-Instruct-2411), pooled across all 20 languages and tallied per 1,000 items. The five benchmarks separated sharply.

The mirror statistic tells the same story from the other side: the share of accuracy-and-mistranslation errors that three annotators agreed were major, not minor, ran 87.6% for HellaSwag, 83.4% for GSM8K, 81.9% for MMLU, 75.4% for TruthfulQA and 73.9% for ARC. Reference-free xCOMET-XXL medians lined up the same way, around .97 to .98 for ARC and .42 to .65 for HellaSwag, depending on language.

A DeepL translation scored closer to human-edited gold than a ChatGPT one, in four of five languages tested

The audit also ran a reference-based comparison: it scored EU20 (DeepL) and Okapi (ChatGPT) against Global-MMLU's human-edited samples on the same MMLU items, in Italian, German, French, Romanian and Spanish, using COMET with the human-edited text as the reference translation. The delta (EU20 score minus Okapi score, on COMET's 0 to 1 scale, reported here as percentage points) favored EU20 in four of five languages: Italian +2.9, German +2.6, French +2.5, Romanian +1.5, and Spanish -0.4, a difference the paper reports as not statistically significant at that sample size (516 items, versus 2,342 for the other four languages).

The mechanism behind the gap is not that one MT engine understands culture and the other does not. It is that xCOMET-XXL scores correlate negatively with output length, with Spearman coefficients as steep as -0.75 for German HellaSwag, and HellaSwag's commonsense-completion items run the longest of the five benchmarks (median 112 to 183 words, against 27 to 44 for ARC). Longer, scenario-dependent text gives a literal engine more chances to mistranslate the specific cultural or situational detail that makes a commonsense answer obviously right or wrong to a native reader, and fewer chances for a reference-free metric to catch it cleanly.

What this means for someone doing this work

A reported multilingual score is only as good as the benchmark's translation layer. If a team cites an LLM's German or Romanian accuracy on HellaSwag without checking which release it came from, that number may reflect how well the model guesses around broken or mistranslated items, not how well it reasons in that language.

Longer, scenario-based content is where review budget should go first. The length-score correlation the paper measured is a targeting tool: a QA team with limited human review hours gets more value auditing long narrative or commonsense content than short factual lookups, because that is where literal translation fails first and most severely.

Engine choice is not a settled question. DeepL's EU20 beat ChatGPT's Okapi against human-edited reference text in four of five languages tested, but the fifth (Spanish) showed no significant difference, and the paper only tested five of the 20 EU20 languages this way. A single default MT vendor choice, applied without per-language checking, risks baking in a quality gap nobody measured.

Automated QA is a triage layer, not a substitute for human sign-off. The authors are explicit that their pipeline is built to prioritize what a human reviewer looks at next, not to certify a translation as final. A clean rate of 19.3% for HellaSwag is not a number you fix by raising a confidence threshold; it is a number that tells you most of that benchmark still needs a person to read it.

Check it yourself

The paper and its bibliographic record are both public. Pull the abstract, the full HTML with every table, and the ACL Anthology bibliographic entry directly:

# arXiv abstract page (submitted 2 April 2026, arXiv:2604.01957)
curl -s https://arxiv.org/abs/2604.01957 | grep -i "EU20"

# full HTML version with all tables (structural audit, COMET, span-level counts)
curl -s https://arxiv.org/html/2604.01957v1 -o eu20_full.html

# LREC 2026 Anthology record: pages 9030-9043, DOI 10.63317/46mkktmq3ytw
curl -s https://aclanthology.org/2026.lrec-1.710/ | grep -i "doi"

# PDF for offline reading
curl -sL https://arxiv.org/pdf/2604.01957 -o eu20_quality_audit.pdf

Read Table 4 and Table 5 in the full HTML version for the per-benchmark CLEAN and major-error percentages cited above, and the reference-based delta table for the DeepL-versus-ChatGPT comparison against human-edited Global-MMLU text.

What would prove this wrong

This finding would be undercut if, by the next LREC or EMNLP cycle (EMNLP 2026 runs in Budapest, with the WMT 2026 shared task's submission deadline already passed on 30 July 2026), an independent group reruns the same three-step audit on the EU20 authors' own released cleaned dataset and finds HellaSwag's clean rate has not meaningfully risen above roughly 20%, despite the fixes. It would also be undercut if a replication using a different trio of LLM span-judges (instead of GPT-4o-mini, Llama-4 Scout and Mistral-Large-Instruct-2411) produces clean rates for HellaSwag within the same range as ARC, since that would suggest the gap is an artifact of which models were used to grade spans rather than a property of the translated text itself. Either result would be checkable from the authors' own released code and corrected datasets, cited above.

FAQ

What is transcreation and how is it different from translation?

Translation maps source sentences to target sentences as closely as meaning allows. Transcreation goes a step further and rebuilds content so it lands the same way for a native reader, which can mean swapping an example, an idiom, or an entire scenario rather than translating it word for word. The EU20 audit is effectively a measurement of what happens when that second step is skipped: HellaSwag, which depends on recognizing a familiar everyday scenario to pick the sensible ending, comes out only 19.3% clean after literal MT into 20 languages, the lowest of the five benchmarks tested, while ARC, which is mostly short factual science questions with less cultural framing, comes out 72.6% clean.

Why do AI translations fail on cultural nuance?

Because the metrics and engines used to grade and produce machine translation are built to reward fluency and local sentence-level meaning, not whether a scenario still reads as sensible to someone from a different everyday context. The EU20 paper found that xCOMET-XXL scores fall as output length rises (Spearman correlations as negative as about -0.75 for German HellaSwag), and that lower scores track with a higher share of span-level accuracy and mistranslation errors (up to 87.6% of flagged HellaSwag errors were rated major). Long, scenario-dependent text is exactly where a literal engine runs out of room to carry cultural context across.

How do you keep human review in an AI localization pipeline?

The EU20 authors use automated checks to decide what a person should look at, not to replace that person. Their pipeline runs a structural audit first (catching things like the 327 HellaSwag items with empty answer options, before anyone reads translation quality into them), then COMET profiling across multiple MT engines to rank languages and datasets by risk, then a three-LLM-annotator majority vote on span-level error type. The output is a prioritized, pre-labeled review queue, not a verdict, and the paper's own framing is that automated QA helps a human reviewer decide where to spend limited time, while the human sign-off remains what the authors call the gold standard.

Sources

1. Klaudia Thellmann, Bernhard Stadler, Michael Färber, "Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite," arXiv:2604.01957, submitted 2 April 2026. arxiv.org/abs/2604.01957
2. Full text with all data tables (structural audit counts, pooled CLEAN/error rates, reference-based deltas). arxiv.org/html/2604.01957v1
3. ACL Anthology record, Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 9030 to 9043, DOI 10.63317/46mkktmq3ytw, published May 2026. aclanthology.org/2026.lrec-1.710