Six LLM judges scored a machine above the human reference on translations the machine failed

A 300-author benchmark landed on 3 September, built specifically out of the sentences models get wrong

3 September 2026. arXiv:2609.04173, core authors Vilém Zouhar (ETH Zürich), Niyati Bafna (Johns Hopkins), Mukund Choudhary (MBZUAI), Maike Züfle (KIT), Sara Rajaee (University of Amsterdam) and Pinzhen Chen (Queen's University Belfast), with more than 300 named contributors. The dataset is CC BY 4.0 at hf.co/datasets/zouhar/last-translation-benchmark, the code is MIT at github.com/zouharvi/last-translation-benchmark.

LTBv1 collected submissions from May 2026 to 1 September 2026. A submission is accepted only if it is translatable, meaning the contributor's own reference translation passes the rules, and difficult, meaning most automatic translations fail at least one rule. That admission test is what makes the numbers below readable: the benchmark is a set of known-solvable problems that models do not solve.

The shape of the data: 3,456 accepted examples, 109 main languages, 94% text, averaging 19 words or 104 characters. Each example carries 1.9 verification rules on average and 10% carry explicit translation instructions. Directionality runs 73% into English, 14% out of English and 13% between two non-English languages. Excluding English, 30.5% of the language mass is ultra-high-resource, 28.3% high, 19.0% medium, 3.2% low, 15.2% minimal and 3.8% zero-resource. Telugu contributes 299 examples, more than Chinese at 288, and Luxembourgish 74, more than Dutch at 75 minus one.

An evaluation subset, LTBv1-eval, holds 911 text-only examples chosen for difficulty, output diversity and balance across language pairs. Every number in this note is from that subset, averaged across it, excluding examples a given model cannot attempt for lack of language support.

The verifier and the judge rank the same translations in opposite orders

Table 1 takes ten of the 29 systems from the benchmark's main results table and puts four evaluation methods side by side on identical outputs. "Verifier" is the benchmark's official measure: the percentage of examples where the translation satisfies every verification rule, adjudicated by Qwen 3.7 Plus. "Judge" is the same model asked for a generic quality score. MetricX QE 24 and Comet QE 22 are reference-free neural metrics. ChrF is character n-gram overlap against the human reference, which is why the human scores 100 on it by construction.

Read the top two rows. The verifier separates the human from the best machine by 57.9 points. The judge reverses the sign and gives the machine a 5.5 point lead. Both numbers describe the same translations of the same sentences.

Then read the Google Translate row against the human row. MetricX QE 24 scores Google Translate 82.8 and the human 78.9. Comet QE 22 scores Google Translate 69.8 and the human 60.3. Google Translate satisfied every verification rule on 1.6% of examples. Both reference-free metrics preferred it to a translation that was correct 99.8% of the time. The reference-based versions of the same metrics behave sanely, MetricX 24 giving the human 93.5 against Gemini 3.1 Pro's 83.2, but a reference is exactly what a production QA gate does not have.

The benchmark also measures how much you can trust each method to give you a stable answer. Table 2 reports average pairwise Kendall tau-b between the system rankings that different evaluator models produce, within a method and against human annotators.

The judge is reasonably self-consistent, 0.713, and almost uncorrelated with what people actually prefer, 0.349. That combination is the dangerous one. A metric that is noisy announces itself. A metric that is confidently and repeatably wrong does not.

Self-preference makes it worse. Measured as the gap between a model's ranking of itself and the average ranking other models give it, Gemma 4 as a judge boosted itself by 28.2%, Qwen 3.7 Flash by 20.2%, GPT-5.4 Mini by 15.5% and Gemini 3.5 Flash Lite by 15.1%. Used as rule verifiers the same models' self-bias fell to 8.9%, 6.9%, 8.5% and 5.9%. Gemini 3.1 Pro is the one model that ranked itself below where others put it, by 8.3% as a judge and 8.9% as a verifier.

A rule is a question with a right answer, and a judge score is not

The difference is not model quality. It is what the evaluator is asked. A judge is asked "how good is this translation", which is a preference elicited with no privileged information. A verifier is asked "does this translation resolve the speaker's gender as feminine, as the source requires", which is a fact the contributor established when they wrote the example.

The benchmark tests this directly. Averaged across the set, LLMs translating unaided passed 7.2% of examples. Given the human-written verification rules in the prompt before translating, the same models passed 89.8%. Asked to write their own verification rules first, then translate, they reached 12.9%, which the authors attribute mostly to longer reasoning rather than insight.

That 7.2 to 89.8 jump is the finding underneath the finding. The models can satisfy these constraints. They cannot work out that the constraint is there. And a judge is a model asked to work out that the constraint is there, which is why judge scores sit in the 80s on translations that are wrong.

What is hard, concretely, from the taxonomy the authors built inductively over the whole set and then scaled with an LLM annotator:

The top of that list is ordinary localization work: metaphor at 1,086, cultural artifact at 986, polysemy at 948, variant specifics at 709, conventions at 480, slang at 453. These are not exotic. They are Tuesday for anyone who localizes marketing copy. Under the aggregated levels, 3,134 examples are semantic or lexical and 1,387 are pragmatic. The benchmark also logs 358 model blockers: 223 irrelevant outputs, 49 incomplete, 44 refusals, 39 tokenization failures and 3 instruction injections, where the model did something other than translate.

What this changes for anyone running an automated localization QA gate

If your pipeline is model translates, model scores, ship above threshold, the numbers above say the second model is not doing the job you think. On content selected to be hard, the judge's ranking correlates with human preference at 0.349 and the judge prefers whichever output looks fluent. Fluent and wrong is the exact output distribution of a strong LLM translator on a hard sentence.

Three things follow that are worth doing rather than worrying about.

Write rules, not rubrics. The operational version of a verification rule is the thing a good localization brief already contains: this product name is not translated, this pronoun is feminine, this pun has to survive, this measurement converts to metric. Those are checkable by an LLM with high self-agreement (0.869) precisely because they have answers. A five-point fluency rubric does not. The benchmark's own contributors wrote 1.9 rules per example and that was enough to move self-bias from 28.2% to 8.9% on Gemma 4.

Stop using reference-free metrics as a release gate. Comet QE 22 and MetricX QE 24 both ranked a system with a 1.6% pass rate above a 99.8% human. Reference-free is the deployment-friendly configuration, and it is the one that fails here. If you keep a neural metric, keep it as a regression detector against a fixed reference set, not as an absolute quality bar.

Budget human review by difficulty, not by volume. The difficulty taxonomy is a sampling frame. Segments carrying metaphor, cultural artifact, polysemy or variant specifics are where the 7.2% unaided pass rate lives. Segments that are plain declarative product copy are not. Reviewing 5% of a corpus chosen by those labels buys more than reviewing 5% at random, and it is the same reviewer cost.

A judgement, marked as one: the 89.8% figure is the commercial argument for human-in-the-loop localization for the next two years. It says the value a linguist adds is no longer producing the target sentence. It is knowing, before anyone translates, which sentences carry a trap. That knowledge is writable down, it transfers to the model at inference time, and nobody currently sells it as a product.

Check it yourself

The dataset, the rules and the model outputs are all public. Reproducing the central claim takes one download and a few lines.

# the benchmark, CC BY 4.0
pip install datasets
python3 - <<'PY'
from datasets import load_dataset
d = load_dataset("zouhar/last-translation-benchmark")
print(d)
ex = d["train"][0] if "train" in d else next(iter(d.values()))[0]
print(ex.keys())          # source, reference, verification rules, difficulty labels
PY

# the code, MIT
git clone https://github.com/zouharvi/last-translation-benchmark
# the live leaderboard and the submission form
open https://last-translation-benchmark.vilda.net

The claim to check first is the inversion in Table 1. Take any example whose difficulty label is metaphor or variant specifics, read the verification rules, read a model output, and decide for yourself whether it is an 87. Then ask an LLM for a 0 to 100 quality score on the same pair without showing it the rules. The paper's Prompt 2 (verification) and Prompt 3 (generic judging) are in its appendix, so the comparison is exactly reproducible.

Two things to watch when you read the tables. The main results table has four column blocks (verifier, judge, metric, human), each block repeating across evaluator models, so a bare number is meaningless without its column. And the human row scores 100 on ChrF because the human translation is the reference that ChrF measures against, which is why this note reports the two reference-free metrics instead.

What would prove this wrong

The claim under test is that generic LLM judging is structurally blind on hard translation, not merely behind. It is wrong if, by 1 September 2027, an LTBv2 or later release reports a generic LLM judge whose ranking agrees with human annotators at Kendall tau-b of 0.70 or better, without the judge being shown verification rules. The 2026 figure is 0.349 against the verifier's 0.905. Closing that in a year would mean the judges learned to detect difficulty they currently cannot name.

A second prediction, marked as judgement: the top verifier pass rate on LTBv1-eval will not exceed 70% before 1 September 2027 for any blind-mode submission. It stands at 41.9%. The oracle-mode number of 89.8%, where the model is handed the rules, is not a counterexample, because it is measuring a different thing. If a blind submission crosses 70%, the argument in this note about where human value sits needs rewriting.

FAQ

Can you use an LLM as a judge to sign off on translation quality?

Not on hard content. All six judges tested on the Last Translation Benchmark scored Gemini 3.1 Pro above the human reference, on examples selected because leading models fail them. Judge rankings agreed with human annotators at Kendall tau-b 0.349, against 0.905 for the rule-based verifier.

Do reference-free metrics like COMET-QE detect hard translation errors?

Not here. MetricX QE 24 scored Google Translate 82.8 and the human reference 78.9. Comet QE 22 scored Google Translate 69.8 and the human 60.3. The verifier put Google Translate at 1.6% and the human at 99.8%.

What is the Last Translation Benchmark?

A live crowdsourced benchmark released 3 September 2026 (arXiv:2609.04173) from ETH Zürich, Johns Hopkins, MBZUAI, KIT, the University of Amsterdam and Queen's University Belfast with more than 300 contributors. LTBv1 holds 3,456 peer-reviewed examples across 109 languages, each with an average of 1.9 verification rules. A translation scores only by passing every rule.

Do models fail because they cannot see the problem or because they cannot solve it?

Mostly because they cannot see it. Unaided pass rate 7.2%. Given the human-written rules in the prompt, 89.8%. Asked to generate their own rules first, 12.9%.

Which content is most at risk in an automated localization pipeline?

The benchmark's difficulty labels point at metaphor (1,086 examples), cultural artifact (986), polysemy (948), language variant specifics (709), conventions (480) and slang (453). Marketing copy, UI microcopy with regional conventions, and anything with a joke in it.

Sources

1. Vilém Zouhar, Niyati Bafna, Mukund Choudhary, Maike Züfle, Sara Rajaee, Pinzhen Chen and more than 300 contributors, Last Translation Benchmark, arXiv:2609.04173, submitted 3 September 2026. Dataset statistics (3,456 examples, 109 languages, 1.9 rules per example, resourcedness and direction mix); Main Table 2 (29 systems across verifier, judge, metric and human evaluation); Table 3 (7.2% unaided, 12.9% with self-generated rules, 89.8% with human rules); Table 4 (Kendall tau-b stability); Table 5 (evaluator self-bias); Table 6 (difficulty taxonomy). PDF.
2. zouhar/last-translation-benchmark on Hugging Face, CC BY 4.0. The released LTBv1 and LTBv1-eval splits.
3. zouharvi/last-translation-benchmark on GitHub, MIT. Collection platform, verification prompts and scoring code.
4. The benchmark's live leaderboard and submission platform, last-translation-benchmark.vilda.net, which distinguishes blind mode from oracle mode.
5. Slator, SlatorPod 270: AI Translation State of the Art with Tom Kocmi and Alon Lavie, 21 November 2025. Industry discussion of the WMT25 results the benchmark's authors cite as evidence of saturation.

Related BLOMEGA guides: Multilingual LLM judges and translationese bias · Omnilingual MT by resource tier · Consented AI training data providers.