Your multilingual LLM judge prefers the machine translation, and agrees with itself at kappa 0.24

What changed between May 2025 and July 2026

LLM-as-a-judge became the default sign-off in multilingual pipelines somewhere in 2025, because it is the only quality signal that scales to 40 locales without 40 review teams. Four papers since then have measured whether it works outside English, and the answers arrived in a specific order.

18 May 2025. Xiyan Fu and Wei Liu posted How Reliable is Multilingual LLM-as-a-Judge? (arXiv:2505.12201, later Findings of EMNLP 2025). Five judges, GPT-3.5-turbo, GPT-4o-2024-08-06, Llama-3.3-70b, Qwen-2.5-72b and Aya-expanse-32b, scored five task sets covering 25 languages: XQuAD (11 languages, 1,191 samples), MGSM (10, 250), WMT23 (8, 196), WikiLingua (20, 142) and XDailyDialog (4, 996). The measure is cross-lingual Fleiss kappa: give the same judge the same item in different languages and ask how often it reaches the same verdict.

11 March 2026. Hongbin Zhang, Kehai Chen, Xuefen Bai, Youcheng Pan, Yang Xiang, Jinpeng Wang and Min Zhang posted Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck (arXiv:2603.10351). They name the failure mode: judges "systematically favouring machine-translated text over human-authored references, particularly in low-resource languages." They also give it a number, which nobody had before.

27 May 2026. Irune Zubiaga, Aitor Soroa and Rodrigo Agerri of the HiTZ Center posted Towards Reliable Multilingual LLMs-as-a-Judge (arXiv:2605.28710), fine-tuning judges on English, Spanish and Basque and testing them in and out of domain. The in-domain numbers look fine. The out-of-domain numbers are the finding.

2 July 2026. A. Seza Doğruöz, Xixian Liao, Verena Blaschke, Jakob Prange, Senyu Li and David Ifeoluwa Adelani posted Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages (arXiv:2607.02235), a survey of the field's own practice. Of 650 papers mentioning LLM-as-a-judge, 33 met their inclusion criteria for low-resource or multilingual work. Of those 33, three deployed a judge on low-resource languages with no human or gold-label validation at all.

None of this is a claim that the judges are useless. It is a claim about where the error lives, and it lives in the languages you cannot read.

How much do the judges agree with themselves across languages?

Fu and Liu report Fleiss kappa per model per task under a Yes/No criterion. We took their 25 published values and computed the per-task mean across the five judges, which they do not print. Every task mean sits below 0.41, the floor Landis and Koch (1977) set for moderate agreement, and three of five sit inside the 0.21 to 0.40 fair band.

Two details in that grid matter more than the averages. Qwen-2.5-72b, one of the two strongest judges overall at a row mean of 0.273, is the single worst judge on translation at 0.0775. Judge quality does not transfer across task types, so a judge you validated on summarization tells you nothing about the same judge on a dub script. And Fu and Liu report a Cohen's kappa of 0.002 between Telugu and English for Llama-3.3 on MGSM. That is not a weak signal. That is the absence of one.

The survey published on 2 July 2026 puts a second number on the same problem from a different angle. It cites Fu and Liu as reporting an average Fleiss kappa of approximately 0.3 across 25 languages. Our computed mean over the published Yes/No table is 0.241. The gap is most likely because the survey averages across both the Yes/No and the Grade criteria and we only have the Yes/No values in front of us. We could not verify which. Treat 0.24 as the figure for binary pass/fail sign-off, which is what a release gate actually is.

Where the bias enters a localization QA pipeline

The bias metric in the March 2026 paper is worth understanding precisely, because it is cheap to reproduce on your own stack. Bias severity, written S_bias, is the fraction of order-consistent judgments that favour the machine-generated output. Order-consistent means the judge gave the same verdict when the pair was shown forward (A then B) and reversed (B then A), which filters out position bias and leaves you with the judge's actual preference. A judge that reliably prefers the human-authored reference scores near 0. A judge flipping a coin scores about 0.5.

In the proxy-task ablation, the variant trained without the consistency proxy tasks scores S_bias 0.421 at accuracy 87.12. Adding the two proxy tasks moves it to 0.147 at accuracy 89.18. The full DIBJudge configuration reaches 0.031 at accuracy 89.85. Read the first number as the state of an ordinary fine-tuned judge with no bias supervision: high accuracy, and a preference for the machine that is closer to a coin flip than to correct. That is our reading of their ablation, marked as a judgement, not their claim.

The Basque study explains why in-house validation usually misses this. Zubiaga, Soroa and Agerri fine-tuned judges and measured Pearson correlation against human scores on their in-domain RECON benchmark and on out-of-domain FLASK. In domain, the best multilingual configuration reaches r 0.836 English, 0.816 Spanish, 0.805 Basque, a spread of 0.031 across a high, a mid and a low-resource language. Move to FLASK and the same approach gives r 0.440 for English at 70B fine-tuned and 0.336 for Basque. Fine-tuning actively hurt the larger model out of domain: Latxa 70B scored r 0.594 zero-shot on FLASK English and 0.440 after fine-tuning, a loss of 0.154.

MM-Eval gives the same shape from the reward-model side. Across its five core subsets and 18 languages, the average judge accuracy is 68.9% against a 50% random baseline, with nine models below 70% and Safety the hardest subset, where most models score below or near random. Its language consistency subset spans 122 languages, and there proprietary models land near or above 0.8 on the consistency index while open-source models struggle to exceed 0.6. The paper's description of the low-resource pattern is the operationally important one: as resource level falls, the score gap between the chosen and the rejected response narrows across all models. The judge does not stop scoring. It stops discriminating.

What this means if you run a localization QA gate

Four decisions follow, in the order you can make them.

Stop treating one judge score as a per-language gate. The July 2026 survey found 16 of 33 qualifying papers relied on a single judge model family and 11 of 33 used GPT alone. Fu and Liu's ensemble of Llama-3.3-70B, Qwen-2.5-72B and Aya-expanse-32B by majority vote improved Fleiss kappa over the worst single model by +0.2479 on XQuAD, +0.1892 on WikiLingua and +0.1628 on XDailyDialog under the Yes/No criterion. It did not help everywhere: WMT23 moved -0.0046 on Yes/No and +0.0644 on Grade. An ensemble is cheap insurance on most tasks and does nothing measurable on translation, which is the task you care about.

Validate per language, not per pipeline. A judge validated on English and deployed on 30 locales is an English-validated judge running unmeasured 29 times. The in-domain Basque number, 0.805, looks fine right up until you leave the benchmark. Ask for the out-of-domain number, and if the vendor does not have one, that is the answer.

Budget human review by resource tier, not evenly. The evidence points the same direction three times: bias severity rises as resource level falls, the chosen-rejected score gap narrows as resource level falls, and out-of-domain correlation drops furthest for the low-resource language. If you sample 5% of output for human review in every locale, you are over-sampling the locales the judge handles well. Weight the sample toward the tiers where the judge stops discriminating.

Do not use judge scores to select training data. This is the compounding failure. If a judge that prefers machine text in 42.1% of order-consistent comparisons is used to filter or rank candidate outputs for a training set, it selects machine-flavoured target-language text and drops the human-authored text. The next model trains on it. Global MMLU already showed what translated-in evaluation does to rankings: 28% of MMLU questions require culturally sensitive knowledge, 84.9% of its geography questions are North American or European, and model rankings shift when scored on the culturally sensitive subset rather than the full set. Filtering with a biased judge is how translationese becomes the target language in the weights.

This is the practical argument for buying in-language human data rather than translating your way into a locale, and it is a narrower argument than the usual one. It is not about quality in the abstract. It is that the only instrument you have for detecting the difference is measurably worst at exactly that comparison.

Check it yourself

You can measure S_bias on your own judge in an afternoon. You need pairs where one side is machine-translated and the other is human-authored in the same target language. BELEBELE covers 122 languages and is the source the March 2026 bias set was derived from.

pip install datasets

# 1. pull a parallel, human-authored multilingual set
python - <<'PY'
from datasets import load_dataset
for lang in ["eng_Latn", "swh_Latn", "tel_Telu", "npi_Deva"]:
    d = load_dataset("facebook/belebele", lang, split="test")
    print(lang, len(d), d[0]["flores_passage"][:80])
PY

# 2. build pairs: human-authored target passage vs your own MT of the English passage
#    (use whatever engine your pipeline actually ships)

# 3. score each pair TWICE with your production judge, swapping the order:
#    prompt A: "Which is better written, Response 1 or Response 2?"  (human first)
#    prompt B: identical, machine first
#    keep only the pairs where the judge gives the SAME winner both times

# 4. S_bias = (order-consistent judgments favouring the MACHINE) / (order-consistent judgments)
#    0.00 = always prefers the human text   0.50 = coin flip   1.00 = always prefers MT

Run it per language and sort by resource tier. Three numbers are worth writing down: your S_bias per tier, your order-consistency rate (the fraction of pairs where swapping the order did not change the verdict, which is a position-bias measure in its own right), and the same figures for a second judge from a different model family. If your low-resource S_bias sits anywhere near 0.421, your gate is not gating.

Primary texts worth reading rather than summaries of: arXiv:2505.12201 Table 1 for the raw kappa grid; arXiv:2603.10351 section 3 for the S_bias definition and the 30-language bias set; arXiv:2607.02235 for the four recommendations and the coverage statistics on the field's own practice; and the MM-Eval repository for the 18-language and 122-language subsets.

What would prove this wrong

The claim under test is that a single LLM judge cannot be relied on as the release gate for a low-resource locale, because its cross-lingual consistency sits in the fair band and it measurably prefers machine-translated text to human-authored text in that tier. It is wrong if, by 1 September 2027, a published evaluation shows a general-purpose judge reaching cross-lingual Fleiss kappa above 0.61, the Landis and Koch substantial-agreement floor, on a translation task across at least 10 languages including at least 3 low-resource ones, with S_bias below 0.10 in the low-resource tier, measured out of domain rather than on the benchmark it was tuned on. Today the best published translation-task figure in Fu and Liu's grid is 0.1691, and the best low-resource out-of-domain correlation in Zubiaga's grid is r 0.336.

A second prediction, marked as judgement rather than finding: the bias-supervised judges will get published numbers below S_bias 0.05 well before anyone publishes a general-purpose judge that clears 0.61 kappa on translation, because bias supervision is a training objective and cross-lingual consistency is not. If that holds, the practical answer for 2027 is a purpose-trained evaluator per language family plus human sampling, not a better frontier model.

FAQ

How do you evaluate an AI model's quality across languages?

Not with a single LLM judge scoring the target language, if you want the number to mean anything. Across the 25 Yes/No Fleiss kappa values Fu and Liu published for five judges on five tasks, the mean is 0.241, inside the 0.21 to 0.40 fair band and below the 0.41 moderate floor. The worst task mean is machine translation at 0.131. Use per-language validation against in-language human raters, an ensemble rather than a single judge, and a reference-based metric where one exists.

Do LLM judges prefer machine translation over human writing?

In measured conditions, yes, and more so as the language gets poorer in data. Bias severity, the fraction of order-consistent judgments favouring the machine output, is 0.421 in the no-proxy ablation of arXiv:2603.10351 (11 March 2026) against 0.031 for their best configuration, on a 30-language set of 200 instances per language derived from BELEBELE. Per-tier numeric values for GPT-4o are not reported in that paper.

Why do AI translations fail on cultural nuance?

Partly because the evaluation does not test for it. Global MMLU found 28% of MMLU questions require culturally sensitive knowledge and 84.9% of geography questions target North America or Europe, and that model rankings shift on the culturally sensitive subset. A benchmark translated out of English carries the source culture with it, so a model that fails cultural adaptation can still score well.

How do you keep human review in an AI localization pipeline?

Validate the judge per language instead of assuming English validation transfers, keep humans scoring a documented subset, document the evaluator population and its competencies, and prefer a non-LLM reference-based metric where one is available. Those are the four recommendations in arXiv:2607.02235 (2 July 2026), whose survey found 24 of 33 qualifying papers reported some human comparison and 3 deployed a judge on low-resource languages with none.

Does a bigger model fix this?

Not on the evidence so far. Fu and Liu report that neither multilingual training nor model scale directly improves judgment consistency, and MuBench (ACL 2026 Findings, 61 languages, 3.9M samples) reports that increasing model size does not improve handling of mixed-language contexts. In the Basque study, fine-tuning made the 70B judge worse out of domain, r 0.594 zero-shot down to 0.440.

Sources

1. X. Fu, W. Liu, How Reliable is Multilingual LLM-as-a-Judge?, arXiv:2505.12201, 18 May 2025; Findings of EMNLP 2025. Five judges, five tasks, 25 languages; the Fleiss kappa grid in Table 1; Cohen's kappa 0.002 Telugu vs English for Llama-3.3 on MGSM; the majority-vote ensemble deltas.
2. H. Zhang, K. Chen, X. Bai, Y. Pan, Y. Xiang, J. Wang, M. Zhang, Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck, arXiv:2603.10351, 11 March 2026. Definition of bias severity; the 30-language, 200-instance-per-language bias set derived from BELEBELE; ablation values 0.421, 0.147 and 0.031.
3. I. Zubiaga, A. Soroa, R. Agerri (HiTZ Center, University of the Basque Country), Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study, arXiv:2605.28710, 27 May 2026. In-domain RECON correlations 0.836 / 0.816 / 0.805; out-of-domain FLASK 0.440 English and 0.336 Basque; the 0.154 fine-tuning loss on Latxa 70B.
4. A. S. Doğruöz, X. Liao, V. Blaschke, J. Prange, S. Li, D. I. Adelani, Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages, arXiv:2607.02235, 2 July 2026. Survey of 650 papers, 33 included; judge-family concentration statistics; the four recommendations; citations to Fu and Liu, Watts et al. and Hada et al.
5. G. Son, D. Yoon, J. Suk, J. Aula-Blasco, M. Aslan, V. T. Kim, S. B. Islam, J. Prats-Cristà, L. Tormo-Bañuelos, S. Kim, MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models, arXiv:2410.17578, October 2024, revised March 2025, ICLR 2026. Average accuracy 68.9% against a 50% baseline; the Safety subset result; the narrowing chosen-rejected gap by resource level; language consistency index across 122 languages.
6. S. Singh, A. Romanou, C. Fourrier, D. I. Adelani et al., Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation, arXiv:2412.03304, 4 December 2024, revised 19 February 2025. 28% culturally sensitive questions; 84.9% North American or European geography questions; 42 languages.
7. W. Han, Y. Zhang, Z. Chen, Binbinliu, M. Pechenizkiy, M. Fang, Y. Zheng, MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages, Findings of ACL 2026, July 2026. 61 languages, 3.9M samples, 34k human-expert-assessed samples across 17 languages; model size does not improve mixed-language handling.
8. J. R. Landis, G. G. Koch, The measurement of observer agreement for categorical data, Biometrics 33(1), 1977. The kappa interpretation bands used here: 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial.
9. MM-Eval repository and the BELEBELE dataset. The subsets and the 122-language parallel data used in the reproduction steps.

Related BLOMEGA guides: Localization is the new default · Article 50 applies to your dub · Consented AI training data providers · Data provenance and chain of title.