An LLM translation judge scored wrong-dialect Arabic as a perfect match, every human scored it zero

What changed, and when

On 7 October 2026, Hala Almaghout, Christian Federmann and Qin Gao at Apple posted Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged (arXiv:2610.10918v1). The paper measures whether four current LLMs, used as MT quality judges, agree with human annotators as well as human annotators agree with each other, across two evaluation schemes: Multidimensional Quality Metrics (MQM, scored from counted error spans) and Error Span Annotation (ESA, scored directly).

The test sets are a long-context 70-language-pair set, the public WMT23 MQM set, and the public WMT25 ESA set. The four LLM judges are Gemini 2.5 Flash, GPT-OSS-120B, DeepSeek-V3.1 and Qwen3.5-397B, each run with reasoning disabled, scored against the same human annotation rounds.

The headline result is not that LLMs lose uniformly. On some language pairs and domains, LLM-human agreement is statistically indistinguishable from, or even exceeds, human-human agreement. The paper's point is that this is unpredictable across schemes, language pairs and domains, which is what makes it unsafe to rely on in general. Egyptian Arabic is the sharpest single failure the paper documents, and it has a traceable cause.

The evidence table

First, the span-annotation agreement that both humans and LLMs reach with the human annotation rounds, on the WMT23 MQM test set, averaged across all language pairs. H-H is human round against human round; the other columns are each LLM against the human rounds, both measured as F1 on matching error spans.

Reproduced from Table 1 of arXiv:2610.10918v1, Section 4.1.2, averaged over all language pairs on the WMT23 MQM test set. * marks a difference from human-human (H-H) agreement significant at p<0.05. Categories with fewer than 5 source segments backing the H-H baseline are omitted.

Human-human agreement on span overlap tops out at F1 = .51, the ceiling for this task even between two trained humans. Every LLM trails it on almost every category, and the gap widens sharply for low-frequency categories: several LLMs score 0.00 F1 on Untranslated Text, Non-translation and Source Error. The one place an LLM matches humans is Grammar, where Gemini 2.5 Flash scores .19 against human .18, not significantly different.

Second, the count of WMT25 ESA language pairs, out of 44 tested, where agreement differs significantly between LLM-human and human-human Krippendorff's alpha.

Reproduced from Table 2 of arXiv:2610.10918v1, Section 4.2.1, WMT25 ESA test set, 44 language pairs across 5 domains, Benjamini-Hochberg FDR correction per LLM, p<0.05.

The Egyptian Arabic failure has a traceable cause

Apple's team did not stop at the aggregate number. For English-to-Egyptian Arabic (en-arz_EG), the agreement pattern inverts: human-human Krippendorff's alpha is above 0.90 on every one of the five domains tested (dialogue, education, literary, news, social), the highest of any language pair in the study, while LLM-human alpha for this pair goes negative, meaning the LLMs disagree with humans more than chance would predict.

Manual review of the annotations found the cause. Roughly 60% of the system translations sent for Egyptian Arabic evaluation are actually written in Modern Standard Arabic (MSA), a related but distinct variant. ESA's annotation instructions tell annotators to score any translation in the wrong language or variant zero, with a single error span, specifically to catch this. Human annotators did exactly that, consistently. The four LLM judges instead scored most of these mismatched translations high, because the text they were shown is fluent, grammatically correct Arabic. It is just the wrong Arabic for the task.

The paper calls this "a critical limitation in the LLMs' ability to distinguish between closely related language variants, a capability essential for locale-specific quality evaluation." It is not a one-off: the same models also measurably struggle with minor-severity errors and with rare MQM categories across every language pair tested, not only Egyptian Arabic. The common thread is that LLM judges are weakest exactly where the quality signal depends on something other than surface fluency.

What it means for anyone using an LLM as a translation-quality judge

An LLM judge needs a variant check it cannot provide itself. The failure here is not noise, it is systematic: the LLMs are consistently fooled by fluent-but-wrong-dialect text. A pipeline that routes MT quality scoring through an LLM judge without an explicit variant-identification step will silently pass translations that every human annotator would reject.

Aggregate correlation numbers hide this. A report that LLM-human correlation is "good on average" across 70 language pairs is compatible with exactly the pattern in Table 2: a handful of pairs where the LLM matches or beats humans, and dozens where it does not, averaging out to something that looks acceptable until you need the specific language pair that is actually bad.

Minor errors and rare categories are where the gap is widest, and that is where it matters most. The paper notes that as MT systems improve, minor errors become the main thing separating closely ranked systems. That is exactly the error type LLM judges are worst at identifying, which means LLM-as-judge evaluation gets less trustworthy as the systems it is judging get better.

Our judgement, specific to the consent and locale side of this: a quality pipeline for any dialect-sensitive language, Arabic variants, Portuguese variants, regional Mandarin, should treat variant identification as a separate, checkable gate before any LLM-as-judge score is trusted, not as something the judge is assumed to handle. This matches what we found in separate work on Arabic dialect sentiment benchmarks: dialect confusion inside a model is a recurring, measurable failure mode, not an edge case.

Check it yourself

The WMT23 MQM and WMT25 ESA test sets are both public, and the paper's Table 1, Table 2 and Figure 2 numbers come directly from them.

open https://arxiv.org/abs/2610.10918                 # paper and Table 1, Table 2, Figure 2
open https://wmt-metrics-task.github.io/               # WMT shared task pages, ESA and MQM data releases

# what to check without rerunning anything:
# Table 1 (WMT23 MQM, Section 4.1.2): H-H span-overlap F1 = .51, the human-human ceiling
#   every LLM column trails it; several LLMs hit 0.00 F1 on rare categories
#   (Untranslated text, Non translation, Source error)
# Table 2 (WMT25 ESA, Section 4.2.1): count language pairs per LLM where
#   LLM-human alpha significantly beats human-human, vs the reverse, out of 44 pairs
# Section 4.2.1, en-arz_EG: human-human alpha > 0.90 on all 5 domains;
#   LLM-human alpha negative; ~60% of translations in the wrong Arabic variant,
#   estimated from items scored 0 with a single error span by a human annotator

python3 - <<'PY'
exceeds = {"Gemini 2.5 Flash": (8, 20), "Qwen3.5-397B": (6, 23),
           "DeepSeek-V3.1": (3, 28), "GPT-OSS-120B": (3, 32)}
for name, (llm_wins, human_wins) in exceeds.items():
    ratio = human_wins / llm_wins
    print(f"{name:16s} human-human wins {human_wins:2d} pairs, LLM wins {llm_wins:2d}, {ratio:.1f}x more often")
PY
# Gemini 2.5 Flash  human-human wins 20 pairs, LLM wins  8, 2.5x more often
# Qwen3.5-397B      human-human wins 23 pairs, LLM wins  6, 3.8x more often
# DeepSeek-V3.1     human-human wins 28 pairs, LLM wins  3, 9.3x more often
# GPT-OSS-120B      human-human wins 32 pairs, LLM wins  3, 10.7x more often

The 60% wrong-variant estimate is the paper's own, computed by counting translations that a human annotator scored zero with exactly one error span, which is how ESA's instructions tell annotators to flag a wrong-language-or-variant translation. It is an estimate of a proxy, not a direct dialect classifier count, and the paper states it as such.

What would prove this wrong

This is an empirical result on specific LLM snapshots (Gemini 2.5 Flash, GPT-OSS-120B, DeepSeek-V3.1, Qwen3.5-397B) and specific test sets. It would be wrong, or at least dated, if a newer model generation closes the Egyptian Arabic gap.

A dated prediction: by 31 December 2027, a published replication using next-generation frontier models (successors to the four tested here) on the same or an equivalent en-arz_EG ESA test set will still show LLM-human Krippendorff's alpha below the human-human alpha on that pair, because the underlying issue, fluency not implying correct-variant, is a property of how these models represent dialect, not a transient capability gap. If a replication shows LLM-human alpha matching or exceeding human-human alpha on en-arz_EG specifically, this prediction is wrong.

Sources

1. Almaghout, H., Federmann, C., Gao, Q. (Apple). Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged. arXiv:2610.10918v1, 7 October 2026. Tables 1, 2, 3; Figures 1, 2; Sections 4.1, 4.2.
2. WMT (Conference on Machine Translation). WMT23 MQM and WMT25 ESA shared-task test sets, the public data this paper measures against.
3. BLOMEGA. Arabic dialect sentiment benchmark overlap and swap-copy baselines.
4. BLOMEGA. Data annotation research: the latest.