On real localisation post-edits, COMETKiwi picks the human fix 53.1% of the time, and a GPT-4.1 judge 40.7%

LILT measured quality estimation on its own post-edits, 16 September 2026

16 September 2026. Kathy Hämmerl (TU Munich and Munich Center for Machine Learning, work done during an internship at LILT), Gabriel Bretschner and Joern Wuebker (LILT) posted LocQE: Principled Domain Adaptation for Localisation Quality Estimation (title shortened here; the full title adds how post-edits are used), arXiv:2609.18720.

The data is what makes it unusual. The test set is LILT's own production history: technical documentation, UI strings, feature lists and other client content across 51 language pairs (16,633 test preference pairs), where each segment carries the machine translation and the version a human reviewer approved. Where the reviewer changed something, the pair becomes a preference test: a metric that understands localisation quality should score the approved version higher. The paper also releases two new evaluation sets, LocCheck (2,564 internal and 841 public minimal pairs across nine localisation rules) and LocHD (300 English UI and help segments translated into 10 languages and scored by 3 to 5 paid professional translators per language with Error Span Annotation).

The question it answers is the one every localisation QA pipeline built on automatic scoring rests on: does the score track what your reviewers fix?

On real post-edits, both off-the-shelf metrics sit at or below chance

Three rows deserve attention.

The GPT-4.1 judge is the worst segment-level ranker and the best global correlator. GEMBA-ESA scores 40.7% on preference accuracy but the highest Kendall τ on LocHD, 0.239 against COMETKiwi's 0.108. It places translations of different sources on a sensible scale and cannot tell two translations of the same source apart. For a QA gate that picks between candidate translations, only the second skill matters.

Training on preference pairs alone buys ranking by destroying calibration. MRL-only fine-tuning reaches the highest real-data preference accuracy in the paper, 66.9%, while LocHD Kendall falls to 0.044 and the ACES general-capability score collapses from 17.94 to 3.67. The authors show why in Figure 5: to reverse the many pairs the baseline got wrong, the model compresses all its scores into a band about twice the 0.02 margin wide.

The fix costs very little data. The multi-task LocQE model uses 10,000 continuous scores plus 5,000 preference pairs from post-edits, under 1% of COMETKiwi's original training data, and moves real-data preference accuracy from 53.1% to 59.6% while keeping Kendall at 0.135 and lifting WMT24 accuracy to 67.6%.

COMETKiwi cannot score a French non-breaking space because its tokeniser erases it

The paper's Table 1 gives three cases a translator would call trivially wrong. A French sentence with the required non-breaking space before "!" and the same sentence with a regular space both score 84.83. A German translation that preserves a cloze blank scores 78.89, while a version that fills the blank with an invented word ("lösen") scores 84.20, higher. A German sentence that changes 7777777 to 777777 drops from 85.91 to 85.57, a 0.34-point penalty for a wrong number.

We reproduced the tokeniser half of this on 19 September 2026 (commands below): with microsoft/infoxlm-large, "une sortie\u00a0!" and "une sortie !" produce identical token ids, and so does the same string with three trailing spaces. One detail the paper glosses over: in our test a Persian word containing a zero-width non-joiner did not produce the same ids as the word without it, so not every special character is invisible. Non-breaking and trailing spaces are.

The tokeniser change alone lifts LocCheck by 4 to 10 points (internal 45.1 to 49.5, public 30.1 to 35.8, Table 3). It is necessary for the NBSP rule and not sufficient: adding an untrained NBSP token first drops French NBSP accuracy from 42.0% to 8.0% until fine-tuning teaches the model what the token means.

If your QA gate is a QE score, it is not checking what your reviewers fix

Keep the regex checks. Numbers, URLs, do-not-translate tokens, locale punctuation and whitespace are deterministic rules. The paper's own conclusion is that "heuristics may equally remain a useful tool in the localisation pipeline", and its augmented model shows why: once trained to satisfy LocCheck rules at 87.9%, it fell back from 59.6% to 57.4% on real post-edit preferences. The authors read that as reviewers not applying the rules consistently. A checker applies them every time.

Do not use an LLM judge as a segment-level chooser without measuring it on your own edits. A 40.7% preference accuracy means that picking the higher-scored candidate with that judge would, on this data, pick against the translator more often than for them. It still ranked documents well globally (τ 0.239). Use it where it works.

Your post-edit history is metric training data. Ten thousand chrF++-scored post-edits and five thousand preference pairs moved the metric by 6.5 points on your own domain. Most localisation teams have that volume in a translation management system already. The paper's subtraction of 20 chrF points when a reviewer had to fix a number, URL or DNT is a simple way to encode "small edit, major error".

Report both kinds of correlation. The MRL-only row is the warning: the model that best picks between two translations was the worst at telling a good segment from a bad one across sources. A vendor quoting only one of the two is quoting the one that flatters them.

A judgement, marked as one: a human preference for the post-edit is itself noisy. LILT's validation study (Appendix A.4, 984 annotations) found annotators preferred the final human version over the MT 61% of the time, preferred the MT 14% of the time, and called the rest ties. So a perfect metric would not score 100% here. It would still score well above 53.1%.

Check it yourself

# 1. The tokeniser half of the NBSP failure (no model weights needed)
pip install transformers sentencepiece
python3 - <<'EOF'
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("microsoft/infoxlm-large")
a = "lorsque vous participez à une sortie\u00a0!"   # correct fr-FR
b = "lorsque vous participez à une sortie !"         # regular space
c = b + "   "                                          # trailing spaces
print(tok(a)["input_ids"] == tok(b)["input_ids"])   # True
print(tok(c)["input_ids"] == tok(b)["input_ids"])   # True
EOF

# 2. The score half (COMETKiwi-22 is gated: accept the licence on Hugging Face first)
pip install unbabel-comet
huggingface-cli login
comet-score -s src.txt -t mt.txt --model Unbabel/wmt22-cometkiwi-da
# src.txt: "These cargo pants are the easiest to dress when you are attending an outing!" (twice)
# mt.txt:  the NBSP and regular-space French lines. The paper reports 84.83 for both.

# 3. The measurement that matters for you: preference accuracy on your own edits
#    for each (mt, approved_post_edit) pair where they differ:
#       correct += score(src, post_edit) > score(src, mt)
#    report correct / n. 50% is chance. The paper's COMETKiwi baseline: 53.1%.

The paper points to github.com/lilt/loc-qe for the LocCheck generator and the LocHD data. On 19 September 2026 that URL returned HTTP 404, and no LocHD dataset appeared on the Hugging Face Hub. The internal LILT post-edits are not published. Until the repository opens, the numbers in Table 1 above can be checked against the paper but the public LocCheck split cannot be regenerated.

What would prove this wrong

The claim is that off-the-shelf reference-free QE, including an LLM judge, cannot reliably pick the human-approved translation on real localisation content, and that small in-domain fine-tuning closes part of the gap. It is wrong if, by 30 June 2027, a second localisation provider publishes preference accuracy for COMETKiwi-22 or a GEMBA-style GPT-4-class judge above 65% on its own production post-edits (at least 5,000 pairs, at least 10 language pairs). That would make LILT's 53.1% and 40.7% a property of LILT's data rather than of the metrics.

A narrower check: when the LocCheck public split is released, baseline COMETKiwi-22 should score at or below 35% on it (the paper reports 30.1%). A result above 50% would mean the challenge set as released differs from the one in the paper.

Sources

1. Kathy Hämmerl, Gabriel Bretschner and Joern Wuebker (TU Munich, MCML, LILT), LocQE: Principled Domain Adaptation for Localisation Quality Estimation (title shortened), arXiv:2609.18720v1, 16 September 2026. Figure 1, Figure 3, Figure 4, Table 1 (COMETKiwi exemplars), Table 3 (LocCheck), Table 9 (per rule), Table 10 (ACES and WMT24), Appendix A.4 (validation study), Appendix B (tokeniser).
2. Ricardo Rei et al., CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task, WMT 2022. Model: Unbabel/wmt22-cometkiwi-da on Hugging Face.
3. Tom Kocmi and Christian Federmann, the GEMBA paper, EAMT 2023, pages 193 to 203, and Kocmi et al., Error Span Annotation, WMT 2024. The GEMBA-ESA QE baseline in the paper uses GPT-4.1.
4. Tokeniser reproduction: microsoft/infoxlm-large, run by BLOMEGA on 19 September 2026 with transformers 4.57.6.

Related BLOMEGA guides: CometKiwi scored the glossary-breaking edit higher at WMT26 · LLM translation noise rates · When human post-editing wins

FAQ

Does COMETKiwi work for localisation quality estimation?

Not out of the box on real localisation content. In LILT's study published 16 September 2026, COMETKiwi-22 preferred the human-approved post-edit over the raw machine translation in 53.1% of pairs, roughly chance, against 66.2% on WMT24. It also gives identical scores (84.83) to a French sentence with and without the required non-breaking space.

Is an LLM judge better than COMETKiwi for picking the better translation?

Not on this data. A GEMBA-ESA judge on GPT-4.1 scored 40.7% preference accuracy on LILT's post-edits, below chance, although it had the best global correlation with professional ESA scores (Kendall tau 0.239 vs 0.108).

How do you adapt a quality estimation model to your own localisation domain?

LocQE fine-tunes COMETKiwi-22 on 10,000 post-edits converted to chrF++-based scores plus 5,000 post-edit preference pairs in the same batches, with a margin ranking loss at margin 0.02 and a tokeniser that keeps special whitespace. That raised preference accuracy from 53.1% to 59.6% using under 1% of the original training data.

Why can't COMETKiwi see non-breaking spaces?

Its encoder uses the XLM-R SentencePiece tokeniser, which normalises a non-breaking space to a regular space and strips trailing spaces before the model sees the text. We confirmed on 19 September 2026 that microsoft/infoxlm-large produces identical token ids for both.

Should localisation QA still use rule-based checks?

Yes. The LocQE authors conclude that heuristics remain useful: numbers, URLs, do-not-translate items, locale punctuation and whitespace are deterministic rules, and training the metric to enforce them lowered its agreement with real reviewer preferences from 59.6% to 57.4%.