BLOMEGA

Amazon's StalePO: the translation model that improved most on MQM scored worst on COMET, chrF and BLEU

Lab note · 20 September 2026 · BLOMEGA

Abstract technical illustration of two overlapping translucent text-line layers, one older and faded, one bright, with a few highlighted segments shifting between them, cyan and orange on a dark ground

Amazon's Translation Services team reported on 14 September 2026 that fine-tuning GPT-OSS 120B on post-edits of an older MT system, with an objective it calls StalePO, raised the share of English-to-Hindi segments passing all seven MQM checks by 14.9 points (13.8 under professional translators). The same model scored worst of eight systems on COMET, chrF and BLEU against those post-edits. Across the eight, COMET and the MQM gain rank in opposite order (Spearman -0.71, our calculation). Plain supervised fine-tuning on the post-edits raised BLEU by 8.4 and the pass rate by 1.8.

What Amazon published on 14 September 2026

arXiv:2609.16340, by Rohit Dhaipule, Sukhdeep Singh Kharbanda, Prasanth Bathala, Pradyumna Lanka and Anubhav Shrimal of Translation Services at Amazon, names a problem every localization program with a translation memory has: the only preference data available is human post-edits of the previous engine. Post-editing is priced per segment, so nobody re-edits millions of segments each time the model is replaced. The authors call this the stale preference problem.

The setup: GPT-OSS 120B with QLoRA (rank 8) on 8 A100s. English to Hindi uses 24k training post-edits, filtered by a Kimi K2.5 classifier to keep edits that modernise style (formal Sanskrit-derived vocabulary to the conversational transliterations the style guide mandates, passive to active) and fix punctuation. English to Turkish uses 10k unfiltered post-edits. The source prompt carries the target-language style guide and termbase entries. Evaluation is on temporally disjoint 5,000-segment test sets, scored by seven binary LLM judges built on Claude Sonnet 4.5 (Verity, Design, Terminology, Locale Convention, Style, Accuracy, Fluency), with professional translators re-annotating 2,000 Hindi segments (24,446 words).

Headline: +14.9 pp all-pass on English to Hindi, +4.6 pp on English to Turkish, +13.8 pp under human MQM on Hindi. Turkish has no human evaluation.

The better the MQM result, the worse the reference-based score

Table 1. English to Hindi, 5,000-segment test set. The first four value columns are percentage-point changes in pass rate against the base model under the seven-category LLM-judged MQM (Table 1, mean of five inference runs; Verity, Design and Locale moved by at most 1.1 pp for every method and are omitted). The last four are reference-based metrics against the human post-edits (Table 4, mean of five runs). Absolute pass rates are not reported: the authors say company policy prevents disclosure.
MethodStyleAccuracyFluencyAll-passTER (lower better)COMETchrFBLEUSource
Base (GPT-OSS 120B)000051.5783.5955.6037.41Tables 1, 4
SFT on post-edits+2.0+1.5+12.1+1.850.1883.6559.6645.82Tables 1, 4
DPO+1.0-0.3+0.7-0.450.3583.8356.3539.07Tables 1, 4
TDPO+1.4+0.5+1.8+0.550.5183.9056.1538.59Tables 1, 4
APO-Down+2.3-1.2+0.2+0.650.5183.7056.0138.94Tables 1, 4
BAPO+1.2-0.4+0.9-0.250.4183.8456.0839.05Tables 1, 4
BAPO-Down+15.4-6.4+16.9+7.456.7983.1052.3441.04Tables 1, 4
StalePO+20.9-5.4+16.5+14.953.5282.8652.1936.37Tables 1, 4
The biggest MQM gain gets the lowest COMET English to Hindi, 5,000-segment test set. arXiv:2609.16340 Tables 1 and 4. Spearman rho = -0.71 (our calculation). All-pass change (pp) COMET vs post-edits 82.5 83.0 83.5 84.0 0 5 10 15 Base (GPT-OSS 120B) 0 83.59 SFT on post-edits +1.8 83.65 DPO -0.4 83.83 TDPO +0.5 83.90 APO-Down +0.6 83.70 BAPO -0.2 83.84 BAPO-Down +7.4 83.10 StalePO +14.9 82.86 COMET axis starts at 82.5
All-pass change and COMET for every English to Hindi system in the paper. COMET's spread across all eight is 1.04 points; the MQM spread is 15.3 points. Source: arXiv:2609.16340, Tables 1 and 4; rank correlation is our calculation.

We ranked the eight systems (base plus seven methods) on each reference-based metric and on the all-pass change. Spearman's rho against the all-pass change is -0.71 for COMET (p = 0.047), -0.54 for TER (sign flipped so higher is better), -0.52 for chrF and -0.07 for BLEU. Eight points is a small sample, so read the direction, not the decimals: none of the four metrics ranks the systems the way the MQM judges and the human annotators do.

The paper's own explanation is Appendix A. On the 24,220 Hindi training segments, post-edits are closer to the legacy NMT output (mean TER 0.426) than to the new model (0.517), and closer to the legacy output in 55.3% of segments against 23.7%. Post-editors correct; they do not retranslate. Score a new model against those edits and you reward it for sounding like the old engine. The example the authors give: the style guide mandates एंटर for "enter", the post-edit kept the formal दर्ज, and the reference metrics penalise the correct choice.

StalePO is not free. Accuracy falls 5.4 pp under the LLM judge and 3.5 pp under humans, and the human breakdown puts most of it in minor untranslated spans; mistranslations and omissions are "nearly unchanged". Terminology drops 2.0 pp under the judge and rises 0.8 under humans.

Human MQM agrees on the aggregate and on the accuracy cost, not on the size of the fluency gain StalePO, English to Hindi, pp change in pass rate vs base. arXiv:2609.16340 Table 1. -10 0 10 20 30 40 Terminology -2.0 +0.8 Style +20.9 +15.9 Accuracy -5.4 -3.5 Fluency +16.5 +37.3 All seven pass +14.9 +13.8 LLM judge (Claude Sonnet 4.5), 5,000 segments Professional translators, 2,000 segments
Judge and humans agree on the direction of the aggregate, style, accuracy and fluency changes. They disagree on terminology and on magnitude: humans see a fluency gain more than twice the judge's. Source: arXiv:2609.16340, Table 1.

Three constraints turn stale post-edits into a gain, and each one alone does nothing

Standard DPO only asks that the preferred response gain likelihood relative to the dispreferred one. When the preferred response, the post-edit, is worse than what the model already writes, DPO can raise its likelihood and pull the model backwards. StalePO combines three fixes from earlier work: downward control from APO-Down (both likelihoods must fall), an anchor to the model's own sampled base output from BAPO, and a token-level KL from TDPO.

A post-edit of an old system is a better answer on two dimensions and a worse one on the rest Legacy NMT output the dispreferred y_l Human post-edit the preferred y_w 24,220 En-Hi segments New model output GPT-OSS 120B, y_b edited, not redone Post-edit sits closer to the OLD system mean TER 0.426 vs 0.517 to the new model closer in 55.3% of segments vs 23.7% Better than y_b on style and fluency, worse on accuracy and terminology in many segments All-pass change vs base, English to Hindi, LLM-judged MQM (arXiv:2609.16340 Table 1): DPO none of the three -0.4 pp TDPO token KL only +0.5 pp BAPO anchor only -0.2 pp APO-Down downward only +0.6 pp BAPO-Down down + anchor +7.4 pp StalePO all three +14.9 pp Down: push the likelihood of BOTH responses down, so the model never imitates the stale post-edit. Anchor: keep the model close to its own base output y_b, which is already better on most dimensions. Token KL: constrain divergence per token, so the fix lands on the few words the post-editor changed. Each alone moves the pass rate by under 1 pp. SFT on the post-edits: +1.8 pp, even though BLEU rises 8.4.
Why the post-edit is a mixed signal, and what each training component contributes. Source: arXiv:2609.16340, Sections 2, 4 and 6.1, Tables 1 to 3.

The ablation is the most useful table in the paper for a practitioner. DPO tuned on its own sweep moves the pass rate by -0.4 pp, TDPO +0.5, BAPO -0.2, APO-Down +0.6: none moves the pass rate by as much as one point. Down plus anchor (BAPO-Down) reaches +7.4. Adding the token-level KL doubles it to +14.9 and recovers a point of accuracy. On the hyperparameters, all-pass peaks at KL coefficient α = 0.25, and without the anchor every metric stays near the base.

If you own a post-edit archive, stop scoring new engines against it

Your translation memory is a biased reference set. Any engine evaluation that uses post-edits of the incumbent engine as references will favour the incumbent and its imitators. In this paper that bias is large enough to invert the ranking. Use reference-free, category-level MQM (human or judged) for engine selection, and keep COMET or BLEU against post-edits only as a regression alarm.

Do not fine-tune on the archive with SFT and call it adaptation. SFT raised BLEU by 8.4 and the pass rate by 1.8, and it lowered terminology by 3.3 pp. It learns the old engine's scaffolding along with the edits.

Filter by what changed, not by volume. The Hindi gain (+14.9) came from 24k edits selected for one kind of change; the unfiltered Turkish set (10k) gave +4.6 on a stronger base. The paper cannot separate the filter's effect from the language's, but a classifier that tags edits by MQM category is cheap next to a new round of post-editing.

Expect an accuracy cost and budget review for it. A model pushed toward conversational style oversimplifies. Route the untranslated-span errors to a human check before release.

A judgement, marked as one: the paper's numbers are relative to an undisclosed base pass rate, the Turkish result has no human check, and the method is from a single company's data. It is still the clearest published evidence that a localization archive's value lies in which categories the editor fixed, which is exactly the metadata most archives never record.

Check it yourself

# Rank correlation between each reference metric and the MQM all-pass change
# (values copied from arXiv:2609.16340 Table 1 and Table 4, English to Hindi)
python3 - <<'EOF'
from scipy.stats import spearmanr
ap  = [0, 1.8, -0.4, 0.5, 0.6, -0.2, 7.4, 14.9]   # Base SFT DPO TDPO APO-Down BAPO BAPO-Down StalePO
ter = [51.57, 50.18, 50.35, 50.51, 50.51, 50.41, 56.79, 53.52]
comet = [83.59, 83.65, 83.83, 83.90, 83.70, 83.84, 83.10, 82.86]
chrf = [55.60, 59.66, 56.35, 56.15, 56.01, 56.08, 52.34, 52.19]
bleu = [37.41, 45.82, 39.07, 38.59, 38.94, 39.05, 41.04, 36.37]
for n, v in [("TER", [-x for x in ter]), ("COMET", comet), ("chrF", chrf), ("BLEU", bleu)]:
    r = spearmanr(v, ap); print(n, round(r.correlation, 3), round(r.pvalue, 3))
EOF
# TER -0.539 0.168
# COMET -0.714 0.047
# chrF -0.524 0.183
# BLEU -0.071 0.867

To test the incumbent bias on your own archive, compute TER from each post-edit to the legacy output and to the candidate engine's output, as in the paper's Appendix A. With sacrebleu: TER().sentence_score(pe, [legacy]).score against TER().sentence_score(pe, [candidate]).score, then count how often the legacy distance is smaller. If it is smaller in most segments, your references are measuring resemblance to the old engine. The paper is at https://arxiv.org/pdf/2609.16340; Table 1 is on page 5, Tables 3 and 4 in Appendix A.

What would prove this wrong

The claim is that reference-based metrics computed against post-edits of an older engine rank newer systems in the wrong order. It is wrong if an independent replication on a public post-edit corpus (for example the WMT or QE shared-task post-edit sets) finds a positive rank correlation, rho above +0.5, between COMET against post-edits and human MQM pass rates across at least six systems trained on those post-edits. We will look for one in the WMT 2026 proceedings and on arXiv by 30 June 2027.

Narrower: if a human MQM evaluation of the English to Turkish StalePO model is published and shows no all-pass gain, the method's +4.6 pp on an unfiltered set was a judge artefact, and the "filter by what changed" advice becomes a requirement rather than a recommendation.

Sources

  1. Rohit Dhaipule, Sukhdeep Singh Kharbanda, Prasanth Bathala, Pradyumna Lanka, Anubhav Shrimal (Translation Services, Amazon), StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translation, arXiv:2609.16340v1, 14 September 2026. Tables 1 to 4, Sections 2, 5, 6, Limitations, Appendix A.
  2. Karel D'Oosterlinck et al., Anchored Preference Optimization and Contrastive Revisions, TACL 2025 (APO-Down).
  3. Rafael Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023.
  4. Arle Lommel et al., Multidimensional Quality Metrics (MQM) definition, QTLaunchPad, 2014.

Related BLOMEGA guides: COMETKiwi on real localisation post-edits (LocQE) · LLM judges rank machines above human translators · WMT 2026 terminology and glossary adherence

FAQ

What is the stale preference problem in machine translation?

Preference data for a new MT model is usually human post-edits of an older engine's output, because re-editing for every new model is too expensive. Those post-edits keep the old engine's wording, so they are better than the new model on a few dimensions (in Amazon's data, style and fluency) and worse on the rest.

Does fine-tuning on post-edits improve a new translation model?

Not by imitation. In Amazon's English to Hindi experiment, supervised fine-tuning on 24k post-edits raised BLEU by 8.4 points but the share of segments passing all MQM checks by only 1.8 points. DPO moved it by -0.4. StalePO, which pushes both responses' likelihood down and anchors to the model's own output, raised it by 14.9.

Can I use COMET or BLEU against post-edits to choose between MT engines?

Not reliably. In arXiv:2609.16340 the system with the largest MQM gain scored lowest on COMET, chrF and BLEU against the post-edits, and across eight systems COMET ranked them in nearly the opposite order (Spearman -0.71, our calculation from the paper's tables).

What does StalePO cost in accuracy?

On English to Hindi, accuracy pass rate fell 5.4 points under the LLM judge and 3.5 under professional translators, mostly minor untranslated spans. Mistranslations and omissions were nearly unchanged.

Which model and judge did Amazon use?

GPT-OSS 120B fine-tuned with QLoRA, judged by seven binary MQM scorers built on Claude Sonnet 4.5, with professional translators re-annotating 2,000 English to Hindi segments.