Amazon's diagnose-then-repair MT post-editor beats one-stage in 13 of 42 of its own cells

What Amazon published on 19 September 2026

arXiv:2609.22793, by Ji Hun Wang and Siyu Wu of Amazon, submitted 19 September 2026 and listed in the ACL 2026 Industry Track proceedings (pages 1683 to 1698). The idea is a division of labour. Claude 3.5 Sonnet translates. Claude Opus 4.5, the evaluator, reads the source, the draft and k retrieved exemplars from a translation memory, and writes MQM-style error spans with a category, a severity and a suggested_edit. A separate post-editor then applies minimal fixes restricted to those spans under what the authors call an edit contract.

The test data is internal e-commerce translation memory: 5,000 held-out segments per direction from English into German, Chinese, French, Spanish, Italian and Japanese, with knowledge bases of 135,000 to 184,000 segment pairs (Table 10). Seven post-editors were compared on Amazon Bedrock: Gemma 3 4B and 27B, Claude 3.5 Haiku, Claude Sonnet 4 and 4.5, and GPT OSS 20B and 120B. Quality is scored with reference-based COMET-22 and reference-free COMETKiwi. No code, outputs or data are released; we found no repository.

This is a different question from our note on Amazon's StalePO, which trains on legacy human post-edits. Here nothing is trained. The repair is pure prompting.

Does the two-stage editor beat one-stage in every language?

The main text shows only German and Chinese, the two directions where two-stage wins clearly. Appendix A has all six. Put the one-stage baseline (Claude Opus 4.5 diagnosing and editing in one call, same P3 prompt with retrieval) next to the best two-stage configuration (Opus 4.5 evaluator, Claude Sonnet 4.5 editor) and the picture splits in half.

All scores are from arXiv:2609.22793 Table 5. Differences, counts and means are our arithmetic; we also re-derived every parenthetical delta printed in Tables 1, 2, 4, 5, 6 and 7 from its baseline, and all match to 0.01.

The appendix text calls French "a notable anomaly" and says two-stage "outperforms the one-stage baseline in the majority of cases across all directions". The table does not support that sentence: one-stage also beats every two-stage editor on Italian (91.25 against a best of 91.05) and Japanese (93.48 against 92.87), and 13 of 42 is 31%, not a majority. The abstract's "consistently improves both COMET-22 and COMETKiwi over one-stage" holds for the best configuration in 3 of 6 directions.

Averaged over the six directions the best configuration does come out ahead, by 0.53 COMET-22 and 0.89 COMETKiwi, and that average is carried by Chinese. Our reading, a judgement: the split is real in German and Chinese and absent in the three directions where one-stage Opus already gained 3.6 to 4.4 points on its own.

COMET-22 rises in 42 of 42 cells and COMETKiwi falls in 25

Every two-stage cell in Table 5 improves COMET-22 over the unedited draft, from +0.22 (GPT OSS 20B, Chinese) to +7.92 (Sonnet 4.5, Spanish). COMETKiwi tells another story. It drops below the unedited draft in 25 of 42 cells: all seven editors on Italian and on Japanese, five of seven on Spanish, three on Chinese, two on French and one on German. The one-stage baseline drops Kiwi in 4 of 6 directions.

The two metrics measure different things here. COMET-22 compares the output to the translation-memory reference; the retrieval step pulls exemplars from the same translation memory. A post-edit that moves toward house phrasing scores well against a house reference even when a reference-free estimator sees no gain. That is our hypothesis, not a finding of the paper, and it is testable: the paper's P2 row (no retrieval) should show a smaller COMET-22 minus Kiwi gap than P3. On German, P2 gives +0.18 and +0.00; P3 gives +3.54 and +1.84, so the gap does widen with retrieval, from 0.18 to 1.70.

The evaluator ablation (Table 6, French) shows the same split. COMET-22 rises with evaluator size, 88.28 for Claude 3.5 Haiku to 90.81 for Opus 4.5, as the paper says. COMETKiwi runs the other way: Haiku 83.97, Sonnet 4 83.59, Sonnet 4.5 83.10, Opus 4.5 83.27. The cheapest evaluator gives the best reference-free score. We covered how often COMETKiwi agrees with human post-edit choices in our LocQE note; here there is no human judgement to break the tie.

Where the +3.54 on German comes from

Table 1 and Table 7 let you decompose the German gain. With the plain GEMBA-MQM prompt (P1), two-stage adds +0.08 COMET-22, slightly less than one-stage GEMBA-MQM's +0.12. Adding a descriptive rubric and the edit contract (P2) reaches +0.18. Adding retrieved translation-memory exemplars (P3) jumps to +3.54. One-stage Opus with the same P3 prompt reaches +2.99.

By our arithmetic, of the 3.54 points, retrieval accounts for 3.36 (P2 to P3 under the same two-stage setup) and the split between judge and editor for 0.55 (one-stage P3 to two-stage P3), and that 0.55 also changes the editing model from Opus 4.5 to Sonnet 4.5. On Chinese the split is more even: P2 already gives +1.10, P3 adds 1.94, and two-stage beats one-stage by 2.10. The title claim, that separating diagnosis from repair is what makes MQM feedback useful, is best supported on Chinese. On German the retrieved exemplars do most of the work.

A second round that flags 207 segments cannot cost 1.86 points without breaking the contract

Section 5.4 feeds the Sonnet 4.5 output to a second editor, C2, which first decides whether the first round resolved every error. Table 3 gives flag counts out of 4,031 German segments, from 205 (Claude 3.5 Haiku, 5.1%) to 1,282 (Sonnet 4.5, 31.8%). Table 4 gives the COMET-22 change. The C2 prompt says: if all errors are resolved, return the first-round output unchanged. So the corpus-level change should come only from flagged segments.

Divide the corpus change by the flagged share and you get the average change on each segment C2 touched. For Sonnet 4.5 it is +0.4. For Claude Sonnet 4 it is -2.9, for Haiku -5.9, for GPT OSS 20B -9.1, for Gemma 3 4B -9.2. For GPT OSS 120B it is -36.2 COMET-22 points per edited segment (1.86 × 4,031 / 207), or -44.9 if the 1.86 is averaged over all 5,000. A minimal edit does not take a segment from about 90 to about 54. Either GPT OSS 120B rewrote segments it did not flag, which is the paraphrastic drift the edit contract exists to stop, or some outputs failed to parse and were scored as broken text. The paper reports neither a parse-failure rate nor an edit rate, so we cannot tell which.

Two smaller inconsistencies in the same section. The text says the second round lowers COMET-22 "in six out of seven cases"; Table 4 shows five decreases, one +0.00 (Gemma 3 27B) and one +0.14. And 4,031 is not 5,000: the paper never says which 969 German segments are missing from the second-round study. One plausible reading is that 4,031 (80.6%) is the share the evaluator flagged in the first place, which would be the only number in the paper close to "what fraction of segments needed an edit". It is not stated.

What the paper does not report

The abstract says the evaluator's "error spans and severities show strong agreement with human MQM annotations and human editor preferences". We searched the full text: there is no human MQM annotation, no agreement statistic, no human preference study and no table or appendix with any of these. The abstract also says "seven languages"; the experiments use six target languages from English. The conclusion says "two language directions"; Appendix A has six.

Four numbers a localization team would need before adopting this are absent: the share of segments the evaluator flagged, the precision of those flags against a human reviewer, the edit rate (TER or changed characters) of the post-editor, and a per-segment cost. The only edit-coverage figure is for the one-stage GEMBA-MQM baseline, which "attempted to correct 44.8% of segments" for a +0.12 COMET-22 gain (Section B.2).

Cost can be bounded from what is reported. Table 8 times the pipeline on 50 segments: evaluator mean 14.34 s, post-editor 3.30 s, end to end 17.64 s, P95 59.41 s. The evaluator is 81.3% of mean latency (14.34 / 17.64, ours). Run serially, one 5,000-segment test set is 24.5 hours (5,000 × 17.64 s, ours). Bedrock list prices in Table 9 put the Opus 4.5 evaluator at $5 / $25 per million input / output tokens against $0.8 / $4 for Claude 3.5 Haiku, 6.25 times cheaper. Token counts per call are not reported, so a dollar figure per segment would be a guess and we do not give one.

What it means if you run LLM post-editing in production

• Test one-stage with retrieval first. On French, Italian and Japanese, a single Opus 4.5 call with translation-memory exemplars beat every two-stage configuration. It is one model call instead of two, and on German it already delivered 2.99 of the 3.54 points.
• Treat retrieval as the main lever. On German the exemplars account for 3.36 points and the rubric plus edit contract for 0.10. If you have a clean translation memory, spend the engineering there before building a separate judge.
• Do not gate on COMET-22 alone. A post-edit that raises COMET-22 while lowering COMETKiwi, as 25 of 42 cells here do, needs a human look before it ships. If your references and your retrieval pool come from the same memory, the reference-based metric is partly measuring agreement with that memory.
• Try the cheap evaluator. Claude 3.5 Haiku as evaluator gave 30% of Opus's COMET-22 gain on French (1.09 of 3.62) and the best COMETKiwi of the four evaluators (+1.32 against +0.62), at 6.25 times lower token price. That is a judgement on one direction, not a result the paper generalises.
• Stop after one round, and log what the second editor touches. The paper's advice is right. The GPT OSS 120B row says why: count edited segments and score only those, or a contract violation hides inside a corpus average.

Check it yourself

Everything above comes from the paper's own tables; nothing else is released. Get the text with minus signs intact:

curl -sL https://arxiv.org/pdf/2609.22793 -o dtr.pdf && pdftotext -layout dtr.pdf dtr.txt
grep -n "Claude Sonnet 4.5\|No post-edit\|One-stage" dtr.txt | sed -n '1,40p'   # Table 5 rows

Then the counts, with Table 5 transcribed as (COMET-22, COMETKiwi):

python3 - <<'PY'
L=["DE","CN","FR","ES","IT","JP"]
base=[(86.39,82.66),(87.76,81.60),(87.19,82.65),(83.01,82.02),(87.36,84.28),(89.08,84.79)]
one =[(89.38,82.84),(88.70,79.57),(90.83,83.29),(89.59,81.73),(91.25,82.77),(93.48,83.96)]
two={"Gemma3-4B":[(88.29,82.43),(91.08,79.82),(89.36,82.90),(83.73,80.16),(89.36,82.11),(89.73,82.52)],
 "Gemma3-27B":[(89.62,83.03),(90.83,81.92),(90.55,83.15),(89.17,81.59),(90.72,82.45),(91.59,82.92)],
 "Haiku3.5":[(89.53,82.69),(90.20,82.51),(90.56,82.97),(90.35,81.75),(90.64,82.10),(92.43,83.06)],
 "Sonnet4":[(89.85,83.45),(90.56,82.94),(90.76,83.22),(90.91,82.05),(91.02,82.57),(92.82,83.79)],
 "Sonnet4.5":[(89.93,84.50),(90.80,83.19),(90.81,83.27),(90.93,82.12),(91.05,82.63),(92.87,83.80)],
 "GPToss20B":[(89.10,82.67),(87.98,79.06),(89.56,82.10),(87.30,79.00),(89.79,81.29),(91.71,82.71)],
 "GPToss120B":[(89.18,82.76),(89.63,79.52),(88.37,81.29),(87.44,78.99),(88.82,80.72),(90.83,82.12)]}
beat=sum(v[i][0]>one[i][0] for v in two.values() for i in range(6))
kdown=sum(v[i][1]<base[i][1] for v in two.values() for i in range(6))
print("two-stage beats one-stage on COMET-22:",beat,"/ 42")      # 13
print("COMETKiwi below unedited MT:",kdown,"/ 42")                # 25
print("GPT OSS 120B, pts per flagged segment:",round(-1.86*4031/207,1))  # -36.2
PY

Expected output: 13, 25 and -36.2. The per-language rows of the table above follow from the same lists.

What would prove this wrong

Our claim is that the paper's own evidence supports "retrieval-grounded MQM feedback helps" much more firmly than "separating the judge from the editor helps", and that the separation effect is language-dependent and near zero in three of six directions.

It would be wrong if a matched comparison, the same model as one-stage editor and as two-stage post-editor (Opus 4.5 in both, or Sonnet 4.5 in both), showed two-stage ahead by more than 1.0 COMET-22 and ahead on COMETKiwi in at least five of the six directions. We predict that by 30 June 2027, any such matched result published by the authors or a replication on public data (for example WMT general-task test sets with an in-domain retrieval pool) will show a six-direction mean two-stage advantage under 1.0 COMET-22, against the 0.53 we computed with unmatched models here.

A second, cheaper test: if the authors release second-round logs, GPT OSS 120B's 207 flagged segments should not account for the whole 1.86-point loss. We predict the logs will show either edits to unflagged segments or malformed outputs. If the flagged segments alone really lost 36 points each with well-formed minimal edits, our reading of Table 4 is wrong.

Sources

1. Ji Hun Wang, Siyu Wu. Diagnose, Then Repair: A Two-Stage MQM-Guided Post-Editing Framework for Domain-Specific Machine Translation. arXiv:2609.22793v1, submitted 19 September 2026. Tables 1 to 10, Sections 5.1 to 5.5, Appendices A to E.
2. ACL Anthology entry 2026.acl-industry.115, Proceedings of ACL 2026 (Industry Track), pages 1683 to 1698. Checked 24 September 2026.
3. Tom Kocmi, Christian Federmann. GEMBA-MQM: Detecting translation quality error spans with GPT-4. arXiv:2310.13988, 2023. The P1 prompt.
4. BLOMEGA arithmetic, 24 September 2026: per-direction two-stage minus one-stage differences, cell counts, six-direction means, ablation decomposition and implied per-flagged-segment change, all from the paper's Tables 1 to 8; re-derivation of every printed delta.

FAQ

What is the Diagnose, Then Repair framework?

A two-stage automatic post-editing pipeline from Amazon (arXiv:2609.22793, ACL 2026 Industry Track). Claude Opus 4.5 reads the source and the machine translation, retrieves similar segments from a translation memory, and writes MQM-style error spans with suggested fixes. A separate LLM, best Claude Sonnet 4.5, applies minimal edits restricted to those spans. It was tested on e-commerce translation memory from English into German, Chinese, French, Spanish, Italian and Japanese, 5,000 test segments each.

Does the two-stage design beat one-stage judge-and-refine?

On average and in some languages. The best two-stage configuration beats one-stage Claude Opus 4.5 by 2.10 COMET-22 points on Chinese, 1.34 on Spanish and 0.55 on German, and trails it by 0.02 on French, 0.20 on Italian and 0.61 on Japanese. Counting all seven post-editors in all six languages, 13 of 42 two-stage cells beat one-stage. The six-language average advantage of the best configuration is 0.53 COMET-22 and 0.89 COMETKiwi, by our arithmetic from Table 5.

Should I run a second post-editing round?

The paper says no, and its own numbers agree: five of seven second-round editors lower COMET-22 on English to German, one leaves it unchanged and Claude Sonnet 4.5 adds 0.14. One number does not add up. GPT OSS 120B flags only 207 of 4,031 segments but lowers corpus COMET-22 by 1.86, which implies a 36-point drop on each touched segment unless it also changed segments it did not flag.