Above a discourse dependency of 15, human post-editors win and XCOMET cannot tell
A paper published on 4 September 2026 gives the localization industry something it has not had: a number, computable from the source text alone, that says when to send a segment to a person. It is called discourse dependency. On English-Korean segments scoring 15 or higher, annotators preferred professional human post-editing over the best LLM strategy by +0.35 on a 1 to 4 scale (one-sided p = 3.0 x 10-5). Below 15 the gap was +0.05 and not significant. On the identical segments, XCOMET measured a gap of -0.005, p = 0.40, and no trend at all.
Somebody finally defined what "hard to translate" means, in a way you can compute before translating
4 September 2026. Ahrii Kim, Chanjun Park and Seong-heum Kim of Soongsil University published Discourse Dependency: A Continuous Criterion for Translation Difficulty (arXiv:2609.04959). The argument starts from an observation the field keeps making without acting on: calls for harder machine translation benchmarks have not said what difficulty is.
Their answer is referential reach, the distance a segment must look back into its document to resolve the entities and pronouns it contains, formalised as discourse dependency (DDP). It is computed from named entity re-mentions and pronominal coreference on the source side only. No reference translation, no model output, no metric. You can run it on a file before you have quoted the job.
The validation matters more than the definition. Against gold coreference annotations on 80 OntoNotes 5.0 documents, segment-level DDP correlates with true look-back distance at Pearson r = 0.81, and the error is one-sided in 99.2% of segments: it underestimates in 86.4%, matches in 12.8% and overestimates in 0.8%, with the overestimates fully attributable to named entity recognition false positives. One-sidedness is what makes it usable operationally. A high DDP score is a certificate that the segment genuinely needs long-range context. A low score is not a guarantee that it does not.
Robustness to the extractor was checked too. Swapping spaCy's en_core_web_trf for Flair gives Pearson r = 0.89 and 69% identical segment scores; a deliberately weaker extractor still gives Spearman rho = 0.88 and 73.4% exact agreement. The domain ordering survives all three.
Human preference climbs with discourse dependency, and the metric flatlines
The experiment is automatic post-editing on English into Korean, chosen because Korean is pro-drop, so omitted subjects and objects have to be restored from across sentence boundaries, and because its honorific system demands cross-segment consistency. WMT24++ supplies professionally post-edited human references. After excluding the speech domain and documents under two segments, 59 English-Korean documents remain across literary, news and social.
Six LLMs were tested (DeepSeek-V4-Pro, GPT-5.4, Gemini-3-Flash, Gemma-4-31B, HyperCLOVA-X-SEED-Think-32B, Qwen3.5-27B), all at a uniform 128K context limit, thinking mode off, temperature 1.0, top_p 0.95, seed 42. Five context injection strategies were compared: no context, the whole document, the five preceding segments, the five most semantically similar segments, and structured declarative knowledge about genre, participants and register extracted by an LLM. Annotators ranked four candidates per segment from 1 to 4 with ties allowed, two independent judgments each, across two non-overlapping subsets of about 500 segments. Inter-annotator Kendall tau was 0.37.
| Measurement | Segments below DDP 15 | Segments at DDP 15 and above | Source |
|---|---|---|---|
| Human preference gap, professional post-editing over 5-segment LLM context | +0.05 | +0.35 | §5.1 |
| Significance of that gap | not significant | Wilcoxon one-sided p = 3.0 x 10-5 | §5.1 |
| Bootstrap 95% confidence interval | not reported | [+0.18, +0.52] | §5.1 |
| Share of segments where the human ranked higher | not reported | 60% | §5.1 |
| XCOMET gap on the identical segments | not reported | -0.005, Wilcoxon p = 0.40 | §5.1 |
| XCOMET trend against DDP | Spearman rho = 0.03, p = 0.50 (none) | §5.1 | |
| Human preference gap correlated with DDP, all annotated segments | Spearman rho = 0.28, p = 1.2 x 10-9, n = 498 | §5.1 | |
The rho of 0.28 across all 498 annotated segments is the load-bearing number, not the threshold. It says the effect is a gradient across the whole range, not an artifact of the tail. The threshold of 15 is where it becomes individually significant.
At the other end, at DDP around 10 or below, several LLM post-editing strategies rank above professional human revision. That is not a hedge, it is the same finding read from the other side: for self-contained segments the machine output is what annotators prefer, and paying a linguist for those segments buys nothing measurable.
The context-strategy result is the practical surprise. Giving the model the entire document ranked consistently below giving it the five immediately preceding segments, across nearly the whole DDP range. Length is not the constraint. The authors put it directly: models can ingest the full document but cannot selectively attend to the antecedents that resolve referential dependencies, while the five-segment window often omits the antecedent entirely. XCOMET, meanwhile, ranked full-document context above the five-segment window across the entire range, reversing the human ordering.
Models get more conservative as segments get harder, and that looks like improvement
A second measurement in the paper explains why aggregate dashboards miss all of this. Generation failure rate, defined as TER above 100, falls for every model as discourse dependency rises.
| Model | Overall | DDP = 0 | DDP 1 to 4 | DDP 5 to 14 | DDP 15+ | Source |
|---|---|---|---|---|---|---|
| Gemini-3-Flash | 26.9% | 44.2% | 16.9% | 15.5% | 4.9% | Table 3 |
| Qwen3.5-27B | 13.8% | 17.7% | 13.2% | 10.0% | 7.0% | Table 3 |
| GPT-5.4 | 11.5% | 13.8% | 16.0% | 6.4% | 2.8% | Table 3 |
| Gemma-4-31B | 10.6% | 13.2% | 12.3% | 6.4% | 4.2% | Table 3 |
| HyperCLOVA-X-SEED-Think-32B | 7.2% | 10.1% | 6.4% | 4.5% | 2.8% | Table 3 |
| DeepSeek-V4-Pro | 3.2% | 5.1% | 1.9% | 1.9% | 0.7% | Table 3 |
Edit size, measured by TER, also decreases as DDP grows and stabilises past DDP around 20. Put those two together: on the segments that need the most repair, the models edit least and fail least. They produce conservative, minimally changed output. A conservative edit that leaves a Korean subject unrestored is fluent, short, close to the draft, and wrong. It scores well on everything except a reader.
The authors are careful here and so should anyone quoting them. Failures are excluded as generation errors and they cluster at low DDP, so part of the TER decline is differential filtering. They read TER as a diagnostic of edit volume, not a quality signal. The 44.2% figure for Gemini-3-Flash at DDP = 0 is worth its own line though: on the easiest segments in the set, more than two in five of its post-edits were discarded as generation failures.
One more structural point about benchmarks. Applying DDP to WMT24++ and WMT25 shows both are skewed toward low-dependency segments, and that domain labels do not separate them. In WMT24++ literary averages DDP 52.8, news 9.7 and social 3.8, so social sits below news against the usual assumption that conversation is discourse-rich. WMT25 shifts everything up, literary to 222.6 with a maximum of 482 and social to 21.3. Across the full benchmark 212 of 849 segments, 25.0%, sit at DDP 15 or above.
What this gives a localization buyer that a quality score does not
A routing rule that runs before the job starts. DDP needs the source file and a named entity recogniser. Nothing else. So the decision "which segments get a human" stops being a percentage negotiated in a contract and becomes a computed subset. On WMT-style content that subset is about 25% of segments, and it is the 25% where the published preference gap is +0.35 rather than +0.05.
A defence against your own dashboard. If you are tracking XCOMET or a reference-free variant, you are tracking something that measured -0.005 where annotators measured +0.35. The paper's own recommendation is to interpret automatic metrics with caution in document-level post-editing evaluation, particularly at high DDP, and to complement them with discourse-stratified analysis. Stratifying an existing scorecard by DDP costs one preprocessing pass and changes what it tells you.
A reason to stop buying context length. Full-document context ranked below a five-segment window. The money spent on pushing whole documents through a long-context model is buying a strategy that annotators liked less. Selecting the right five segments, which is what the DDP computation identifies, is the cheaper and better-performing intervention.
A caution about Korean, and about generalising from it. This is one language pair, chosen because it is maximally sensitive to the effect. English-Chinese results are reported in the appendix for cross-lingual consistency. Treat the threshold of 15 as calibrated for English-Korean until somebody recalibrates it for your pair. The underlying claim, that referential reach predicts where humans still win, is what transfers.
A judgement, marked as one: this is the most directly monetisable result in machine translation research this year for anyone selling human review. It converts "humans catch things machines miss" from a sales line into a per-segment score with a published p-value, and it simultaneously says which segments to stop charging for. Vendors who adopt it will quote less work and defend the remainder better.
Check it yourself
DDP is source-side and metric-free, which means you can compute it on your own content today without any of the paper's infrastructure. The definition is entity re-mentions plus pronominal coreference, scored by look-back distance within the document.
# the benchmarks the paper analyses
pip install datasets spacy
python3 -m spacy download en_core_web_trf
python3 - <<'PY'
from datasets import load_dataset
# WMT24++ (Deutsch et al., 2025), the source of the En-Ko documents used here
d = load_dataset("google/wmt24pp", "en-ko_KR")
print(d)
# group by document id, then for each segment measure how far back the
# antecedent of each pronoun and each repeated named entity was last seen.
PYThe paper's own validation set is public too: OntoNotes 5.0 gold coreference, 80 documents, against which DDP scored Pearson r = 0.81 with one-sided error in 99.2% of segments. If you reimplement the measure, that is the check to run first, because one-sidedness is the property the routing rule depends on.
Two numbers to sanity-check your implementation against, both from the paper: WMT24++ literary segments should average around 52.8 and reach a maximum near 171, and WMT25 literary should average around 222.6 with a maximum of 482. If your literary and news distributions overlap heavily, your entity linker is probably missing re-mentions.
What would prove this wrong
The claim under test is that the human advantage in post-editing is concentrated in referential reach, and that current automatic metrics cannot see it. It is wrong if, by 4 September 2027, a published study on the same WMT24++ English-Korean documents shows an automatic metric whose per-segment preference gap between human and LLM post-editing correlates with DDP at Spearman rho of 0.20 or better. XCOMET managed 0.03 with p = 0.50. A metric reaching 0.20 would mean the blind spot has been closed and the human routing argument needs to be rebuilt on something else.
It is also wrong in the other direction if an LLM strategy closes the +0.35 gap at DDP 15 and above. The most likely candidate is not longer context, which lost here, but explicit coreference resolution fed into the prompt as structured antecedents. The paper's APE-exp strategy, structured declarative knowledge of genre, participants and register, did not do it. A strategy that passes resolved antecedents specifically has not been tested and would be the first thing to try.
A third prediction, marked as judgement: no commercial translation management system will expose a per-segment discourse dependency score in its routing rules before 4 September 2027. The computation is cheap and the paper is open. The obstacle is commercial, because the same score that justifies human review on 25% of segments withdraws the justification from the other 75%.
FAQ
When is human post-editing still better than an LLM?
On segments with discourse dependency of 15 or higher, where annotators preferred professional human post-editing by a mean of +0.35 on a 1 to 4 scale (one-sided p = 3.0 x 10-5, 95% CI [+0.18, +0.52], human preferred in 60% of segments). Below 15 the gap was +0.05 and not significant, and at very low DDP several LLM strategies ranked above the human.
What is discourse dependency?
A source-side, metric-free measure of how far a segment must look back into its document to resolve its entities and pronouns, computed from named entity re-mentions and pronominal coreference. Validated on 80 OntoNotes 5.0 documents at Pearson r = 0.81 with one-sided error in 99.2% of segments.
Do automatic metrics detect these errors?
Not in this study. XCOMET measured a gap of -0.005 (p = 0.40) on the same segments where annotators measured +0.35, with no trend against DDP (Spearman rho = 0.03, p = 0.50). It also ranked full-document context above a five-segment window, reversing the human preference. The reference-free metrics had the flattest trajectories of all.
Does giving the model the whole document fix it?
No. Full-document context ranked below the five preceding segments across nearly the whole range. The authors read it as selective attention, not context length.
Which content has high discourse dependency?
Literary most: WMT24++ literary averages 52.8 against news 9.7 and social 3.8; WMT25 literary averages 222.6 with a maximum of 482. Domain labels are unreliable though. In WMT24++ social scores below news, and 25.0% of all segments sit at DDP 15 or above regardless of label.
Sources
- Ahrii Kim, Chanjun Park and Seong-heum Kim, Soongsil University, Discourse Dependency: A Continuous Criterion for Translation Difficulty, arXiv:2609.04959, submitted 4 September 2026. Section 3.2 (OntoNotes validation, r = 0.81, 99.2% one-sided); Section 3.3 (WMT24++ and WMT25 DDP distributions); Section 4 (En-Ko setup, six models, five context strategies); Section 5.1 (human preference, +0.35 at DDP 15+, XCOMET p = 0.40); Table 3 (generation failure by stratum); Appendix A.1 and Table 4 (extractor robustness). PDF.
- Daniel Deutsch et al., WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages and Dialects, 2025. The professionally post-edited English-Korean documents used in the experiment.
- Tom Kocmi et al., Findings of the WMT25 general machine translation shared task, 2025. The second benchmark analysed for DDP distribution.
- OntoNotes 5.0 (LDC2013T19). The gold coreference annotations DDP was validated against.
- DeepL, Defining translation quality in the age of AI with Slator's Alex Edwards and Florian Faes, 1 June 2026. Industry context for the metric gap.
Related BLOMEGA guides: Multilingual LLM judges and translationese bias · When a subtitle score is a segmentation artifact · The cost of a dubbed minute in 2026.