Deleting the 47.5% of examples the model already got right raised glossary accuracy from 78.7% to 89.9%

The WMT 2026 terminology results are in, and the human reference is not at 100%

9 September 2026. Xixian Liao and Maite Melero of the Barcelona Supercomputing Center published SalamandraTA at WMT 2026 Terminology Shared Task: Hard Examples Are Better Teachers (arXiv:2609.09999), describing the BSC submission to Track 1 and the data-selection study behind it.

Track 1 measures lemmatised exclusive term success, the share of prescribed glossary terms that appear correctly in the output, alongside document-level chrF++ as a check that translation quality has not been traded away. Two directions were evaluated in proper mode, English into Polish and Spanish into Basque, across 22 submissions. The findings paper for the task is Charkiewicz et al., 2026.

The number to start from is the gold row. The human reference translation achieved 96.4% term success on average, and 93.6% into Polish. Professional human translators working with the prescribed glossary in front of them missed one prescribed term in sixteen on the Polish direction. Any procurement conversation that assumes 100% glossary compliance is achievable by anyone is starting from a number nobody has hit.

Eight of 22 systems, and the gap between the best machine and the human is 0.7 points

Read the TaT row against the gold row. TaT reached 97.2% into Polish, beating the human reference's 93.6% on that direction, and then dropped to 91.7% into Basque. Directional variance inside a single system is larger than the spread across the top seven systems. Whoever quotes you a single glossary-compliance number for all your locales is averaging away the thing you care about.

BSC's own system placed seventh on average term success at 94.2% and 74.6 chrF++, with only two of the twenty-two submissions (COZY and Agentic-OPUS) improving on both axes simultaneously, and four more reaching higher term success at lower chrF++. What makes the paper worth reading is not that placement. It is the ablation underneath it.

Most glossary training data teaches nothing, because the model already agreed with the glossary

The standard recipe is to fine-tune on glossary-annotated translation pairs. The BSC observation is that for most such pairs the glossary prescribes exactly what the model would have produced unprompted. Those examples cost the same compute and transmit no information about the behaviour you actually want, which is reading the glossary and deferring to it.

The controlled study holds everything fixed and varies one quantity. Starting from the SalamandraTA-7B base checkpoint, each point on the curve is a complete instruction-tuning run over an identical background mixture of 682,431 instances, one epoch, learning rate 1e-5 with cosine decay and 3% warmup, context length 8,192, effective batch size 32. Into that background goes a terminology component of fixed size, 76,000 instances across the 19 directions the EMEA pipeline produces, capped at 4,000 per direction. The only thing that changes between runs is the share of hard examples in that component.

Evaluation is on the WMT 2025 Terminology Track 1 test set: English into German, Spanish and Russian. None of those three appears as a training direction in the terminology component, so every gain reported is transfer, not memorisation. Term accuracy is micro-averaged over all 1,589 term occurrences, with confidence intervals from a clustered paired bootstrap over source segments at 95% with 10,000 resamples.

Three things in that table are worth separating. First, the headline: 78.7 to 89.9, +11.2 points, 95% CI [+9.5, +12.8], p < 0.0001. Second, chrF does not move. It sits between 67.0 and 67.4 for every mixture containing any hard examples, and the all-easy mixture is the weakest on chrF too at 66.5, so nothing is being traded. Third, a quarter dose delivers 86.7, which is 71% of the full effect, and the dip at 50% traces to English into Russian and sits inside the noise (25% against 50% is +0.3 points, 95% CI [-0.6, +1.3], p = 0.55).

The random baseline is the honest caveat and the authors flag it themselves. An unfiltered sample reaches 88.5, close to the fully filtered 89.9, because EMEA is naturally rich in hard sentences: across 19 directions the hard pool holds 522,276 records against 471,717 easy, a 52.5% natural hard rate, ranging from 32.4% for German into English to 63.7% for Spanish into Hungarian. A forced 50/50 split trails the fully filtered set by 3.5 points. If your own corpus is already majority-hard, filtering buys you less. If it is majority-easy, which is what a mature translation memory looks like after years of consistent terminology work, it buys you the full eleven.

That is the uncomfortable inversion for a well-run localization operation. The cleaner your historical terminology compliance, the less your archive teaches a model about complying.

Your quality metric will argue against your glossary, and it will be locally right

The paper's discussion section contains the finding a terminology manager should take to their vendor. Test glossaries are not injective. In the WMT 2026 Polish set, boot lid and trunk lid share the single prescribed target klapa bagażnika. A document that deliberately enumerates both synonyms can only be faithfully translated as zarówno klapa bagażnika, jak i klapa bagażnika, which no editor would leave standing.

CometKiwi scored the smoothed, glossary-breaking variant 0.13 higher. Across BSC's whole post-editing stage, the quality estimation metric consistently assigned positive deltas to exactly the edits that damaged adherence. Their fix is an explicit constraint rather than a better metric: an edit is applied only if it raises CometKiwi and does not lower document-level term accuracy. Most edits that survive that gate fix grammar, spelling or punctuation, or remove a genuinely surplus repetition.

Four things follow for anyone running terminology at scale.

Filter your terminology training data against your current model, not against a quality score. The filter needs no annotator and no threshold. Translate each candidate with the model you have, compare to the glossary with a lenient matcher that ignores casing, punctuation and spacing, and keep the mismatches. The paper's whole claim is that this binary, self-referential definition of hardness is what carries the gain.

Expect the filter to transfer beyond glossaries. The authors' reading is that the value of hard examples is not the term pairs they contain but the behaviour they force, reading the constraint when the default would have been something else. That should apply equally to style guides, register constraints and do-not-translate lists, where naturally occurring supervision is similarly redundant. That is a hypothesis, stated as one, not a measured result.

Never gate terminology work on a quality estimation metric alone. 0.13 of CometKiwi is what it costs to break a glossary in a way the metric prefers. If your post-editing or automated QA stage optimises a quality score without an adherence constraint bolted alongside it, it is systematically removing the compliance you paid for.

Ordinary supervised fine-tuning was enough. BSC's WMT 2025 submission optimised the same model family with GRPO under a joint adherence and quality reward. On last year's benchmark the plain supervised v3.0 release improved on both axes, from 67.3 to 69.4 average chrF and from 91.3% to 94.0% average term accuracy, with no reinforcement learning, no task-specific adaptation and no access to the test-time domain. The authors are explicit that this is a system-level comparison rather than a controlled ablation, because the broader instruction mixture also changed. Read it as evidence that the expensive machinery was not the thing doing the work.

A judgement, marked as one: the gold row at 96.4% is the number that will end up in contracts, and it should. It establishes that glossary adherence is a rate, not a guarantee, for humans and machines alike, and it puts the current best system 0.7 points behind a professional translator on the metric localization buyers actually audit.

Check it yourself

The model is a public release, the evaluation set is on GitHub, and the filter is ten lines.

# the released model used, unmodified, as the WMT26 Track 1 submission
# https://huggingface.co/BSC-LT/salamandraTA-7b-instruct

# the evaluation set for the ablation (en-de, en-es, en-ru)
git clone https://github.com/wmt-conference/wmt25-terminology

# the filter, in full:
#   1. translate each glossary-annotated source with YOUR current model,
#      unconstrained, no glossary in the prompt
#   2. lenient match the output against the prescribed target term
#      (ignore casing, punctuation, spacing)
#   3. if the term is already there, discard the example as easy
#   4. keep only the mismatches, and train on those
# Report the hard share. BSC measured 52.5% over 19 directions
# (522,276 hard against 471,717 easy), 32.4% for de-en, 63.7% for es-hu.

The claim worth reproducing on your own data is not the +11.2. It is your own hard share. That single number tells you whether the filter is worth running: at BSC's natural 52.5% an unfiltered random draw already reached 88.5 of the 89.9, so the filter added 1.4 points. On a corpus that is 80% easy, which is what a disciplined translation memory looks like, the gap should be much wider. Measure it before you build anything.

Two caveats the authors state and that should travel with any quotation of these figures. The ablation runs do not reproduce the released v3.0 recipe: the background mixture follows the earlier v2.0 recipe and the terminology component uses EMEA data only, at a fixed 76,000 instances, so the hard-example ratio is the only moving part. And the synthetic terminology data behind the release, 33,615 instances covering 94 directed pairs across 29 languages, was generated by Gemma-4-31B from MeSpEn medical glossaries and the EMEA parallel corpus, with a LaBSE similarity gate of 0.80 on both the term pairs and the full texts. It is medical-domain synthetic data, and the gains reported transfer out of that domain but were not measured in yours.

What would prove this wrong

The claim under test is that hardness-filtered selection, defined against the model's own output, is what produces the gain, rather than any incidental property of the examples that survive. It is wrong if, by 9 September 2027, a published replication at fixed data volume on the WMT 2025 Terminology Track 1 set shows a filtered mixture within 3 points of an unfiltered mixture drawn from a corpus whose natural hard share is below 30%. The BSC corpus sat at 52.5%, which is why their random baseline came close; a genuinely easy corpus is the discriminating test and nobody has run it.

A second prediction, marked as judgement: the human gold reference will score below 98% term success again at WMT 2027. It scored 96.4% in 2026 and 93.6% into Polish. If organisers publish a gold row above 98%, either the glossaries got more injective or the reference production process changed, and the "adherence is a rate" framing above needs softening.

FAQ

How do you make a translation model follow a glossary?

Train it only on the examples it currently gets wrong. Filtering a terminology set to the examples where the model's own unconstrained translation contradicts the prescribed term raised term accuracy from 78.7% to 89.9% at identical data volume, +11.2 points (95% CI [+9.5, +12.8], p < 0.0001). A quarter dose delivered 71% of the effect.

How accurate is glossary adherence at the state of the art?

The best of 22 WMT 2026 Track 1 submissions reached 95.7% lemmatised exclusive term success at 75.3 document chrF++. The human gold reference scored 96.4% on average and 93.6% into Polish, so the machine-to-human gap on terminology was 0.7 points and the ceiling is not 100%.

Why do quality metrics disagree with glossary compliance?

Because glossaries are often many-to-one. When two English synonyms share one prescribed target, a faithful translation must repeat that target. CometKiwi scored the smoothed, adherence-breaking variant 0.13 higher, and across BSC's post-editing stage it consistently rewarded the edits that damaged adherence.

Does reinforcement learning beat supervised fine-tuning for terminology?

Not in this comparison. The same team's GRPO system from WMT 2025 was beaten on both axes by their plain supervised release, 67.3 to 69.4 chrF and 91.3% to 94.0% term accuracy. The authors call it a system-level comparison rather than a controlled ablation.

How much of a training corpus is usually "hard"?

In BSC's EMEA-derived pool, 52.5% across 19 directions: 522,276 hard against 471,717 easy, ranging from 32.4% for German into English to 63.7% for Spanish into Hungarian. The filter is worth most on corpora with a low hard share.

Sources

1. Xixian Liao and Maite Melero, Barcelona Supercomputing Center, SalamandraTA at WMT 2026 Terminology Shared Task: Hard Examples Are Better Teachers, arXiv:2609.09999, submitted 9 September 2026. Table 1 (official WMT 2026 Track 1 results, top eight of 22); Section 4 (controlled study setup, six mixtures, fixed 76,000-instance terminology component); Section 6.2 (comparison with the WMT 2025 GRPO system); Section 7 (many-to-one glossaries and the CometKiwi 0.13 delta); Table 3 (per-direction hard and easy pool sizes); Table 4 (full ablation numbers). PDF.
2. Charkiewicz et al., Findings of the WMT 2026 Terminology Shared Task, 2026. The official results table reproduced in Table 1 above. Task page: statmt.org/wmt26.
3. BSC-LT/salamandraTA-7b-instruct on Hugging Face. The public v3.0 release submitted unmodified to Track 1, and the v2.0 tag used to compute the hard/easy filter.
4. wmt-conference/wmt25-terminology on GitHub. The English into German, Spanish and Russian test set used for the controlled study, and its official term accuracy scorer.
5. Source data for the synthetic terminology pipelines: MeSpEn bilingual medical glossaries (Villegas et al., 2018) and the EMEA parallel corpus via OPUS (Tiedemann, 2012).

Related BLOMEGA guides: Multilingual LLM judges and translationese bias · Consented AI training data providers · How to license AI training datasets.