TermJudge ranks first on MT terminology only while one-off terms count, and its code repo is empty

TermJudge's first place rests on terms that appear once per document

The paper evaluates on 50 English to French biomedical documents from bio-MQM, 10 systems (8 MT plus 2 human references), and 1,320 aligned term occurrences per system (Table 2). In the headline configuration (Table 4) TermJudge leads all 11 metrics on SPA. In a "no-hapax" variant that keeps only terms with two or more occurrences in the document (48 documents, 275 segments), the ordering reverses at the top. We ranked the 11 SPA values in each of the paper's six meta-evaluation tables. With hapaxes TermJudge is 1st in all three aggregations. Without them it is 6th (occurrence, Table 18), 5th (source form, Table 19) and 8th (concept, Table 20), and first-translation glossary conformity is 1st in all three.

The authors state this in Appendix H: the reordering "is not significant" (p of 0.17 or more for every pair involving TermJudge). That is the fair reading of the no-hapax tables. It also applies to the headline: with only 45 system pairs, the chance level is 0.592 with a spread of 0.089 (Table 4 caption), and TermJudge's 0.041 margin over GEMBA-div is worth about 1.8 pairs of agreement (0.041 × 45, our arithmetic). The paper already marks that comparison, and the one against CometKiwi-div, as ties.

Our judgement on why: for an out-of-glossary term that occurs once, conformity metrics have nothing to compare against, so they cannot penalise it, while an LLM judge can. The paper says as much ("only a judgement of the translation itself, by an LLM or a human, can evaluate them"). So the lead measures that TermJudge scores terms the baselines ignore, which is real value, but it is also the slice where the judge is least accurate: binary error agreement on STEP is 0.664 for hapaxes against 0.787 for repeated terms (Table 3).

What the TermJudge authors published on 28 September 2026

arXiv:2609.35017, TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation, by Nicolas Dahan, François Yvon and Rachel Bawden (Inria Paris; Sorbonne Université, CNRS, ISIR), submitted 28 September 2026 and funded by the ANR MaTOS project. It builds on the same group's Improving Term Evaluation in Machine Translation: Variation Matters (AMTA 2026), whose PARA NLP and IWSLT23 outputs it re-scores.

The pipeline (Section 3.2) fixes a reference translation R(s) per source form: the glossary entry if one exists, otherwise the system's own most frequent rendering, checked by the judge. An exact lemma match is CONFORMING with no LLM call. A divergence goes to an error judge (Q1, 7 wrong-equivalent labels A1 to A7 plus 6 grouped families C to H, penalties 1 to 4) and, if error-free, to a consistency judge (Q2, JUSTIFIED at penalty 0 or UNJUSTIFIED at 0.5). Terms that occur once always go to Q1. Both judges are gpt-4.1-mini at temperature 0 (Section 3.3, Appendix L).

The paper promises two artefacts: open-source code at github.com/nicolasdahan/termjudge (footnote 1) and BioMQM-Terms, a 729-concept English to French biomedical glossary with 13,200 aligned term to translation pairs over ten systems, "released with our code" (Section 1). The underlying bio-MQM data is public on Hugging Face as zouhar/bio-mqm-dataset; the term layer on top of it is the new contribution.

The paper's numbers against its release and its own tables

All paper values from arXiv:2609.35017v1. Ranks, sums, Kendall tau and release checks are BLOMEGA computations, 30 September 2026.

The empty repository is the finding with the most practical weight. On 30 September 2026 the GitHub API reported size: 0, pushed_at 2026-09-12T00:48:33Z, no branches, and "Git Repository is empty" for the commit list. Until it fills, nobody outside the authors can re-run the judge, recompute SPA, or inspect the saved verdicts that the penalty-sensitivity study in Appendix G re-scores. The pipeline also depends on gpt-4.1-mini for alignment and variation labelling (Appendix A), so the prompts in Appendix L are necessary but not sufficient to reproduce a run.

One internal inconsistency, small but checkable: Section 5.3 says that on IWSLT23 glossary conformity "ranks Qwen3-8B first among the baseline runs, where TermJudge ranks it last". Table 6 gives conformity 0.731 for EuroLLM-22B and 0.723 for Qwen3-8B, so conformity ranks Qwen3-8B second. The contrast the authors draw still holds (second against last), but the sentence does not match the table.

We also cross-checked Table 7 against Table 22 and found them consistent. The four baseline systems on PARA NLP receive 1,393, 1,471, 1,537 and 1,481 Q2 judgements, a mean of 1,470.5, which is 13.2% of 11,122 occurrences; Table 7 gives JUSTIFIED 11.6% plus UNJUSTIFIED 1.6%, also 13.2%. The glossary-prompted runs give 10.5% both ways.

How a term occurrence reaches the gpt-4.1-mini judge

Every non-zero penalty in TermJudge is an LLM verdict, because CONFORMING and JUSTIFIED both score 0. On PARA NLP baseline runs, Table 7 puts 72.2% of occurrences at CONFORMING, which includes error-free hapaxes the judge already saw. The remaining 27.8% all carry a judge verdict, and the consistency judge excuses 41.7% of those as JUSTIFIED (11.6 of 27.8, our arithmetic). On IWSLT23 baseline runs it excuses 28.3% (9.2 of 32.5).

That forgiven share is where TermJudge differs from counting. A conformity metric marks all divergences as failures; TermJudge clears roughly 3 to 4 in 10 of them, and "document usage" alone accounts for 76.5% to 81.0% of all consistency-judge labels (Table 22). Everything else in the ranking is carried by the error labels, and those move the same way conformity does: glossary prompting cut wrong-equivalent errors from 11.7% to 8.4% on PARA NLP and from 18.6% to 9.3% on IWSLT23 (Table 7).

So how different is the ranking? Over the eight outputs per corpus we counted pairwise order agreement between the two metrics. On PARA NLP they agree on 27 of 28 pairs; the only swap is EuroLLM-9B against EuroLLM-22B at baseline. On IWSLT23 they agree on 22 of 28, with three flips inside each prompting condition. For scale, TermJudge's own default and exhaustive judging paths agree on 25 of 28 IWSLT23 pairs (Table 21), so conformity sits 3 pairs further from TermJudge than TermJudge sits from itself. Both metrics put every glossary-prompted run above every baseline run. General QE does not: CometKiwi prefers the baseline run in 11 of 16 mixed pairs on IWSLT23 (Section 5.3).

What this means for localization teams doing terminology QA

• For ranking MT vendors or prompts on repeated terminology, exact glossary conformity is close to free and close to TermJudge. 27 of 28 pairs on PARA NLP, and no significant SPA gap on repeated terms. Our judgement: run conformity first and spend judge calls only where it cannot decide.
• Spend the LLM judge on one-off and out-of-glossary terms. That is where TermJudge's measured advantage comes from and where conformity is blind. Budget human review there too, since hapax accuracy on STEP is 0.664 and the judge accepted 40 of 65 invented calques.
• Use the JUSTIFIED labels as a glossary backlog, not only as forgiveness. "Document usage" dominates the clearances (Table 22, 76.5% to 81.0% of all Q2 labels). A consistent, correct rendering that is not the glossary term, such as traitement where the entry says thérapeutique (Section 5.4), is a candidate glossary variant.
• Cost is small next to review. Appendix K: about $50 of judging plus $33 of alignment for eight PARA NLP outputs (137,123 source words each), $6.8 plus $4 for eight IWSLT23 outputs, $6 plus $5 for BioMQM-Terms. By our division that is about $0.08 to $0.15 per 1,000 source words at gpt-4.1-mini list price, with alignment 37% to 45% of the bill.
• Treat judge-model dependence as unknown. One proprietary judge, no open-weight judge, and stability across judges "was not measured" (Limitations). Our earlier note on multilingual LLM judges agreeing with themselves at kappa 0.24 is a reason to pin the judge model and prompt version in any QA contract that uses scores like these.

For the production side of the same problem, getting systems to follow a glossary in the first place, see how WMT 2026 glossary training raised term accuracy from 78.7% to 89.9%. TermJudge's result that glossary injection helped all four models on both corpora (Table 6) is consistent with that note.

Check it yourself

The release state takes one command each:

curl -s https://api.github.com/repos/nicolasdahan/termjudge | grep -E '"(created_at|pushed_at|size)"'
curl -s https://api.github.com/repos/nicolasdahan/termjudge/contents/
# 30 Sep 2026: size 0, pushed_at 2026-09-12T00:48:33Z, "This repository is empty."
curl -s "https://gitlab.inria.fr/api/v4/projects/almanach%2Fconcordancer" | grep -o '"last_activity_at":"[^"]*"'

The ranks come straight from the paper's tables. Copy the SPA columns and the Table 6 rows:

python3 - <<'PY'
from itertools import combinations
names=['TJ','GEMBAdiv','MXdiv','CKdiv','conf_first','conf_major','GEMBAany','GEMBAcat','GEMBAgen','MXgen','CKgen']
spa={'T4':[.844,.803,.769,.786,.730,.723,.775,.689,.736,.736,.729],
     'T18':[.750,.725,.752,.752,.778,.770,.775,.689,.736,.736,.729],
     'T19':[.741,.708,.750,.735,.785,.776,.730,.646,.721,.728,.747],
     'T20':[.731,.717,.754,.744,.794,.785,.725,.641,.736,.744,.766]}
for t,v in spa.items(): print(t,'TermJudge rank',1+sum(x>v[0] for x in v),'of 11; top',names[v.index(max(v))])
tj={'PARA':[.322,.350,.287,.312,.220,.214,.236,.224],'IWSLT':[.468,.502,.437,.439,.236,.245,.330,.230]}
cf={'PARA':[.773,.768,.776,.783,.851,.864,.826,.844],'IWSLT':[.713,.723,.719,.731,.848,.850,.788,.833]}
for c in tj:
    agree=sum((tj[c][j]-tj[c][i])*(cf[c][i]-cf[c][j])>0 for i,j in combinations(range(8),2))
    print(c,'pairs agreeing',agree,'of 28, tau',round((2*agree-28)/28,3))
PY

Expected: T4 rank 1; T18 rank 6, T19 rank 5, T20 rank 8, top conf_first in all three; PARA 27 of 28 (tau 0.929); IWSLT 22 of 28 (tau 0.571). The Qwen3-8B sentence is in Section 5.3, page 9 of the PDF; Table 6 is on page 10 (pdftotext -f 10 -l 10 -layout 2609.35017.pdf -).

What would prove this wrong

Our claim is that TermJudge's measured advantage over glossary conformity comes from single-occurrence terms, and that on repeated terms the two metrics rank systems nearly the same way.

It would be wrong if a TermJudge evaluation on a second human-annotated benchmark, or a re-run on BioMQM-Terms with the released verdicts, shows TermJudge significantly ahead of first-translation conformity on SPA (p below 0.05) with hapax chains excluded. We predict that by 31 March 2027 no such result will be published, and that when the repository fills, recomputing Tables 18 to 20 from its data will still put conformity at or above TermJudge on repeated terms. A published TermJudge score from a second judge model, open-weight or not, that reorders the eight IWSLT23 outputs on more than 3 of 28 pairs would also tell buyers that the judge, not the method, is setting the rank.

Sources

1. Nicolas Dahan, François Yvon, Rachel Bawden. TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation. arXiv:2609.35017v1, 28 September 2026. Sections 3 to 5, Limitations, Tables 1 to 7, Appendices A, D, H, I, J, K (Tables 15 to 22).
2. github.com/nicolasdahan/termjudge, checked via api.github.com on 30 September 2026: created 12 September 2026, empty.
3. almanach/Concordancer on gitlab.inria.fr, public, last activity 17 December 2025, checked 30 September 2026.
4. Dahan et al. Improving Term Evaluation in Machine Translation: Variation Matters. arXiv:2609.08779, AMTA 2026 (ACL Anthology). Source of the PARA NLP and IWSLT23 outputs.
5. Zouhar et al. bio-mqm-dataset on Hugging Face, last modified 23 July 2024; the MQM annotations under BioMQM-Terms.
6. BLOMEGA computations, 30 September 2026: SPA ranks across Tables 4 and 16 to 20, pairwise agreement and Kendall tau from Table 6, Table 7 sums, Table 22 cross-check, cost per 1,000 source words from Table 2 and Appendix K.

FAQ

Does TermJudge beat glossary-conformity metrics for MT terminology?

On BioMQM-Terms with all occurrences, yes: system-level SPA 0.844 against 0.730 for first-translation conformity, significant for 8 of 10 comparison metrics (Table 4). With single-occurrence terms removed (Appendix H, Tables 18 to 20) TermJudge scores 0.750, 0.741 and 0.731 across three aggregations and ranks 6th, 5th and 8th of 11, with conformity nominally first; the paper reports none of those differences as significant (p of 0.17 or more).

How much of the TermJudge score comes from the LLM judge?

All of the penalty does: a CONFORMING match scores 0 without an LLM call, so every non-zero penalty is an LLM verdict. From Table 7, 20.8% to 32.5% of occurrences end with a non-conforming verdict (27.8% on PARA NLP baseline runs), and the consistency judge excuses 28% to 43% of those as JUSTIFIED. The judge also decides error-free hapaxes, which are counted inside CONFORMING, so the true judged share is higher.

Can I run TermJudge today, and what does it cost?

Not from the release: github.com/nicolasdahan/termjudge, cited in the paper for code and the BioMQM-Terms data, was created on 12 September 2026 and returned 'This repository is empty' on 30 September 2026. The upstream CONCORDANCER term spotter is public on gitlab.inria.fr. Appendix K puts gpt-4.1-mini list-price cost at about $83 for eight PARA NLP outputs (judging plus alignment), roughly $0.08 per 1,000 source words by our division.