Ten points of F1 that frontier-LLM labels alone never bought a multilingual PII tagger

What changed, and when

Jonathan Graehl posted Strong Multilingual Privacy Tagging at Encoder Speed (arXiv:2609.38630) on 29 September 2026, with revisions through v3 on 13 October 2026. The system is a named-entity tagger for privacy redaction, fine-tuned from XLM-R-large (0.55B parameters), covering 35 languages.

Training data has two sources. The larger one is silver labels: text annotated by prompted frontier LLMs (mainly one labeled "GPT-6-Luna" in the paper, with other model generations contributing a development set), totaling 49,812 unique sentence-oriented training texts in the final model. The smaller one is mapped gold: human-annotated corpora, MAPA, AQMAR, OpenNER and Wojood, with their existing entity labels remapped onto this paper's own entity schema, totaling 80,099 unique texts across the full set of training configurations tested. The paper's central ablation compares models trained with different proportions of each.

The evidence table

Source: arXiv:2609.38630v3, abstract and Section 5. Gold-7 is the paper's human-annotated test set spanning 7 languages.

Source: arXiv:2609.38630v3, Table 2. The open-weight LLMs tested also scored lower on F1 (65.1 and 56.5 respectively) and are not on the same evaluation footing as the two rows above them.

A model can look validated while grading its own labels

The paper's Section 5.2 isolates what the 88.2 F1 headline obscures: training data is a mix of LLM-generated silver labels (49,812 texts) and human-annotated gold data mapped from MAPA, AQMAR, OpenNER and Wojood onto this paper's schema (80,099 texts across configurations). A model trained on silver labels alone can be evaluated two different ways, against more silver-labeled text, or against the human-annotated gold test set, and those two evaluations answer different questions. The silver-against-silver score measures whether the model learned to reproduce the LLM's own labeling pattern, including the LLM's own mistakes. The gold score measures whether the model's labels match a human's.

The paper reports that adding the mapped gold data to training raised F1 by ten points specifically on the gold evaluation, "without reducing it on silver evaluation text." That asymmetry is the finding: the LLM-only model was not visibly deficient when graded by the same kind of label it was trained on. It became visibly deficient only once measured against a human.

What it means for anyone buying LLM-labeled training data

Ask what the evaluation set is made of, not just what score it reports. An 88.2 F1 claim is meaningless on its own if the test set shares a labeling source with the training set. This paper is unusually transparent about running both evaluations side by side; most "LLM-as-annotator" releases report one number against one test set, and do not say which kind.

A small amount of real human annotation is disproportionately valuable, specifically as a check, not just as more data. 80,099 mapped human-annotated texts against 49,812 LLM-labeled ones is not a small corpus, but it is not the dominant one either, and it bought ten F1 points exactly where it was needed, on the evaluation that matters. The implication for a buyer with a fixed annotation budget is to spend some of it on human gold specifically to validate against, even if the bulk of training labels stay LLM-generated.

Speed and accuracy are not the same claim, and this paper's throughput numbers are real but separate from the F1 story. 4.9 times GLiNER2's CPU throughput is a genuine engineering result (a smaller, purpose-built encoder beating a more general one on latency), independent of whether the labels used to train either system were trustworthy.

Our judgement: this is a single-author paper without a named-institution evaluation, and the "essentially no change" claim on silver evaluation is reported, not shown with a fully broken-out number in the excerpt we could verify; the ten-point gold figure is the one stated as an exact result in Section 5.2.

Check it yourself

The human-annotated source corpora are all public, and the comparison that matters is the LLM-only versus LLM-plus-gold ablation, not the headline F1 number alone.

open https://arxiv.org/abs/2609.38630          # abstract, author, version history (v1 29 Sep, v3 13 Oct 2026)
open https://arxiv.org/html/2609.38630v3       # full text: Table 2 (throughput), Section 5.2 (the ablation)

# the four human-annotated corpora mapped onto this paper's entity schema
# MAPA, AQMAR, OpenNER, Wojood

python3 - <<'PY'
presidio, gliner2, tagger = 61.4, 69.1, 88.2
print("tagger vs GLiNER2: %+.1f F1" % (tagger - gliner2))
print("tagger vs Presidio: %+.1f F1" % (tagger - presidio))

cpu_tagger, cpu_gliner2 = 675, 138
print("CPU throughput ratio: %.1fx" % (cpu_tagger / cpu_gliner2))
PY
# tagger vs GLiNER2: +19.1 F1
# tagger vs Presidio: +26.8 F1
# CPU throughput ratio: 4.9x

These are direct differences and ratios between the paper's own reported numbers. We did not retrain the tagger or re-run GLiNER2 and Presidio ourselves.

What would prove this wrong

The ten-point gold gain is reported from this paper's own ablation, on its own Gold-7 test set, built from four specific human-annotated corpora the author chose to map onto a custom entity schema. A different choice of human test corpus, or a different privacy-entity schema, could show a smaller or larger gap between LLM-only and LLM-plus-gold training.

A dated prediction: by 31 December 2027, an independent replication on a different human-annotated multilingual PII test set will report whether LLM-labeled-only training closes to within a few F1 points of mixed training, as frontier labeling models improve, or whether the gap this paper measures persists. If the gap closes substantially with a newer labeling model and no added human gold, the specific claim that human annotation is what bought the ten points, rather than simply more data of any kind, would need revisiting.

Sources

1. Graehl, J. Strong Multilingual Privacy Tagging at Encoder Speed. arXiv:2609.38630, v1 29 September 2026, v3 13 October 2026. Table 2, Section 5.2; HTML version.
2. Source human-annotated corpora mapped onto this paper's schema: MAPA, AQMAR, OpenNER, Wojood, as listed in arXiv:2609.38630v3.
3. BLOMEGA. LLM annotation source, cost and quality on ASSISTments.
4. BLOMEGA. Consented AI training data providers.
5. BLOMEGA. Data annotation research: the latest.