Eight Sinhala-to-English translators, one hallucination detector: firings range from 51 to 433 out of 1,485 sentences, and the paper says that is risk, not an error count

Detector firings vary nearly an order of magnitude by architecture, and the paper will not call that a hallucination rate

The benchmark runs eight Sinhala-to-English systems spanning five model families (NLLB-200 distilled 600M and 1.3B, M2M-100 418M and 1.2B, mBART-50 many-to-many and many-to-one, a community mT5 fine-tune, and Helsinki-NLP's OPUS-MT mul-en) on the same 1,485 unique test sources, all with beam search (4 beams, 256 max tokens). Table 6 reports how many sentences each system's detector flags: 51 for NLLB-200 600M, 70 for NLLB-200 1.3B, 93 for mBART-50 many-to-one, 129 for M2M-100 1.2B, 140 for mBART-50 many-to-many, 253 for M2M-100 418M, 329 for the mT5 fine-tune, and 433 for OPUS-MT. The two NLLB models, which report the strongest published FLORES-200 Sinhala scores, produce both the fewest detections and the highest per-word confidence (mean log-probability −0.103 to −0.132 correlation with the risk score). Scaling M2M-100 from 418M to 1.2B parameters roughly halves detections, 253 to 129, while the two NLLB sizes behave almost identically (51 and 70), suggesting architecture and training data matter more than raw parameter count here.

What Obeysekara and Jayatilleke built for a language pair with no prior benchmark

arXiv:2610.11389v1, Fact over Fiction: Detection of Pathological Hallucinations in Sinhala-to-English Neural Machine Translation, by Navam Obeysekara (Informatics Institute of Technology, Sri Lanka) and Nevidu Jayatilleke (University of Moratuwa, Sri Lanka), was submitted 8 October 2026. The paper's starting point is that existing hallucination benchmarks such as HalOmi exclude the Sinhala-English pair entirely, leaving no labelled data for this direction (most prior Sinhala NMT work studies English-to-Sinhala instead).

The method has two stages. A generation stage builds a 45,000-sample synthetic corpus by corrupting reference English translations through five linguistically motivated strategies (dependency swap, entity swap, numeric distortion, semantic drift, and high-temperature sampling), with a character-level "semantic rescue" step that keeps morphological variants (spelling differences, inflections) from being mislabelled as hallucinations. A detection stage fine-tunes mDeBERTa-v3 as a token-level sequence labeller on that corpus, combined in a three-signal ensemble with sequence log-probability and LaBSE cross-lingual sentence similarity.

The detector's own numbers, and the control that checks whether it is cheating

All values are as reported in arXiv:2610.11389v1, Tables 3, 4 and 6. "Not reported" rows reflect an explicit limitation stated by the authors, not a gap we found independently.

Why the control matters: a detector trained on monolingual corruptions could just be reading the English

Every training example is built by corrupting an English hypothesis sentence alone, never by touching the Sinhala source. That creates an obvious risk the paper names directly in §2.3: a supervised detector trained this way could learn to spot corruption artefacts in the English text itself (odd phrasing, antonym swaps) without ever using the Sinhala source to check faithfulness, the same failure mode documented for hypothesis-only baselines in natural language inference research. The source-ablation control tests this by keeping the English hypothesis fixed and either shuffling the paired Sinhala source among other test sentences or deleting it outright.

The result rules out the easy failure mode. A detector that only read monolingual artefacts would not care whether the attached source was correct or swapped; this one does, falling from AUROC 0.970 to 0.540 (shuffled) and 0.507 (removed), both within noise of random guessing. The hypothesis-only baseline (trained and evaluated with no source at all) still reaches F1 0.622, confirming the synthetic corruption process does leave some monolingual trace, which is why the full model's +0.22 F1 margin over it matters: that margin is specifically the part of detection that comes from checking the Sinhala source, not from spotting disfluent English.

What this means for a localization team shipping Sinhala-English translation

• Firing-rate rankings are a risk signal for system selection, not an error audit. The 8.5-fold spread (51 to 433) is a legitimate reason to prefer NLLB-200 over OPUS-MT or the mT5 fine-tune for this pair, but it is not a count of actual mistranslations: of 129 M2M-100 1.2B detections the authors could partially cross-check, only 27.9% also failed a LaBSE similarity threshold, with the remainder either fluent hallucinations or false positives.
• No single signal is sufficient on its own. Detector-only flags (22 to 97 sentences per system) catch confident, fluent hallucinations that log-probability misses entirely; log-probability alone fires on about 30% of all outputs by construction, far too liberal to use unfiltered. The three signals correlate only weakly (Spearman ρ between −0.10 and −0.44, all p<0.001), meaning they are genuinely complementary rather than redundant.
• Syntactic forcing is a known false-positive source specific to this language pair. Sinhala's diglossia means colloquial input often omits subjects or pronouns that formal training data always includes; when a model inserts a grammatically required English agent with no direct source counterpart, word-level alignment penalizes the insertion during training and the detector can flag a valid translation as a hallucination.
• The corpus and detector are released, the benchmark outputs are not labelled. The 45,000-row synthetic dataset, splits and code are public on GitHub with no stated licence file as of this check; a team adopting this detector inherits the same gap the authors flag, no verified precision number on real production translations.

How to rerun the benchmark and the ablation control

# The released synthetic corpus and results
git clone https://github.com/Imanjith/synthetic-sinhala-english-dataset
ls synthetic-sinhala-english-dataset/Results
# -> table3_detector.csv, table4_partitions.csv, table5_ensemble.csv, benchmark_outputs.csv

# Confirm release state and licence
curl -s https://api.github.com/repos/Imanjith/synthetic-sinhala-english-dataset \
  | python3 -c "import json,sys;d=json.load(sys.stdin);print(d['pushed_at'], d['license'])"
# -> 2026-09-25T18:34:40Z None

# Re-fetch the paper and confirm the Table 4 ablation numbers
curl -sL https://arxiv.org/pdf/2610.11389 -o sinhala.pdf
python3 -c "import pypdf; t='\n'.join(p.extract_text() for p in pypdf.PdfReader('sinhala.pdf').pages); i=t.find('Table 4'); print(t[i-600:i+200])"

The eight benchmarked NMT systems are all publicly available (NLLB-200, M2M-100, mBART-50 via Meta/Facebook research releases, the community mT5 fine-tune at huggingface.co/thilina/mt5-sinhalese-english, and Helsinki-NLP's OPUS-MT), so the firing-rate ranking in Table 6 can be reproduced end to end from public weights and the public test split.

What would prove this reading wrong

Our claim is that the detector's firing-rate ranking (NLLB-200 lowest, OPUS-MT and the mT5 fine-tune highest) is a defensible relative-risk signal, while the raw counts themselves (51 to 433) should not be read as hallucination rates without human verification, because the paper itself only partially cross-checked 129 of its flagged sentences. We would be wrong if a subsequent human-annotated study of the same eight systems' outputs on this exact 1,485-sentence test set found precision on natural (non-synthetic) text close to the detector's synthetic-test F1 of 0.841, which would mean the firing counts were a reasonably direct hallucination estimate after all. We predict that by 30 June 2027 no publicly available human-annotated precision estimate for this detector on natural Sinhala-to-English output will exceed 0.70, given that the authors' own partial check (LaBSE agreement on M2M-100 1.2B) found agreement with only 27.9% of flags.

Sources

1. Navam Obeysekara, Nevidu Jayatilleke. Fact over Fiction: Detection of Pathological Hallucinations in Sinhala-to-English Neural Machine Translation. arXiv:2610.11389v1, submitted 8 October 2026 (Informatics Institute of Technology, Sri Lanka; University of Moratuwa, Sri Lanka). §1-5, Tables 1-7.
2. github.com/Imanjith/synthetic-sinhala-english-dataset, checked via api.github.com on 13 October 2026: 27 files, pushed 25 September 2026, no licence file.
3. huggingface.co/thilina/mt5-sinhalese-english, community fine-tune benchmarked as one of the eight systems.
4. NLLB Team et al. No Language Left Behind, cited in arXiv:2610.11389v1 for NLLB-200's reported FLORES-200 Sinhala performance.
5. BLOMEGA computations, 13 October 2026: chart rendering of Tables 4 and 6, GitHub release-state check.

FAQ

Which Sinhala-to-English translation systems hallucinate the most?

On a reference-free detector benchmark (arXiv:2610.11389), OPUS-MT (Helsinki-NLP mul-en) and a community mT5 fine-tune produced the most detector firings (433 and 329 of 1,485 test sources), while NLLB-200 (distilled 600M and 1.3B) produced the fewest (51 and 70). The authors are explicit that these are detector-flagged sentences, which indicate relative risk, not a verified hallucination count, since no human annotation of the benchmark outputs was performed.

How accurate is the hallucination detector itself?

The mDeBERTa-v3 token-level detector reaches F1 0.841 (plus or minus 0.001 over three seeds) on a source-disjoint synthetic test set, 5.2 points above an XLM-RoBERTa baseline. A source-ablation control shows this depends on the actual Sinhala source: shuffling or removing the source drops token F1 to about 0.30 and sentence-level AUROC from 0.970 to chance (0.54 and 0.51).

Is detector firing rate the same as hallucination rate?

No, and the paper says so directly. Without human annotation of the benchmark outputs, precision on natural translations cannot be measured. Of 129 detector-flagged M2M-100 1.2B sentences, only 36 (27.9%) also fell below a LaBSE cross-lingual similarity threshold; the rest are either fluent hallucinations that preserve overall meaning, or false positives from lexical paraphrasing and valid syntactic insertions.