TeochewBench: returning the Teochew source unchanged beats 2 of 11 LLMs at translating it into Mandarin

What TeochewBench is, as of 16 September 2026

arXiv:2609.18156 is a single-author, 11-page paper. Teochew (Chaozhou) is a Southern Min variety of the Chaoshan region of Guangdong and its diaspora, written in Hanzi that overlap heavily with Mandarin while carrying different words, particles, idioms and meanings. The benchmark has 300 items in five categories: 60 basic vocabulary, 100 everyday sentences, 60 Teochew-specific expressions, 50 tone, politeness and context, and 30 idioms, ambiguous and culturally specific expressions. Candidates were compiled from public dictionaries including PUJDICT; one native Teochew speaker reviewed and revised all 300, and two more checked selected items. No inter-annotator agreement is reported because no second reviewer covered the full set.

Every item also carries a textual-specificity label: low (no Teochew-specific feature, nearly identical to Mandarin at the character level, 82 items), ordinary (one feature, 146), medium (two, 39), high (three or more, or slang, idiom, allusion or non-literal meaning, 33). The 11 main models produced 6,600 predictions and two base checkpoints another 1,200. The paper gives no download link for the dataset.

Two instruction models score below the copy-the-source control into Mandarin

The paper leaves the all-item copy score to the reader. The arithmetic is exact because chrF-style is averaged per item: (82 x 67.69 + 146 x 21.86 + 39 x 11.23 + 33 x 8.16) / 300 = 31.50. The same weighting of the exact-match rates (52.4% of 82 low items, 3.4% of 146 ordinary items) gives about 48 items where the untouched Teochew source equals a Mandarin reference after normalisation.

Seed-OSS-36B-Instruct is the odd case. It scores 54.05 into English, close to the leaders, and 30.95 into Mandarin, below the copy. The paper does not explain the 23.1-point gap; the author stripped </seed:think> tags and kept malformed outputs in the evaluation with zero scores for empty ones. We could not check the raw outputs because they are not released.

At the top, the ranking is Qwen3.5-27B (60.63 overall), Qwen2.5-72B-Instruct (56.61), Gemma-3-27B-IT (56.36), GLM-4-32B-0414 (55.82). The author declines to read small gaps as capability differences, which is right: a copy baseline at 31.5 means the first thirty or so points into Mandarin are available without any translation.

A shared script turns character-overlap metrics into a measure of how Mandarin-like the source already is

chrF counts character n-grams (orders 1 to 6 here, beta = 2) shared between output and reference. Between two languages in different scripts, overlap has to be earned by translating. Between Teochew Hanzi and Mandarin, much of it is present before the model does anything. The item-level copy score correlates with the specificity tier at Spearman rho = -0.634.

There is a confound the author flags: specificity and content category are associated (Cramér's V = 0.292), so part of the tier effect is category mix. Nine of the 11 models still show a strictly falling score from low to high, and 10 of 11 score lowest on the high tier.

If you evaluate a regional variety that shares a script with a big language, ship a copy baseline with every score

The problem is not specific to Teochew. Any pair where source and target share a script and much vocabulary has it: Cantonese or Hokkien written in Hanzi into Mandarin, Galician into Portuguese, Malay into Indonesian, Serbian into Croatian in Latin script. A vendor or model that "scores 55 chrF" on such a pair has told you little until you know what the unchanged source scores.

Report the copy control per tier and overall. TeochewBench does the first. Do both, so a reader can see that 31.5 of the Mandarin score is free.

Weight the hard items. The 33 high-specificity items are where Teochew knowledge shows, and they are 11% of the set. A benchmark built to rank models on a variety should be majority hard items, or report the hard tier as its headline.

Use humans on the tail. String metrics under-reward valid paraphrase: 240 of 300 items have one English reference. The author plans blinded human evaluation; until then, the high tier is the part of this benchmark worth reading, and it says every tested model is weak there. A judgement: for a localization team, that is the actionable result. None of the 11 open models can be trusted with Teochew idiom or politeness without a native reviewer.

Check it yourself

# The all-item copy score and exact-match count, from the paper's Section 5.3 tier table
python3 - <<'EOF'
tiers = [(82, 67.69, 0.524), (146, 21.86, 0.034), (39, 11.23, 0.0), (33, 8.16, 0.0)]
copy = sum(n * c for n, c, _ in tiers) / 300
exact = sum(round(n * e) for n, _, e in tiers)
print(round(copy, 2), exact)
zh = {"Seed-OSS-36B-Instruct": 30.95, "MiniCPM4-0.5B": 26.27, "Qwen2.5-7B-Instruct": 44.07}
print({m: round(v - copy, 2) for m, v in zh.items()})
EOF
# 31.5 48
# {'Seed-OSS-36B-Instruct': -0.55, 'MiniCPM4-0.5B': -5.23, 'Qwen2.5-7B-Instruct': 12.57}

To add the same control to your own benchmark, score the source text as if it were a system output: with sacrebleu, CHRF().corpus_score(sources, [references]).score. Note that TeochewBench's metric is a local chrF variant averaged per prediction (orders 1 to 6, beta 2, NFKC normalisation, case folding, non-alphanumerics stripped, best of multiple references), so sacrebleu's corpus-level chrF will not reproduce its numbers exactly. The paper's prompts are printed verbatim in Appendix A of https://arxiv.org/pdf/2609.18156.

What would prove this wrong

Our reading is that TeochewBench's aggregate chrF-style mostly measures overlap with Mandarin and that its high tier, not its headline, ranks Teochew ability. The author's planned blinded human evaluation is the test. If, when published, human rankings of the 11 checkpoints correlate with overall chrF-style at Spearman 0.8 or higher, the aggregate is a better proxy than we claim. If they correlate better with the high-tier scores than with the overall scores, our reading holds. We will check for a v2 or companion paper by 30 June 2027.

Narrower: if the released dataset, once published, shows a Hanzi-copy Mandarin chrF-style over all 300 items that differs from 31.5 by more than 0.5, our weighting assumption (per-item averaging) is wrong.

Sources

1. Jianan Wu (Peking University), TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation, arXiv:2609.18156v1, 16 September 2026. Sections 3 to 7, Appendices A and B.
2. Maja Popović, chrF: character n-gram F-score for automatic MT evaluation, WMT 2015.
3. PUJDICT contributors, PUJDICT: Péh-Uē-Jī Dictionary of the Teochew-Swatow Dialect, GitHub.
4. Linrong Pan et al., Teochew-Wild: The First In-the-Wild Teochew Dataset with Orthographic Annotations, arXiv:2505.05056, 2025.

Related BLOMEGA guides: Japanese subtitle scores are a segmentation artefact · Low-resource speech translation: hours vs BLEU · Multilingual LLM judges and translationese bias

FAQ

What is TeochewBench?

A 300-item benchmark, posted to arXiv on 16 September 2026 by Jianan Wu of Peking University, for translating written Teochew (Chaozhou) Hanzi into Mandarin Chinese and English. Items are labelled by how Teochew-specific they are, and 11 instruction-tuned open models were evaluated with a chrF-style metric.

Which LLM translates Teochew best?

On TeochewBench's overall chrF-style score, Qwen3.5-27B leads at 60.63, followed by Qwen2.5-72B-Instruct (56.61), Gemma-3-27B-IT (56.36) and GLM-4-32B-0414 (55.82). On the 33 most Teochew-specific items all models score low and differ little.

Why does copying the source score well when translating Teochew into Mandarin?

Teochew and Mandarin share Chinese characters, so a character n-gram metric rewards overlap even when nothing is translated. Returning the Teochew source unchanged scores 67.69 chrF-style on the least specific items and about 31.5 over all 300 items, above two of the 11 models tested.

How should dialect or regional-variety translation be evaluated?

Report a copy-the-source control alongside every model score, stratify items by how different they are from the standard language, weight or headline the hardest tier, and add human review for idioms and politeness, where string metrics under-reward valid paraphrases.

Is the TeochewBench dataset available?

The paper gives no download link as of 20 September 2026. It describes the construction, review and scoring protocol and prints all prompts.