A model that never sees the Chinese scores 76.5 on CIBuzzBench's Chinese task

A model that never sees the Chinese scores 76.5 on the Chinese task

CIBuzzBench was posted to arXiv on 18 September 2026: 3,001 Chinese internet buzzwords with English meanings, English equivalents, category labels and harmfulness labels, and three tasks built on them. Task 2, Equivalent Selection, shows a model the Chinese buzzword and four English phrases and asks which is the gold equivalent. Six frontier models score between 79.66 and 88.03 macro F1 on it.

We ran the same task with the Chinese term deleted from the prompt. A model shown nothing but the four English options, and told the published recipe for how the three distractors were built, scores 76.50% accuracy and 76.51 macro F1. The same model with the Chinese term scores 86.50%. So 88.4% of the sighted score is available with no Chinese input at all, and the blind score lands 3.2 points below the weakest of the six published models.

The English option set, not the Chinese buzzword, is carrying most of this task.

What was released on 18 September 2026

arXiv:2609.21722, CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords, by Yifan Wang, Junyu Lu, Qifan Wang, Shun Zhang, Chaozhuo Li, Jiahao Liu, Zhijun Cao, Lingbin Bu and Fanliang Bu. The data and evaluation code are at github.com/SuperYFan/CIBuzzBench, last pushed 21 July 2026.

The construction is documented honestly. Chinese buzzword and explanation pairs were collected from Chinese buzzword dictionary sites. GPT-5.5 drafted the English annotations for all three tasks. Three NLP graduate students, one per task, reviewed the drafts; a seven-member group confirmed accepted cases and voted on disputes. Independent validation on stratified samples: annotators picked the gold equivalent in 95.3% of 900 judgements, with 91.7% three-way exact agreement, and Fleiss' kappa 0.6920 on harmfulness.

Two facts about the task design matter for what follows. First, the evaluation prompt shows the model only term_zh; the Chinese explanation that anchors the gold sense is never in the prompt. Second, the metric is macro F1 over the four option letters A, B, C and D, computed by classification_metrics() in scripts/evaluate_api.py, with the option order shuffled by a per-row seeded RNG. We reused the same shuffle, so our numbers are comparable to theirs.

The blind probe

We sampled 400 of the 3,001 items with a fixed seed, built the option text with the benchmark's own build_options(row, 1111), and ran two conditions against the same model on the same items: sighted, which is the benchmark prompt, and blind, which deletes the Chinese term and instead names the three distractor types. The model is Gemini Flash reached through our internal gateway at temperature 0.4. Both conditions parsed 400 of 400 responses.

Paired, the two conditions pick the same letter on 326 of 400 items (81.5%). Seeing the Chinese fixes 54 items and breaks 14, McNemar p = 1.1e-06. The Chinese term helps, significantly and repeatably. It is just not doing most of the work.

When the blind model is wrong, it is almost always wrong in one direction: 60 of its 94 errors pick the pragmatic-neighbour distractor, against 19 cultural mismatches and 15 literal glosses. The literal gloss, the distractor type the paper's failure analysis dwells on, is the easiest one to reject from English alone.

Why the option set leaks

The three distractors are generated by a fixed, published recipe: a literal or surface-form gloss, a cultural mismatch, and a pragmatic neighbour with a related but different speech act. A recipe that consistent produces a consistent signature. We measured it across all 3,001 items.

Picking the longest option by character count scores 41.39 macro F1 against a uniform-random 25.53, averaged over five shuffle seeds. That is 16 free points with no model, no Chinese and no English comprehension. The gold is strictly the longest option in 1,180 of 3,001 items (39.3%) and longer than the pragmatic neighbour in 75.2%.

Length is only the crudest channel. A language model reading four English phrases can also see which one reads like a curated idiom gloss and which one reads like a deliberately-wrong neighbour. That is the gap between 41 and 76.

The data itself is clean on every axis we could check. No option set contains a duplicate string. All 3,001 Chinese terms are distinct. No test term appears in the training split. Only 2.80% of distractor strings appear as the gold for some other item. The leak is not sloppiness; it is what happens when one generator writes both the answer and the wrong answers.

The Chinese term earns its keep on puns, and only there

Splitting the paired probe by the benchmark's own category label makes the shape of the task visible.

Category and harmfulness counts from arXiv:2609.21722 Table 1, independently recounted from data/cibuzzbench.csv and matching exactly. Blind and sighted columns are our measurement, 22 September 2026.

Homophonic puns are the 13.3% of the benchmark where the meaning is carried by Chinese sound, so an English-only reader genuinely cannot get there. That subset shows a 17.6-point blind-to-sighted gap, nearly double every other category. On Slang and Abbreviation the blind model is already at 82 to 83%.

This also lines up with the paper's own Table 6, which reports that Homophonic pun and Stylistic device get the lowest judge scores on the open-ended Meaning Explanation task. The hard subset is consistent across tasks. It is just small.

The harmfulness task, for contrast

Task 3 asks whether a buzzword's documented sense is harmful. The labels are 2,574 non-harmful and 427 harmful, which we recounted from the CSV and which matches Table 1 exactly at 85.8% and 14.2%. Always answering non-harmful therefore scores 92.34 F1 on the majority class, 0 on the minority, and 46.17 macro F1.

The six models score 70.41 to 80.45 macro F1 on it. That is 24 to 34 points clear of the trivial baseline, which is a real result and a much healthier one than Task 2's. The interesting number there is the harmful-class F1: 48.61 to 66.17, averaging 57.01 under English prompts. Against 427 harmful items, an F1 around 57 means a safety filter built on this behaviour misses a large fraction of the coded insults, which is the failure mode the paper opens with.

Harmfulness is not evenly spread. Homophonic puns are 24.4% harmful and Abbreviations 21.3%, against 7.1% for Quotations. Coded hostility hides in sound and in initialisms, which is exactly where the cross-lingual signal is weakest.

What this means if you localize user-generated content

The practical question behind this benchmark is real: can a model be trusted to carry Chinese internet slang into English, for moderation, for community management, for subtitling? CIBuzzBench is the first public dataset that lets you ask it, and 3,001 human-confirmed items with category and harmfulness labels is a genuinely useful artifact.

What our probe changes is how you read the scores.

• Do not quote the Equivalent Selection number as cross-lingual capability. 88.4% of it survives deleting the Chinese. If you are choosing a vendor or a model on this task, the discriminating range is roughly 76 to 88, not 25 to 88, and a 4-point difference between two models sits inside that noise.
• Report a blind arm. Any multiple-choice benchmark where one generator writes both the gold and the distractors should publish what a model scores without the source item. It costs one extra run and it is the only way to separate the task from the option set. This is cheap: our whole probe was 800 calls.
• The open-ended task is the one to watch. Task 1 asks for a free-text English explanation and is scored by humans and an LLM judge on a 0 to 5 scale with an average QWK of 0.75. There is no option set to leak. If you need one number from this paper, take that one.
• Buy human signal for the pun subset. The 398 homophonic puns are where models actually fail and where the harmful rate is highest at 24.4%. That is a 400-item annotation job, not a research programme.

None of this is a defect unique to CIBuzzBench. It is the standard failure of LLM-generated distractors, and this benchmark is more transparent than most about how its distractors were made, which is the only reason we could test it.

Check it yourself

The dataset is one CSV. The length baseline takes about two seconds.

git clone https://github.com/SuperYFan/CIBuzzBench && cd CIBuzzBench
python3 - <<'PY'
import csv, random
from sklearn.metrics import precision_recall_fscore_support
R = list(csv.DictReader(open('data/cibuzzbench.csv', encoding='utf-8')))
OF = [("d_literal","distractor_literal_en"), ("d_cultural","distractor_cultural_mismatch_en"),
      ("d_pragmatic","distractor_pragmatic_neighbor_en"), ("correct","equivalent_en")]
L = list("ABCD")
yt, yp = [], []
for row in R:                                  # the benchmark's own shuffle
    opts = [(s, row[f]) for s, f in OF]
    rng = random.Random(f"1111:{row['row_id']}:{row['term_zh']}")
    rng.shuffle(opts)
    yt.append(L[[o[0] for o in opts].index("correct")])
    yp.append(L[max(range(4), key=lambda i: (len(opts[i][1]), -i))])   # longest option
print("longest-by-chars macro F1:",
      100*precision_recall_fscore_support(yt, yp, labels=L, average='macro', zero_division=0)[2])
PY

Expected: 41.39 at seed 1111, against the paper's 79.66 to 88.03 for six frontier models and a uniform-random 25.53.

To reproduce the blind arm, take equivalent_messages() from scripts/evaluate_api.py, delete the term line from the user message, and put this in the system message instead: "Exactly one of these four English phrases is the gold equivalent of a Chinese internet buzzword you are not shown. The other three are a literal gloss, a cultural mismatch, and a pragmatic neighbour." Run it against whatever model you are considering. Our result, on 400 items with Gemini Flash at temperature 0.4, is 76.50%.

The harmfulness floor is one line: 2*(0.858*1)/(0.858+1)/2 gives 0.4617, the macro F1 of always answering non-harmful.

What would prove this wrong

Our claim is that Equivalent Selection is mostly a test of English option-set discrimination. The clean falsification is a distractor set built so that length, register and idiom-likeness carry no signal: resample the three distractors from the pool of gold equivalents belonging to other items, matched on character length to within 10%, so that every option is a real curated English idiom gloss of the same size.

We predict that by 30 June 2027, under a length-matched gold-pool distractor set, a blind model will score under 40 macro F1 and the sighted-minus-blind gap will exceed 30 points, against the 10.0 points we measured. If a blind model still clears 65 on a length-matched set, our explanation is wrong and something other than surface form is leaking.

A second test with a shorter horizon: run the blind arm on the six models in the paper. If GPT-5.5 and Gemini 3.1 Pro come back near 80 blind, the published 87 to 88 figures compress to a 7-point sighted advantage across the whole frontier and the ranking in Table 3 is largely a ranking of distractor-detection.

Sources

1. Yifan Wang, Junyu Lu, Qifan Wang, Shun Zhang, Chaozhuo Li, Jiahao Liu, Zhijun Cao, Lingbin Bu, Fanliang Bu. CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords. arXiv:2609.21722, submitted 18 September 2026. Tables 1, 3, 4 and 6; Appendix A.3.
2. github.com/SuperYFan/CIBuzzBench, the released dataset and evaluation code. data/cibuzzbench.csv (3,001 rows, 2,401 train and 600 test) and scripts/evaluate_api.py (build_options, equivalent_messages, classification_metrics). Last pushed 21 July 2026, checked 22 September 2026.
3. BLOMEGA measurement, 22 September 2026: blind and sighted paired probe on 400 items (Gemini Flash, temperature 0.4, benchmark shuffle seed 1111); length-heuristic baselines over all 3,001 items across five seeds; recount of the category and harmfulness distributions; duplicate, split-leakage and option-length audits.

FAQ

What is CIBuzzBench?

A benchmark of 3,001 Chinese internet buzzwords released on 18 September 2026, each annotated with an English meaning explanation, an English equivalent plus three distractors, a category label and a harmfulness label. It supports three tasks: Meaning Explanation, Equivalent Selection and Harmfulness Detection.

How well do models do on it?

On Equivalent Selection, six frontier models score 79.66 to 88.03 macro F1 under English prompts. On Harmfulness Detection they score 70.41 to 80.45 macro F1, with harmful-class F1 of 48.61 to 66.17. Meaning Explanation is scored by humans and an LLM judge on a 0 to 5 scale.

Does Equivalent Selection measure cross-lingual understanding?

Only partly. We ran the same task with the Chinese buzzword deleted from the prompt. Gemini Flash scored 76.50% blind against 86.50% sighted on the same 400 items, so 88.4% of the sighted score survives having no Chinese input. Picking the longest option by character count alone scores 41.39 macro F1 against a random floor of 25.53.

Why does the option set leak?

The three distractors follow a published recipe: a literal gloss, a cultural mismatch and a pragmatic neighbour. The pragmatic neighbour averages 14.92 characters against the gold's 22.18, and the gold is strictly the longest option in 39.3% of items. A model told the recipe can reject distractors on English register alone.

Which part of the benchmark still needs Chinese?

Homophonic puns. Seeing the Chinese term is worth 17.6 percentage points on that category and only 7.3 to 10.2 points on the other five. Homophonic puns are 398 of the 3,001 items and carry the highest harmful rate at 24.4%.

What is the trivial baseline on the harmfulness task?

The labels are 85.8% non-harmful and 14.2% harmful, so always answering non-harmful gives 92.34 F1 on the majority class, 0 on the minority, and 46.17 macro F1. The six evaluated models are 24 to 34 points above that, which is a genuine margin.