Always answering "stigma" scores F1 0.312 on the new mental health stigma benchmark, and three of the paper's systems score lower

What the Melbourne team released on 2 October 2026

Naomi Baes and Jemima Kang (joint first authors), with Nick Haslam, Chris Groot, Alsa Wu, Luc Raszewski and Yulia Otmakhova, posted Automatic Evaluation of Mental Health Stigma in Online Communication as arXiv:2610.02775v1 (cs.CL) on 2 October 2026. The affiliations are the Melbourne School of Psychological Sciences and the School of Computing and Information Systems at the University of Melbourne, plus the University of Tübingen.

The benchmark holds 470 excerpts that mention one of six conditions: ADHD, alcoholism, autism, borderline personality disorder (BPD), depression and schizophrenia. 249 come from U.S. news in the News on the Web (NOW) corpus, 2010 to 2025, and 221 from Reddit posts, 2012 to 2022. Each text gets a binary stigma label; stigma-present texts also get a mode (expressed/enacted, perceived/experienced, or both), a domain (public, structural, associative, self) and, for public stigma, cognitive, affective and behavioural components. Four annotators labelled them: two psychology professors and two psychology PhD students, the students paid USD 40 an hour for 50 hours.

The paper scores Claude Sonnet 4.6, GPT-5.5, Gemini 3 Flash and DeepSeek V4 Flash under five prompt settings, from a bare task description to definitions, examples and the annotators' operational rules (Table 2). It adds a Qwen2.5-7B LoRA classifier (Table 3) and tests whether off-the-shelf sentiment, hate speech and toxicity models track stigma; they do not (chi-square p from .277 to .813, toxicity point-biserial r = 0.064). The release lives at github.com/jemimakang/mh_stigma, with a mirror at Hugging Face.

The authors say plainly that label proportions are not prevalence estimates because later rounds used targeted sampling. What follows takes that warning one step further, using their own released files.

Does the release match the paper, and what do trivial predictors score?

Every count we could check matches the paper. The baseline rows are ours.

Paper numbers from arXiv:2610.02775v1. BLOMEGA rows computed on 5 October 2026 from data/labels_gold.csv and data/labels_annotators.csv at jemimakang/mh_stigma main, commit 8d3a8a6 (5 October 2026).

The always-yes line is the floor any binary F1 on this benchmark should be read against, and the paper does not print it. DeepSeek V4 Flash with the baseline prompt sits 0.010 under it. The Qwen2.5-7B LoRA cross-validated on the benchmark, which the paper reports as removing over-prediction (bias -0.013), lands 0.002 under it, and the model trained on Meng et al.'s chatbot data sits 0.046 under it. Gemini 3 Flash's baseline run, the paper's top binary F1, clears it by 0.194.

Accuracy points the other way. Because 81.5% of items are stigma-absent, predicting "no" always scores 0.815, and no LLM run reaches it; the closest is Claude Sonnet 4.6 with the operational rules at 0.781. Macro F1 is the one reported metric where both trivial predictors are clearly beaten: always-no gets 0.449, and the weakest LLM run gets 0.538.

We also checked that Table 2 is internally consistent. From each row's recall and bias we rebuilt the confusion matrix (true positives = recall x 87, predicted positives = (0.185 + bias) x 470) and recomputed accuracy. All 20 match the printed value to three decimals, and the rebuilt counts match the text's own example: Claude Sonnet 4.6 drops from 102 false positives under the definition prompt to 47 with rules, while false negatives rise from 40 to 56.

A fairer reference than always-yes is a random guesser that flags stigma at the same rate the model does. Its expected precision is the base rate, so its F1 is 2bq/(b+q) for base rate b and predicted rate q. Against that, every one of the 20 runs is ahead, by 0.094 (DeepSeek, baseline prompt) to 0.263 (Gemini, baseline prompt), mean 0.182. The models carry real signal; the precision lift of 1.45x to 2.14x over chance is how much.

Why "overprediction" partly reflects how the 470 items were sampled

The released batch column maps to the paper's four rounds. Round 1 is a random draw, 30 items with 4 positive (13.3%); the paper says random sampling gave about 85% stigma-absent, and we count 86.7%. Round 2 items were pre-filtered by GPT-5.2 Pro as likely stigma: 120 items, of which 19 (15.8%) ended as gold stigma-present. Round 3 introduced the explicit decision rule: 150 items, 53 positive (35.3%). Round 4 sampled texts where Claude Sonnet 4.6, Gemini 3 Flash and GPT-5.5 disagreed: 170 items, 11 positive (6.5%).

Two consequences follow, both our reading. First, Round 2 is itself an LLM overprediction result the paper does not frame as one: a frontier model flagged these texts as stigma, a manual screen removed clear misses, and still 84.2% were labelled stigma-absent. Some borderline texts were kept deliberately as validity checks, so this is an upper bound on the filter's error, not a clean estimate. The paper says 231 texts were retained for Round 2; the release has 120 Round 2 items, and Table 19 also reports n = 120 for that round, so we could not reconcile the 231.

Second, Round 4 supplies 159 of the 383 stigma-absent items (41.5%). Each was chosen because the three models disagreed on it, which means at least one of the three said "stigma" on every Round 4 negative under whatever prompt was used for selection. Those 159 are, by construction, items that some of the evaluated models flag. The paper reports the procedure honestly, but does not break results down by round. A per-round score would show how much of each model's false-positive count comes from items its own disagreement selected. That needs the per-item predictions, which are not in the repo.

The authors' difficulty variable carries the same coupling. Difficulty is derived from annotator disagreement and confidence, and in the released labels 32 of 48 "hard" items are stigma-present (66.7%), against 22 of 232 "easy" (9.5%). Table 6 reports that models err more on hard items. So does a constant predictor:

Annotator disagreement on a binary task is most likely when at least one annotator says "yes", which is why hard items skew positive. The rising LLM error rate from easy to hard is therefore partly a class-mix effect. Our judgement: the paper's conclusion that models fail where humans deliberated is plausible, but Table 6 alone cannot separate it from "models fail on positives", and a per-class version of the table would.

Per condition, the always-yes floor moves from 0.096 to 0.440

Base rates differ sharply by condition: 4 of 79 depression texts are stigma-present (5.1%) and 22 of 78 schizophrenia texts (28.2%). The always-yes F1 floor moves with them, from 0.096 for depression to 0.440 for schizophrenia. Claude Sonnet 4.6, averaged over its five prompts, scores 0.358 on BPD, below that condition's floor of 0.400, and 0.447 on schizophrenia, 0.007 above its floor of 0.440. Its largest margins are on depression (+0.320, from 4 positives) and autism (+0.154).

The paper reads schizophrenia's high bias (+0.228) as the model overgeneralising from danger and clinical cues. Our check agrees on direction and adds that schizophrenia is also where a constant guess is hardest to beat, because news coverage of schizophrenia carries 17 of the 38 news positives. The authors flag these per-condition estimates as fragile, and with 4 to 22 positives per condition, we agree.

Do the released annotator labels reproduce the agreement numbers?

Yes, for the rounds the paper reports. Table 19 gives pairwise agreement for Rounds 2 and 3. Computing Gwet's AC1 on labels_annotators.csv, we reproduce all six pairs to three decimals once the paper's position labels are mapped to annotator ids: in Round 3, A3 is ann_2, and the low pairs are ann_1 to ann_2 (exact 0.540, AC1 0.081) and ann_2 to ann_4 (0.553, 0.113).

Round 4, the largest round, is missing from Table 19, and its AC1 of 0.820 looks excellent. Cohen's kappa on the same 170 pairs is 0.216. The gap is the prevalence effect: in Round 4 one annotator said stigma 6.5% of the time and the other 14.1%, so agreeing on "no" is cheap. AC1 is designed to resist that, kappa is not, and on rare positives the truth about whether two people find the same stigma sits closer to kappa. Pooled across all pairs, kappa is 0.242.

The per-annotator file also shows that humans overpredict. Against the adjudicated gold on the items each one labelled, ann_2 said stigma on 38.8% of 320 items whose gold rate is 20.0%, a bias of +0.188, which is the same size as the median LLM bias in Table 2. In Round 3, the round that introduced the decision rule, the same annotator said stigma on 66.7% of items against a gold rate of 35.3%. The other three annotators sit between -0.049 and +0.051. 109 of the 383 gold stigma-absent items had at least one annotator say "stigma" before adjudication.

A keyword baseline does worse than always-yes. On the 249 news texts, the only ones with public text, a danger lexicon (dangerous, violent, kill, murder, threat, attack) fires on 44 texts with precision 0.114, below the news base rate of 0.153. A slur lexicon (crazy, psycho, insane and similar) reaches F1 0.190, and all four of our lexicons combined reach 0.222, both under news always-yes at 0.265. This fits the paper's point that negativity is not stigma.

What to do if you build or score a stigma classifier on this data

Print the trivial baselines in every results table. On 470 items at 18.5% positive, binary F1 under 0.312 or accuracy under 0.815 has not beaten a constant. For subsets, recompute: news alone is 0.265 always-yes F1, Reddit 0.363, and per condition from 0.096 to 0.440. Better, report precision lift over base rate or F1 against a rate-matched random guesser; those separate "overpredicts" from "has no signal".

Report results per sampling round. The batch column is in the release. Round 4 was selected by disagreement among Claude Sonnet 4.6, Gemini 3 Flash and GPT-5.5, and Round 2 by GPT-5.2 Pro. Any model from those families is scored partly on items its relatives picked. Round 1's 30 random items are the only part of the benchmark that resembles a natural sample, and 4 positives is too few to score on alone.

Calibrate humans the way you calibrate models. The paper's main lesson, that explicit operational rules cut false positives, applied to its own annotators first. One annotator in this release ran +0.188 above gold, and the Round 3 rule did not close that by itself. In our experience, per-annotator positive rate by batch, watched over the first 50 to 100 items, catches this before adjudication has to absorb it. See our note on how majority vote behaves on rare-event labels.

Use kappa next to AC1 when positives are rare. The 0.820 versus 0.216 split in Round 4 is a decision-relevant gap. If you are deciding whether two annotators agree on what counts as stigma, the stigma-present cells carry the answer, and kappa weighs them more.

Plan for the gated half. 221 Reddit texts, holding 49 of the 87 positives and all evidence spans for them, need an application. Without it, a replication covers 249 news texts and 38 positives. The gating is a reasonable ethics choice for self-disclosure posts; it does mean public leaderboards will likely run on the news half. A licence question also sits open, in our judgement: the Hugging Face card says CC BY 4.0 while the news excerpts come from NOW, a corpus distributed by English-Corpora.org under its own terms, and the GitHub repo has no licence file.

Check it yourself

Two CSVs, about 370 KB together, no gated text needed. This prints the base rate, both constant baselines, the per-round base rates and the Round 4 agreement.

# repo: github.com/jemimakang/mh_stigma (main, 8 commits, last 8d3a8a6 on 2026-10-05)
#   data/labels_gold.csv        fields: item_id, source, keyword, batch, stigma_present, mode, difficulty, text, stigma_span
#   data/labels_annotators.csv  fields: item_id, annotator, batch, stigma_present, stigma_confidence
python3 - <<'PY'
import pandas as pd
B="https://raw.githubusercontent.com/jemimakang/mh_stigma/main/data/"
g=pd.read_csv(B+"labels_gold.csv"); a=pd.read_csv(B+"labels_annotators.csv")
y=g.stigma_present.eq("yes"); b=y.mean()
print(len(g), y.sum(), round(b,4), "always-yes F1", round(2*b/(1+b),4), "always-no acc", round(1-b,4))
print(g.groupby("batch").stigma_present.apply(lambda s:(len(s),int((s=="yes").sum()))).to_dict())
print(g.groupby("source").text.apply(lambda s:s.notna().sum()).to_dict())
w=a[a.batch=="batch4"].pivot(index="item_id",columns="annotator",values="stigma_present")[["ann_2","ann_4"]].dropna().eq("yes")
x,z=w.ann_2,w.ann_4; po=(x==z).mean(); pi=(x.mean()+z.mean())/2
pk=x.mean()*z.mean()+(1-x.mean())*(1-z.mean())
print(len(w), round(po,3), "AC1", round((po-2*pi*(1-pi))/(1-2*pi*(1-pi)),3), "kappa", round((po-pk)/(1-pk),3))
PY
# 470 87 0.1851 always-yes F1 0.3124 always-no acc 0.8149
# {'batch1': (30, 4), 'batch2': (120, 19), 'batch3': (150, 53), 'batch4': (170, 11)}
# {'now': 249, 'reddit': 0}
# 170 0.853 AC1 0.82 kappa 0.216

curl -s https://api.github.com/repos/jemimakang/mh_stigma | grep '"license"'        # null
curl -s https://huggingface.co/api/datasets/jemimakang/mh_stigma | grep -o 'license:[a-z0-9.-]*'   # license:cc-by-4.0

The Table 2 rebuild uses only printed numbers: true positives = round(R x 87), predicted positives = round((87/470 + bias) x 470), accuracy = (TP + 383 - FP) / 470. Per-annotator bias compares each annotator's "yes" share with the gold "yes" share on the same items. The keyword lexicons we used are listed in the agreement section; they were fixed before scoring and not tuned. The Reddit application form at forms.cloud.microsoft/r/KJbaMPAW03 returned HTTP 200 on 5 October 2026; we did not apply.

What would prove this wrong

Our claim is that a large part of the reported LLM overprediction on this benchmark is tied to how the stigma-absent items were sampled, especially Round 4, rather than being a uniform property of the models.

It is falsified if per-round scores, once published by the authors or anyone with the per-item predictions, show the evaluated LLMs' false-positive rate on Round 4 negatives within 5 points of their rate on Rounds 1 to 3 negatives. If Round 4 negatives draw a false-positive rate more than 10 points higher for Claude Sonnet 4.6, Gemini 3 Flash and GPT-5.5 but not for DeepSeek V4 Flash, which took no part in the selection, the sampling account holds.

A dated prediction: by 30 June 2027, no published system on the full 470 items will report binary F1 above 0.60 together with accuracy above 0.815, the always-no line. A system that does both would show that stigma presence on this data can be detected well above both constant predictors at once.

Sources

1. Baes, N., Kang, J., Haslam, N., Groot, C., Wu, A., Raszewski, L., Otmakhova, Y. Automatic Evaluation of Mental Health Stigma in Online Communication. arXiv:2610.02775v1, 2 October 2026. Sections 3.4, 4.2, 5.1, 5.2, 5.3; Tables 1, 2, 3, 5, 6, 16, 19; Appendices D, I.
2. Kang, J. jemimakang/mh_stigma on GitHub, created 28 September 2026, last commit 5 October 2026 (8d3a8a6). Accessed 5 October 2026.
3. Kang, J. jemimakang/mh_stigma on Hugging Face, CC BY 4.0, last modified 5 October 2026. Accessed 5 October 2026.
4. Davies, M. News on the Web (NOW) corpus, English-Corpora.org. Accessed 5 October 2026.
5. Gwet, K. L. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology 61(1), 2008.
6. BLOMEGA. Once individual accuracy drops below 50%, majority vote makes rare-event labels worse.
7. BLOMEGA. LLM annotators and near-zero kappa.

FAQ

What does a trivial baseline score on the mental health stigma benchmark (arXiv:2610.02775)?

87 of the 470 texts are stigma-present, a base rate of 18.5%. Predicting stigma for every text gives precision 0.185, recall 1.0 and binary F1 0.312 (macro F1 0.156). Predicting no stigma for every text gives accuracy 0.815 and macro F1 0.449. In the paper, DeepSeek V4 Flash with the baseline prompt scores binary F1 0.302, the Qwen2.5-7B LoRA cross-validated on the benchmark 0.310 and the Qwen model trained on Meng et al. 0.266. No LLM run reaches 0.815 accuracy; the best is 0.781.

How much of the mental health stigma benchmark is publicly released?

All 470 items have labels on GitHub and Hugging Face, with 1,201 per-annotator rows. Text is released for the 249 news excerpts only. The 221 Reddit excerpts, which hold 49 of the 87 stigma-present items, are released on application. The GitHub repo has no licence file; the Hugging Face card declares CC BY 4.0. Checked 5 October 2026.

Do human annotators also overpredict stigma on this benchmark?

One did. Annotator ann_2 said stigma on 38.8% of 320 items whose adjudicated positive rate is 20.0%, a bias of +0.188, close to the LLM range in the paper; in Round 3 the figure was 66.7% against 35.3%. The other three annotators sit between -0.049 and +0.051, and 109 of the 383 gold stigma-absent items had at least one annotator say stigma before adjudication.