11 of the 100 tweets in "Nobody Truly Agrees on Sentiment" are 3-3 rater splits, and the tie-break moves RoBERTa's kappa by 0.12

What arXiv:2610.10318 claims, and what its authors released

Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao, Michael L. Nelson and Michele C. Weigle (Old Dominion University, Norfolk, Virginia) submitted Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts on 7 October 2026 (v1, cs.CL, 11 pages, accepted at TPDL 2026). The work was funded in part by the Minerva Research Initiative with AFOSR, grant FA9550-22-1-0297.

The data are 100 tweets containing "Site C, Khayelitsha", posted January 2022 to December 2024 and collected for a study of residents' perceptions of safety in Khayelitsha Township, South Africa. Six Computer Science graduate students (R1 to R6) labeled each tweet positive, neutral or negative. The paper compares them with TextBlob, VADER, cardiffnlp's Twitter-RoBERTa-base (RB), and three LLMs prompted one-shot: Qwen3-32B, GPT-OSS-120B and Llama-4-Maverick-17B.

Four claims carry the paper. Human raters reach only fair agreement (Fleiss 0.39 three-class). Binary framings (negative vs non-negative, positive vs non-positive) agree better than three-class "across both humans and automated tools". RB aligns best with the human majority (0.73, 0.88, 0.61 Cohen kappa on the three framings). LLMs agree with each other at 0.74 to 0.86 and with humans at moderate levels. Section 3 adds: "No ties occurred during the majority voting process; therefore, no tie-breaking procedure was required." Section 5 says agreement values are point estimates without intervals.

Reference 17 points to himarshaj/hard-to-reach-environments, MIT licence. On 8 October 2026 it held five files: the 100 tweets with translations, the six human labels plus TextBlob, VADER and RB labels, the binary recodings, a majority-vote file, and a README with the LLM prompts and zero-shot result tables. The data files were committed on 10 March 2025; the README gained the LLM tables and prompts on 19 July 2026. The LLM labels themselves are not in the repository. That makes every human and bespoke-tool number checkable and every LLM number uncheckable.

Do the released labels reproduce the paper's kappas?

Yes, every one we could reach. The table lists a sample of the 35 values; the rest are in the check script output.

Paper values from arXiv:2610.10318v1. BLOMEGA values computed on 8 October 2026 from sentiment_analysis_tools/sampled_tweets100_translated-majorityvote.tsv at himarshaj/hard-to-reach-environments main (last commit 19 July 2026), with scikit-learn's cohen_kappa_score and statsmodels' fleiss_kappa.

Two release details matter for anyone reusing the set. The tools were run on the tweet_translated column, and 8 of the 100 tweets were machine-translated from isiXhosa or mixed text, several into word salad ("Site Scaleries in the Site C..."). For one, the translated text is the string "Error in translation"; the original is a list of Africa's ten biggest slums, which all six humans called neutral. The tools never saw that tweet's content. Our VADER rerun differs from the released labels on 5 tweets, three of them translated ones, which is consistent with VADER having been run on a different text version or library release.

Where do the 11 ties go, and how much does it matter?

A 6-rater, 3-class vote can tie, and here it does. Seven tweets split three negative against three neutral; four split three neutral against three positive. The released majority labels all seven of the first group negative and three of the four neutral-positive ties positive. One neutral-positive tie (row 29, "18 Profitable Small Business Ideas to Start in South Africa in 2023") went to neutral, so the rule is not applied consistently either. The binary majority columns were computed separately: 9 of 10 negative vs non-negative ties went to negative, and 3 of 5 positive vs non-positive ties went to positive. On rows 62 and 79 the binary majority contradicts the three-class majority.

The net effect is a majority that leans away from neutral: 44 negative, 36 neutral and 20 positive as released, against 37, 46 and 17 if ties go to neutral. A tool's kappa against that majority depends on how much its own neutral rate matches. RB labels 42 tweets neutral, closer to the tie-to-neutral count of 46 than to 36, which is why its scores move the way they do below.

RB's negative vs non-negative score is stable: 0.876 as released, 0.788 with ties to neutral, 0.883 with the 11 tweets dropped. Its positive vs non-positive score is not: 0.610, 0.729 and 0.751. The paper's one exception to "RB is best", Llama-4-Maverick at 0.71 on positive vs non-positive, sits between those values. We cannot say whether Llama would also move, because its labels are not released. What we can say is that the ranking on that column is decided by how 11 tweets were broken, not by the models.

TextBlob and VADER move too. With ties sent to neutral, TextBlob's three-class kappa rises from 0.368 to 0.399 and VADER's falls from 0.396 to 0.361, so even their order flips. TextBlob's positive vs non-positive score goes from 0.292 to 0.378.

Is binary labeling more reliable, or is neutral just the weak category?

Fleiss kappa has a property that makes the paper's comparison less informative than it looks. Collapsing to "category j vs the rest" gives exactly the category-specific kappa of j, and the multi-class kappa is the average of the category kappas weighted by p(1-p). For the six humans that gives negative 0.454, neutral 0.257 and positive 0.484; the weighted mean is 0.385, the paper's 0.39. Both binary framings beat three-class because they each drop the neutral term, which is both the lowest and, at 44.3% of all labels, the heaviest.

So the result is real but narrower than stated: on these tweets, raters disagree most about where neutral ends. That is the actionable finding, and it is a statement about the neutral boundary, not about binary schemes in general. The gains are also modest. A 2,000-draw tweet bootstrap puts the human gain at 0.069 for negative vs non-negative (95% interval 0.014 to 0.128) and 0.099 for positive vs non-positive (0.010 to 0.181). Human three-class kappa itself runs 0.312 to 0.455.

For tools against the majority, the paper's own Table 2 contradicts the abstract's "across both humans and automated tools". Positive vs non-positive is below three-class for all three bespoke tools: TextBlob 0.29 against 0.37, VADER 0.37 against 0.40, RB 0.61 against 0.73. In Table 1, the three tools together score 0.43 on both. The bootstrap interval for RB's positive minus three-class difference is -0.291 to 0.010. The README's zero-shot tables add one more wrinkle the paper does not show: Qwen3-32B zero-shot beats its one-shot score on all three framings (0.53, 0.61, 0.68 against 0.48, 0.60, 0.61), while the paper reports one-shot as "slightly better or comparable".

What should a labeling team change after this paper?

Write the tie rule down before labeling, whatever the rater count. Six raters on three classes left 11% of items tied here, and an undocumented rule that favoured polar labels changed one of four model rankings. With five raters a 3-class vote can still tie (2-2-1), so the rule has to exist either way; publish it with the labels and report results under at least one alternative.

Report per-category kappa alongside any binary vs multi-class contrast. The neutral kappa of 0.257 is the number a guideline author needs: it says where to spend the next calibration round. Most of the 11 tied tweets are news or political commentary (a retail slump, the Khayelitsha police inquiry, ANC and EFF politics), where raters must decide whether reporting bad news is itself negative. Our judgement: that is a missing guideline rule, not rater noise.

Score tools against individual raters and the soft distribution, as well as the majority. When the majority itself moves 0.12 kappa with a tie rule, a model ranking based on it is fragile. Cohen kappa per rater, averaged, or cross-entropy against the six-rater distribution, uses the 11 split tweets instead of hiding them.

Check what the model actually read. Eight tweets were machine-translated before the tools saw them, and one input was a translation error string. In a 100-item evaluation, 8 items is 8 points of accuracy. Store the text each system scored alongside its label.

Release every system's labels. The authors released the hard part, six human labels, under MIT, which is why this check was possible in an afternoon. The LLM labels are the only piece missing, and they are the piece behind the paper's comparison with LLMs. Our judgement: the conclusion that a 2020 fine-tuned RoBERTa beats current LLMs is plausible on negative sentiment (0.88 against 0.71 at best, stable across tie rules) and unsettled on positive sentiment.

How to reproduce every number here

One 80 KB TSV holds all human and tool labels. Fields: sentiment_Rater1 to sentiment_Rater6, sentiment_textblob, sentiment_VADER, sentiment_roBERTa, the _NvsNonN / _PvsNonP recodings, and majority_sentiment_human_raters with its two binary versions.

# data: github.com/himarshaj/hard-to-reach-environments (MIT), main, last commit 2026-07-19
curl -sLO https://raw.githubusercontent.com/himarshaj/hard-to-reach-environments/main/sentiment_analysis_tools/sampled_tweets100_translated-majorityvote.tsv
python3 -m venv v && v/bin/pip -q install pandas scikit-learn statsmodels
v/bin/python - <<'PY'
import pandas as pd, numpy as np, collections
from sklearn.metrics import cohen_kappa_score as ck
from statsmodels.stats.inter_rater import fleiss_kappa, aggregate_raters
d=pd.read_csv("sampled_tweets100_translated-majorityvote.tsv",sep="\t")
L=lambda c:d[c].str.strip().str.lower().values
H=np.array([L(f"sentiment_Rater{i}") for i in range(1,7)]).T
t,cats=aggregate_raters(H); print("Fleiss 3-class",round(fleiss_kappa(t),3))   # 0.385
p=t.sum(0)/600
kj=[1-(t[:,j]*(6-t[:,j])).sum()/(600*5*p[j]*(1-p[j])) for j in range(3)]
print(dict(zip(cats,np.round(kj,3))))       # negative 0.454, neutral 0.257, positive 0.484
tie=[(lambda c:len(c)>1 and c[0][1]==c[1][1])(collections.Counter(r).most_common()) for r in H]
print("3-3 ties",sum(tie))                  # 11
M=L("majority_sentiment_human_raters"); RB=L("sentiment_roBERTa")
Mn=np.array([("neutral" if tie[i] and "neutral" in set(H[i]) else M[i]) for i in range(100)])
pb=lambda a:a=="positive"
print("RB pos-vs-rest, released",round(ck(L("majority_sentiment_human_raters_PvsNonP")=="positive",pb(RB)),3),
      "ties->neutral",round(ck(pb(Mn),pb(RB)),3))   # 0.610, 0.729
PY
# release check (8 Oct 2026): no LLM label file
curl -s https://api.github.com/repos/himarshaj/hard-to-reach-environments/contents/sentiment_analysis_tools | grep '"name"'

The tool rerun used TextBlob 0.20.1 (polarity sign), vaderSentiment 3.3.2 (compound at plus or minus 0.05) and cardiffnlp/twitter-roberta-base-sentiment (argmax, with and without @user/http masking, same result) on tweet_translated. Matches to the released labels: 100, 95 and 97 of 100. Against the released majority our rerun gives TextBlob 0.368 / 0.405 / 0.292 (identical), VADER 0.396 / 0.518 / 0.370 and RB 0.717 / 0.836 / 0.635. The bootstrap resamples tweets with replacement, 2,000 draws, numpy seed 0. The "ties dropped" condition scores on the 89 untied tweets.

What result would show this reading is wrong?

Our reading has two parts: the positive vs non-positive model ranking is set by the tie rule, and the binary gain comes from a weak neutral category, not from binary framing as such.

First, if the authors release the three LLM label files, the prediction is that under the ties-to-neutral rule RB's positive vs non-positive kappa (0.729) will be within 0.05 of Llama-4-Maverick's or above it. If Llama stays more than 0.05 ahead of RB under both alternative rules, the ranking is real and our point about it is wrong.

Second, a dated prediction: by 30 June 2027, any published three-class tweet sentiment study with at least 3 raters per item and per-category Fleiss kappa reported will show neutral as the lowest of the three. A study with neutral at or above both polar categories, and still higher agreement under both binary framings, would contradict the arithmetic explanation offered here.

Sources

1. Jayanetti, H. R., Dhanushkanda, S., Hao, S., Nelson, M. L., Weigle, M. C. (Old Dominion University). Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts. arXiv:2610.10318v1, submitted 7 October 2026, accepted at TPDL 2026. Sections 3, 4, 5, Tables 1 and 2.
2. Jayanetti, H. R. hard-to-reach-environments/sentiment_analysis_tools (MIT). Data committed 10 March 2025, README with LLM tables and prompts 19 July 2026. Accessed 8 October 2026.
3. Barbieri, F., Camacho-Collados, J., Espinosa Anke, L., Neves, L. TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification. Findings of EMNLP 2020. Model card: cardiffnlp/twitter-roberta-base-sentiment.
4. Hutto, C., Gilbert, E. VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. ICWSM 2014. Code: cjhutto/vaderSentiment.
5. Fleiss, J. L. Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5), 378 to 382, 1971 (category-specific kappa and its weighted-average relation to the overall coefficient).
6. BLOMEGA. Treat the three annotators as a sample and eight LLMs tie.
7. BLOMEGA. LLM judges, soft labels and human disagreement.

FAQ

Did the six human raters in arXiv:2610.10318 really have no majority-vote ties?

No. The paper says no ties occurred, but in the released majorityvote.tsv 11 of the 100 tweets split 3-3 between two classes on the three-class task (7 negative vs neutral, 4 neutral vs positive), 10 split 3-3 on negative vs non-negative and 5 on positive vs non-positive. The released majority resolved 10 of the 11 three-class ties toward the polar class and 1 toward neutral. Counted by BLOMEGA on 8 October 2026.

Is binary sentiment labeling really more reliable than three-class labeling?

For the six humans in this study, the gain is real but small and mostly mechanical. Fleiss kappa for negative vs rest equals the category-specific kappa of negative (0.454) and positive vs rest equals that of positive (0.484), while the three-class kappa of 0.385 is a weighted average of those two and neutral's 0.257. Bootstrap 95% intervals for the gains over three-class are 0.014 to 0.128 and 0.010 to 0.181. For the three tools scored against the human majority, positive vs non-positive is lower than three-class in all three cases (0.29, 0.37, 0.61 against 0.37, 0.40, 0.73).

Does Twitter-RoBERTa really agree with humans better than the LLMs in this paper?

On negative vs non-negative it scores 0.88 against the human majority, above the best LLM at 0.71, and that holds under every tie rule BLOMEGA tried (0.79 to 0.88). On positive vs non-positive the paper ranks Llama-4-Maverick first at 0.71 over RoBERTa at 0.61, but resolving the 11 ties toward neutral raises RoBERTa to 0.73, and its bootstrap interval runs 0.39 to 0.79. The LLM labels are not in the public repository, so the LLM side of that comparison cannot be recomputed.