Annotator agreement dropped 32 points across one labeling campaign, and the cause was timing, not task difficulty

What changed, and when

On 26 May 2026, Idris Abdulmumin, Mokgadi Penelope Matloga, Tadesse Destaw Belay, Botshelo Kondowe, Letlhogonolo Mohleleng, Hareaipha Nkopo Letsoalo, Shamsuddeen Hassan Muhammad and Vukosi Marivate posted Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora (arXiv:2605.27239). Three native Setswana-speaking undergraduate annotators (Ann. A, Ann. B, Ann. C) labeled 3,565 tweets for sentiment: a 65-tweet training batch for calibration, then seven production batches of 500 tweets each, labeled independently and asynchronously, with each annotator's submission timestamped in UTC.

Most inter-annotator agreement studies report one number for the whole campaign and move on. This paper instead asked what predicts agreement at the level of an individual tweet and an individual batch, running separate analyses for annotation speed, token count, language-identification rank, label confusion patterns and run-length drift, alongside the one that turned out to predict agreement best: the elapsed time between the first and last annotator's timestamp on the same tweet.

The evidence table

For every tweet, the authors computed a time span: the gap between the earliest and latest of the three annotators' UTC timestamps on that tweet. Binning tweets by that span and computing Kappa within each bin produced a near-monotonic decline.

Source: arXiv:2605.27239, Section on temporal-gap analysis (RQ5). The more-than-1-day bin is 60.3% of the dataset. Annotation speed and tweet-level linguistic features (language-identification rank, token count) showed no comparable association: Kappa was 0.757 for tweets ranked as top-1 Setswana by language identification (n=2,497) and 0.757 for tweets that were not (n=1,058); across token-count bins of 4 to 6, 7 to 10, 11 to 15, and 16-plus tokens, Kappa moved only between 0.73 and 0.77 with no consistent trend.

The same decline shows up at the batch level. The paper's text gives exact figures for four of the seven production batches: Batch 1 reached Kappa 92.17, Batch 4 reached 82.13, Batch 5 reached 64.29 (a 17.8-point drop from Batch 4, the sharpest single-batch decline in the campaign), and Batch 7 reached 60.09. Batches 2, 3 and 6 appear only on the paper's chart and are not given exact figures in the extractable text, so they are not reported here. The cumulative Kappa across the full campaign was 75.66, rounding to the aggregate 0.76 the abstract reports, a number that on its own gives no hint of a 32-point swing underneath it.

Label confusion concentrated on one boundary: annotators disagreed on Negative versus Neutral in 1,448 pairwise cases, against 420 pairwise disagreements on Positive versus Neutral, and the paper reports the Negative/Neutral confusion darkening specifically in Batches 4 through 7, the same batches where Kappa was falling. The authors attribute this to indirect or ironic phrasing common in political and social commentary on the platform, a plausible reading they do not claim to have tested directly.

Why the clock, not the calendar of difficulty, predicts the drop

The mechanism the authors propose is shared context, not shared skill. Three annotators who label the same tweet within a minute of each other are, in effect, applying the same momentary interpretation of the guidelines, the same recent exposure to similar tweets, and the same unarticulated sense of what "neutral" means that morning. Annotators separated by a day or more have had time to drift: to interpret an ambiguous case differently after seeing different subsequent examples, to forget an edge case discussed in training, or simply to re-derive their own private rule for a boundary the guidelines under-specify.

A second, compounding mechanism runs alongside the clock: fatigue inside a labeling session. The authors tracked each annotator's mean run length, the average number of consecutive tweets in a row given the identical label, as a proxy for non-deliberate labeling. In the training batch, mean run length sat around 1.4 to 1.5. By Batches 6 and 7, it had risen to roughly 1.7 to 1.9, with Ann. B and Ann. C showing a consistent upward trend from Batch 5 onward and counts of runs five labels long or longer rising in the same window. Annotation speed itself was not the driver: batch median labeling time fell from about 20 seconds in training to under 9 seconds by Batch 6, but the sharpest speed-up happened between Batches 1 and 2, while the sharpest Kappa drop happened between Batches 4 and 5, after speed had already plateaued. Speed and agreement moved on different clocks.

What this changes about running an annotation campaign

A single aggregate Kappa is close to useless for catching this. The campaign's overall Kappa of 0.76 reads as "excellent" by the usual thresholds. The per-batch breakdown shows a 32-point fall that the aggregate number is mathematically built to smooth over, since early high-agreement batches and late low-agreement batches average out. A quality dashboard that reports one Kappa per dataset, not per batch, would have approved this dataset without ever seeing the problem.

The fix the paper proposes is a scheduling constraint, not a training intervention. Capping the time between the first and last annotator on a batch, the authors suggest 48 hours or shared real-time labeling sessions, is cheap relative to re-training annotators or rewriting guidelines, and it targets the variable their own data shows actually predicts agreement. Tools built around annotation work already log per-item timestamps for other reasons (audit trails, pay-per-item billing); this finding is an argument for surfacing the gap between annotators' timestamps as a live quality signal inside the tool, not just a post-hoc audit field. A platform like Label Studio, which already timestamps individual annotations, is a natural place to add a per-item "annotator time span" view rather than building a separate tracking spreadsheet, though we have not tested this ourselves and that is our inference, not the paper's claim.

This is upstream of confident-learning label-noise detection, not a substitute for it. Tools in the Cleanlab family find label errors by comparing a trained model's predicted confidence against the given label, after labeling is finished. The temporal-gap effect in this paper happens during labeling, before any model exists to compare against, so bounding annotator time spans is a prevention step that would reduce how much noise a confident-learning pass later has to find, not a replacement for running one.

Managed-workforce annotation at scale has to coordinate, not just hire, simultaneity. A vendor running annotation with many contractors across time zones, the kind of setup a vendor like iMerit operates, faces this problem structurally: get enough annotators working the same batch within the same short window, and simultaneity becomes a scheduling and incentive design question, not just a data-quality metric measured afterward.

Our judgement: three annotators, one language, one platform (Twitter/X), and one four-month university cohort is a narrow base for a general claim about simultaneity. The authors themselves flag the single dialect, the single institution, and the academic-calendar explanation for the time-span spread as a plausible but untested story. The run-length and time-span effects are also correlated in this dataset (both worsen in later batches), and the paper does not report an analysis that cleanly separates how much of the Kappa decline each one independently explains.

Check it yourself

The paper's own text, not just its figures, gives the four batch-level Kappa values and the four time-span bins used above.

open https://arxiv.org/abs/2605.27239     # abstract, authors, submission date (26 May 2026)
open https://arxiv.org/html/2605.27239v1  # full text: temporal-gap analysis (RQ5), run-length analysis (RQ3), Appendix F recommendations

python3 - <<'PY'
# recompute the headline comparisons directly from the paper's reported numbers
batch1, batch7 = 92.17, 60.09
print("campaign-wide Kappa-point drop, Batch 1 to Batch 7: %.2f" % (batch1 - batch7))

batch4, batch5 = 82.13, 64.29
print("single sharpest batch-to-batch drop, Batch 4 to 5: %.2f" % (batch4 - batch5))

under_1min, over_1day = 0.98, 0.65
print("Kappa gap, under 1 minute vs over 1 day: %.2f" % (under_1min - over_1day))

neg_neutral, pos_neutral = 1448, 420
print("Negative/Neutral disagreements per Positive/Neutral disagreement: %.2fx" % (neg_neutral / pos_neutral))
PY
# campaign-wide Kappa-point drop, Batch 1 to Batch 7: 32.08
# single sharpest batch-to-batch drop, Batch 4 to 5: 17.84
# Kappa gap, under 1 minute vs over 1 day: 0.33
# Negative/Neutral disagreements per Positive/Neutral disagreement: 3.45x

The 32.08-point figure matches the paper's own framing of "more than 32 points." The 17.84-point figure matches its description of the Batch 4-to-5 drop as the sharpest single decline. The dataset, per-annotation timestamps, and analysis code are stated as released alongside the paper; we did not download or re-run them, only recomputed ratios from the numbers reported in the text.

What would prove this wrong

The claim rests on one campaign: one language, one platform, three annotators from one institution, one four-month window shaped by an academic calendar the authors themselves say may explain the growing time spans. A different campaign with a different cause for its scheduling drift, or one run as shared real-time sessions from the start, could show no temporal-gap effect at all, which the paper's own framing would predict as the correct outcome of its own recommendation, not a disconfirmation.

A dated prediction: by 31 December 2027, an independent team running a comparable sentiment or stance-labeling campaign in a different language, with per-item annotator timestamps published, will report whether Kappa binned by annotator time-span shows the same pattern, high agreement under roughly an hour, a material drop past a day. If independent campaigns instead show time span uncorrelated with Kappa once run length and annotator pairing are controlled for, the generalizable version of this claim, that simultaneity itself (not something it happens to correlate with in this campaign) predicts annotation quality, is wrong.

Sources

1. Abdulmumin, I., Matloga, M. P., Belay, T. D., Kondowe, B., Mohleleng, L., Letsoalo, H. N., Muhammad, S. H., Marivate, V. Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora. arXiv:2605.27239, 26 May 2026. HTML version.
2. BLOMEGA. Tweet sentiment Kappa and majority-vote ties labeled neutral.
3. BLOMEGA. Intra-annotator boundary label gap, checked against a threshold.
4. BLOMEGA. Data annotation research: the latest.