A pre-annotation stage that raises identity resonance also lowers inter-annotator agreement, on purpose

What changed, and when

On 8 October 2026, Sheza Munir, Ramaravind Kommiya Mothilal and Syed Ishtiaque Ahmed (University of Toronto) posted Activating the Annotator: Operationalising Lived Experience in Data Annotation (arXiv:2610.12701v1).

Perspectivist NLP treats annotator positionality, who the annotator is and what they have experienced, as a real driver of labels on subjective tasks, not noise to be averaged away. In practice, positionality is usually captured once, as a demographic form at recruitment, and then used as a static covariate. The paper's objection is that this assumes the relevant knowledge is active in the annotator's mind at the moment they read an item, which a form filled out once does not guarantee.

AXLE (Axis-guided eXperience eLicitation) is their alternative: before labeling begins, a scenario vignette anchors a short conversation (up to 5 turns) that is synthesized into a first-person micro-narrative and shown back to the annotator immediately before they label. The study is a 2x2 between-subjects design: elicitation versus control, crossed with annotators recruited for lived experience (LE) versus not (non-LE), on items about immigration, religion, and gender or sexuality.

The evidence table

Reproduced from Table 1 of arXiv:2610.12701v1. Each participant annotates all 10 items in their assigned scenario: 104 participants × 10 items = 1,040 annotations.

Reproduced from arXiv:2610.12701v1, Sections 4.1, 4.2 and 4.5, and Appendix F (Table 5).

Why agreement falls when engagement rises

The paper's argument hinges on ruling out the obvious alternative explanation: that lower agreement just means annotators stopped paying attention. Inattention would show up as shorter rationales and less time on item. Neither happens. Session time rises, not falls, and the elicitation stage itself adds a median 18.6 minutes of overhead against 7.9 minutes in the control arm, for a task that only labels 10 items. Annotators are doing more work, not less, and getting less agreement out of it.

The effect is also not uniform across annotators. It is larger for non-LE annotators (recruited without screening for lived experience) than for LE annotators: resonance rose 0.51 points for non-LE versus 0.31 for LE at the item level, and harm ratings rose 0.40 versus 0.28. Elicited non-LE annotators ended up reporting resonance (3.19) on par with unelicited LE annotators (3.20), which the authors read as elicitation substituting a constructed, task-relevant perspective for a recruitment-based one.

What it means for anyone running a subjective-label pipeline

Rising agreement is not always the right optimization target. For subjective tasks like hate speech, where the paper's framing is perspectivist (the target is the distribution of valid readings, not one ground truth), a method that converges annotators onto a shared reading can be actively hiding the signal you want. AXLE is a direct test case: it was built to surface plurality, and the low alpha it produces is the method working, not failing.

A near-zero Krippendorff's alpha is not automatically a quality defect. The paper is explicit that its own alpha values (.066 and .111) are too low to interpret as reliability estimates in the usual sense, because a dataset deliberately built to maximize experiential divergence should not reach high agreement. The only valid comparison is between arms on the same fixed item set, which is what the entropy and alpha deltas report.

The cost is concrete and front-loaded. 18.6 minutes of elicitation overhead against 7.9 minutes of control overhead, before any labeling starts, is the number a budget has to absorb. The paper's own framing is that this amortizes if elicitation is reused across many items per annotator, since the per-item marginal cost after elicitation is flat or slightly negative; that claim rests on projections from medians in a 10-item task, not a measured amortization curve.

Domain effects complicate a single policy. On gender and sexuality items, mean harm ratings barely moved (2.94 to 2.88) while entropy rose the most of any domain (1.74 to 2.11): annotators engaged more and moved in opposite directions, which cancels in the mean and would be invisible to anyone checking only the average harm score. A pipeline that monitors mean label drift and not label dispersion would miss exactly this.

Check it yourself

open https://arxiv.org/abs/2610.12701              # paper, Tables 1 and 5, Figures 3 and 4

# what to check without rerunning anything:
# Table 1: design cells, 104 participants x 10 items = 1,040 annotations
# Section 4.1: identity resonance 2.94 -> 3.35, harm rating 2.78 -> 3.12
# Section 4.2 / Figure 4: entropy 1.88 -> 2.05 bits, alpha .111 -> .066
# Section 4.3: LE vs non-LE item-level deltas, resonance +0.31 (LE) vs +0.51 (non-LE)
# Section 4.5: median session time 43.8 -> 51.4 min, overhead 7.9 -> 18.6 min

python3 - <<'PY'
control = {"resonance": 2.94, "harm": 2.78, "entropy_bits": 1.88, "alpha": 0.111}
elicited = {"resonance": 3.35, "harm": 3.12, "entropy_bits": 2.05, "alpha": 0.066}
for k in control:
    delta = elicited[k] - control[k]
    direction = "up" if delta > 0 else "down"
    print(f"{k:14s} control={control[k]:.3f}  elicited={elicited[k]:.3f}  {direction} {abs(delta):.3f}")
PY
# resonance      control=2.940  elicited=3.350  up 0.410
# harm           control=2.780  elicited=3.120  up 0.340
# entropy_bits   control=1.880  elicited=2.050  up 0.170
# alpha          control=0.111  elicited=0.066  down 0.045

What would prove this wrong

The paper's central claim is that elicitation raises plurality rather than collapsing it through demand characteristics or inattention. It rules out inattention with time-on-task and rationale-length data from this one study. The open question is whether the effect replicates outside the three domains tested (immigration, religion, gender or sexuality), all chosen for high baseline ambiguity.

A dated prediction: by 31 December 2027, an independent replication of AXLE on a domain with lower baseline ambiguity (a factual or low-controversy labeling task) will show a substantially smaller or absent entropy increase, because the mechanism depends on there being genuine plurality to surface in the first place. If a replication on a low-ambiguity task shows the same magnitude of entropy increase as this study's hate-speech domains, the "surfaces existing plurality" explanation is wrong, and a generic demand-characteristic effect becomes the better explanation.

Sources

1. Munir, S., Kommiya Mothilal, R., Ahmed, S.I. (University of Toronto). Activating the Annotator: Operationalising Lived Experience in Data Annotation. arXiv:2610.12701v1, 8 October 2026. Tables 1, 5; Figures 3, 4; Sections 3, 4.
2. BLOMEGA. Data annotation research: the latest.
3. BLOMEGA. Consented AI training data providers.