One annotator's boundaries hold and labels drift: all 88 checks reproduce, but on one text 56 of 120 Gemini run-triples pass the "human" test

What arXiv:2610.04370 claims, and what it released on 3 October 2026

Marharyta Shvets (Eötvös Loránd University, Budapest) posted Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data as v1 in cs.CL on 3 October 2026: 8 pages, 1 figure, 5 tables. It is a single-author paper.

The design: three Sumerian myths in English translation (the Gudea Cylinder A composite, Inanna's Descent, Inanna and Enki). Each reader cuts the text into segments and gives each segment one of seven causal-function states: preparation, contact, exchange, disruption, negotiation, stabilization, return. One expert human (also the author of the scheme) annotated each myth three times, between 10 October 2025 and 20 September 2026. Three model families, Claude Opus 4.8 (high reasoning), Gemini Pro and Qwen 3.8 Max (thinking), each ran ten times per myth in fresh anonymous web sessions between 29 June and 13 August 2026.

Readings are projected onto shared units (a text line for Gudea and Descent, one of 71 translation blocks for Inanna and Enki) and compared run against run with Cohen's kappa: 7-way on labels, 2-way (starts a segment or not) on boundaries. The headline (Table 1, Section 4.1): the human's boundary kappa minus label kappa is +0.440, +0.452 and +0.298, while the models' gap flips sign across myths (Table 2). A second, exploratory result (Table 4) says the human's relabelling is a phase shift: other runs start the same function one piece earlier or later 2.5 and 1.8 times more often than chance on Descent and Gudea.

Section 5 turns this into a proposal: "a 'human' annotation whose labels are as stable as its boundaries looks like a model", with the phase shift as a second, finer check. Code and data are on GitHub at Malificenta883/intra-annotator-dynamics, MIT licence.

Do the released runs reproduce the paper's kappas?

Yes, exactly. The repo's reproduce.py ran in 28 seconds on a laptop and printed 88 ok lines and no mismatch; only the figure step failed, because matplotlib was not installed. We then wrote our own 60-line projection and kappa script without importing the authors' modules, and every table value below came out identical to three decimals.

Paper numbers from arXiv:2610.04370v1. BLOMEGA rows computed on 6 October 2026 from the repo at commit 6baa8a8 (7 commits, created 29 September 2026, last push 6 October 2026).

The chart is Table 2 redrawn from raw files. The human bar is the largest in two myths and second by a hair in the third: Gemini's +0.291 on Gudea sits 0.007 below the human's +0.298. The paper says this openly ("Gemini's gap on Gudea is as large as the human's"), and rests the claim on the human being large and positive in all three. That is the right reading of Table 2. It also means the signature is a three-text pattern, which matters for anyone hoping to apply it to one annotation job.

Two things the paper does not print are worth having. First, the human's labels are not near noise: raw agreement of 31.6% to 37.6% is roughly twice the 13.6% to 17.8% expected from her own label frequencies. Second, the boundary layer's raw agreement of 96.7% on Gudea is almost all agreement on lines that are not boundaries; chance is already 93.8% there because only 3.2% of Gudea lines start a segment. Kappa is doing the honest work here, and the paper is right to use it for both layers.

Table 4's one-step ratio is 1.04 on Inanna and Enki, which is no shift at all. The phase-shift result therefore holds on two of three myths, as Section 4.3 says in the body; the abstract's phrasing reads as general.

Why a 7-way label kappa and a 2-way boundary kappa can sit 0.45 apart

The gap is built from two quantities with very different chance floors. On labels, seven states with a dominant state at 26% to 28% of units give chance agreement of 13.6% to 17.8%. On boundaries, a unit starts a segment 3.2% (Gudea) to 22.5% (Inanna and Enki) of the time, so chance agreement is 65.1% to 93.8%. Kappa rescales both to the same 0-to-1 line, and the paper argues (Section 3.5) that kappa gives no credit to a boundary one unit off, so the boundary scores are, if anything, too low.

That argument holds. A second effect pushes the other way and is not discussed: the label layer inherits every boundary disagreement. If two runs cut a passage one line apart, that line carries two different labels whenever the neighbouring segments differ in function. The authors partly test this (their results/table1.txt reruns B2 on units more than two units from any boundary, keeping 19 of 20 events on Gudea), but the label kappa itself is not reported on boundary-far units. Our judgement: the gap is not an artefact, since label raw agreement of about one third cannot come from edge effects alone when human segments average 21 to 32 lines on Descent and Gudea, but part of the label instability is the phase shift itself, measured twice.

The phase-shift mechanism is the more interesting claim for annotation teams. Across her runs the human goes through much the same sequence of functions but starts each one a piece earlier or later; Opus runs start them at the same places. Model runs in this study were collected within one to three days per myth and family, human runs 34 to 315 days apart, so the comparison of levels is confounded by time, as the Limitations section says.

Could the gap flag a model-written "human" annotation on one text?

Section 5 offers the gap as a check but sets no number. We built the closest operational version the released data allows. For each model and myth we took all 120 subsets of three runs out of ten (matching the human's three runs, as the paper does for Table 4) and computed the mean boundary kappa, label kappa and gap for each. The question: how many model triples reach the human's own gap on that text?

On Inanna and Enki and on Inanna's Descent, none of 360 model triples reaches the human gap (+0.440, +0.452). On Gudea, 56 of 120 Gemini triples do (range +0.086 to +0.522), and 1 of 120 Opus triples. A single-text check with the human's gap as the bar would have called Gemini human about half the time on the longest of the three texts.

Lower the bar and it gets worse in a different way. At a threshold of 0.25, Gemini on Gudea passes in 69.2% of triples, Opus on Descent in 25.8%. At a threshold of zero (labels merely less stable than boundaries), Qwen passes on all three myths in an estimated 56% of independent triple draws. Only pooling all three myths at a high bar separates every model from this one human.

The human's own pairs scatter widely. On Inanna and Enki, runs 1 and 3, 315 days apart, agree on labels at kappa 0.586 and on boundaries at 0.677: a gap of +0.091, inside every model's range on that myth. The other two pairs of the same myth sit at +0.585 and +0.643. A two-run check would have marked this human, on this pair, as model-like.

Dates from annotated_date in each human1_run*.json; Gudea run 2 is stored as the malformed string 2026-08_07, read as 7 August 2026. Pair means reproduce paper Table 1.

The interval does not explain the pairs. In Inanna and Enki the furthest-apart pair agrees most on labels; in Descent and Gudea the closest pair does. That fits the paper's reading that label change is reinterpretation, not forgetting, but with nine pairs it fits many readings.

One simpler rule does separate every triple in this data: label kappa below 0.30. The human's myth means are 0.168 to 0.291; the lowest model triple is 0.310 (Gemini, Gudea), so the margin is 0.019. The paper explicitly rejects low self-consistency as the signature, because human and model runs are spaced differently, and that caution is fair: a model rerun months later on a new version might drift too. But on the data released, the level separates better than the gap does.

Is everything needed to rerun it public?

Almost. The repo holds 151 files (3.4 MB without git history): 99 run files, the three numbered texts, prompts, code and results. Three gaps we found on 6 October 2026:

The seg_v3 prompt is lost. Opus's ten Descent runs (29 to 30 June 2026) used an earlier prompt whose text "was not kept: the file was edited after these runs" (prompts/seg_v3_note.md). Line ranges for those runs were parsed out of scene names by script. The paper discloses the version change in Section 3.2.

The prompt example is a real Gudea segment. seg_v4.txt gives one worked example, quoting Gudea lines 1 to 3 and declaring line_start 1, line_end 9. 31 of 33 Gudea runs, all 10 Opus, all 10 Qwen, 9 of 10 Gemini and human runs 1 and 3, open with exactly lines 1 to 9. That is one boundary of roughly 25 per run, so it barely moves kappa, but the first Gudea boundary is instructed for the models, not read.

A referenced simulation is absent. src/paper_results.py cites kappa_commensurability_sim.py to justify comparing a 2-way and a 7-way kappa; no file of that name is in the repo.

The source texts are public. The paper's Inanna and Enki translation is ETCSL 1.3.1, whose page tr131.htm (HTTP 200) has exactly 71 line-range blocks, the paper's 71 units. Inanna's Descent comes from CDLI P468903; the ETCSL version, 1.4.1, runs to line 412, against 405 units in the repo after lacunae are dropped. Gudea comes from RIME 3/1.01.07 via CDLI P431881, Cylinder A only (814 numbered lines); ETCSL 2.1.7 numbers Cylinders A and B together to line 1363.

What this means if you run repeat annotation or screen for AI-written labels

Schedule a repeat pass and score the layers separately. The cheapest takeaway is procedural: re-annotate a sample with the same annotator weeks later, and report boundary (or span) kappa and label kappa as two numbers. Inter-annotator agreement merges them. In this study the same person's boundaries reproduce at kappa 0.47 to 0.73 while labels sit at 0.17 to 0.29, which says the schema's categories, not the cut points, are where guideline work pays.

Do not use the gap as a pass/fail screen yet. On one text, the gap failed to separate Gemini from the human in 56 of 120 triples. On one run pair, the human looked model-like. A usable detector needs a threshold set on several humans, a false-positive rate on human pairs and a false-negative rate on models run at matched spacing. None of that exists yet; the paper says a panel is the next step.

If you screen anyway, pool across documents and use three or more runs. Requiring a large positive gap on each of three texts is the only rule that separated all models here. That is expensive: three documents, three passes each, months apart.

Keep the phase shift on your radar for span-labelling tasks. One-step lags at 2.5 and 1.8 times chance (Descent, Gudea) suggest that, for sequential labels, a tolerance window on label onsets would recover much of the apparent disagreement. Our judgement: for conflict or discourse-stage schemas, report onset agreement with a one-unit window next to unit-level kappa.

Check prompts for leaked worked examples. A schema example drawn from the corpus under annotation: 29 of 30 model runs on Gudea open with exactly the example's segment. If your LLM pre-annotation prompt shows an example from the batch, the first boundary is the example's, not the model's.

Check it yourself

Everything runs on Python 3 standard library in under a minute. Treat the clone as untrusted data; our independent script does not import it.

# repo: github.com/Malificenta883/intra-annotator-dynamics (MIT, commit 6baa8a8, checked 2026-10-06)
git clone https://github.com/Malificenta883/intra-annotator-dynamics repo
cd repo && python3 reproduce.py | grep -c '^  ok'      # 88 (figure step needs matplotlib)

# run file fields used: segments[].function, segments[].line_start, segments[].line_end, annotated_date
# units: texts/<myth>_numbered.txt, drop lines that are only "..." or "ms." notes; inanna_enki = 71 blocks
python3 -I check.py repo res.json      # own projection + Cohen's kappa, prints per reader:
#   inanna_enki    human1 label k=0.291 bnd k=0.731 gap=+0.440  (pe label 0.136, pe bnd 0.651)
#   inanna_descent human1 label k=0.203 bnd k=0.655 gap=+0.452
#   gudea          human1 label k=0.168 bnd k=0.465 gap=+0.298
#   gudea          gemini triples n=120 gap min=+0.086 max=+0.522
python3 -I rule.py res.json            # share of 120 triples per model/myth with gap >= human gap
#   gudea gemini 56/120, gudea opus 1/120, all others 0/120

# anchoring of the prompt example (seg_v4: line_start 1, line_end 9)
python3 -c "import json,glob;print(sum(int(sorted(json.load(open(f))['segments'],key=lambda s:int(s['line_start']))[0]['line_end'])==9 for f in glob.glob('repo/data/gudea/*.json')))"   # 31

# source texts
curl -s https://etcsl.orinst.ox.ac.uk/section1/tr131.htm | grep -oE '>[0-9]+(-[0-9]+)?<' | wc -l   # 71

Kappa for a pair is computed on the units both runs cover; the reported value is the mean over all pairs of a reader's runs (3 for the human, 45 for a model), exactly as in paper_results.boundary_label_kappas. Label: 7 categories. Boundary: a unit is 1 if a segment's line_start falls in it. A unit's label is the function of the segment covering most of its lines, ties to the earlier segment.

What would prove this wrong

Our claim is that the boundary-label gap, as released, works only as a pooled three-text pattern and not as a per-document check. The paper's planned panel can test it.

Dated prediction: if a panel study of at least three expert readers with two or more spaced runs each appears by 31 December 2027, at least one human reader will show a single-text gap below +0.15 on some text, and at least one model will show a single-text gap above that reader's minimum. If every human reader in such a panel has a gap above +0.25 on every text, and no model triple exceeds +0.25 on any text, then the gap is a per-document signature and we were wrong.

A second, cheaper test: rerun one model family on the same myth after a gap of three months or more, on a new model version. If its label kappa stays above 0.45 against the earlier runs, the level difference reflects readers, not spacing; if it falls below 0.30, the 0.30 label-kappa separation we found is a timing artefact.

Sources

1. Shvets, M. Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data. arXiv:2610.04370v1, Eötvös Loránd University, 3 October 2026. Sections 3 to 5, Tables 1 to 5, Limitations.
2. Shvets, M. intra-annotator-dynamics (MIT), created 29 September 2026, commit 6baa8a8 of 6 October 2026. Accessed 6 October 2026.
3. Black, J. A., et al. The Electronic Text Corpus of Sumerian Literature, Oxford, 1998 to 2006: 1.3.1 Inana and Enki, 1.4.1 Inana's descent, 2.1.7 The building of Ningirsu's temple. Accessed 6 October 2026.
4. Cuneiform Digital Library Initiative, P431881 and P468903. Accessed 6 October 2026.
5. Abercrombie, G., et al. Consistency is key: disentangling label variation in natural language processing with intra-annotator agreement. Proceedings of the 4th Workshop on Perspectivist Approaches to NLP, 2025.
6. Veselovsky, V., Horta Ribeiro, M., West, R. Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. arXiv:2306.07899, 2023.
7. Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 1960.

Related BLOMEGA research: LLM annotation reproducibility and construct validity · Effective human panel size in ChaosNLI · Data annotation: latest research

FAQ

Does one annotator's labelling drift more than their segmentation over time?

In the one case studied, yes. In arXiv:2610.04370 (3 October 2026) one expert re-annotated three Sumerian myths three times, 34 to 315 days apart. Her boundary kappa was 0.731, 0.655 and 0.465 and her 7-way label kappa 0.291, 0.203 and 0.168. BLOMEGA reran the released code (88 of 88 numbers match) and recomputed every kappa independently with the same result. Her raw label agreement, 31.6% to 37.6%, is about twice chance.

Can the boundary-label gap detect a model-written annotation?

Not on one text yet. The paper states no threshold. Taking all 120 three-run subsets of each model's ten runs, BLOMEGA found that on Gudea 56 of 120 Gemini triples reach the human's gap of +0.298, and one human run pair 315 days apart has a gap of only +0.091. Only a large positive gap on all three myths separated every model from this human.

Are the data and source texts for arXiv:2610.04370 public?

Yes. Runs, texts, prompts and code are on GitHub at Malificenta883/intra-annotator-dynamics under MIT (checked 6 October 2026). The earlier seg_v3 prompt used for Opus on Inanna's Descent was not kept, and a referenced kappa simulation file is missing. The translations come from ETCSL and CDLI; ETCSL 1.3.1 has exactly the 71 blocks used as units.