One LLM labelled all 186,642 sentences of the Public Discourse Corpus, and its released check file disagrees with its paper

What changed, and when

Bo Chen (Institute of Computing Technology, Chinese Academy of Sciences, sole author) submitted on 27 July 2026, and arXiv announced in its September 2026 listings, The Public Discourse Corpus (PDC): A Speaker-Attributed Dataset for Valence and Epistemic Modality with Target Speaker Participation (arXiv:2609.20232v1). The corpus covers 998 YouTube interview videos of 100 public figures in seven professional domains (politics 22 speakers, finance 21, media 16, academia 15, geopolitics 14, central banking 9, technology 3), cut into 186,642 sentences and 3,113,156 words.

The human work went into one place: a five-way Target Speaker Participation (TSP) taxonomy that decides whether a retrieved video actually contains the target speaker talking. Two annotators labelled 188 candidate videos at kappa 0.616 (70.2% observed agreement), and 0.690 on the binary keep-or-drop decision. The sentence labels, affective valence (positive, negative, neutral) and epistemic modality (emphatic, hedged, neutral), were then produced zero-shot by DeepSeek-V4-Flash at temperature 0 in batches of 50. The paper validates them against a second LLM, GPT-5.5, on a stratified 1% sample. A human gold set of about 400 sentences is described as "currently in progress".

The dataset sits on Hugging Face under ictchenbo/public-discourse-corpus (CC BY-NC 4.0, created 27 July 2026, ungated). It ships the corpus, speaker and video metadata, and two validation files: GPT-5.5 at reasoning effort none, and a second run at extra_high that the paper does not mention. That second file lets anyone separate provider effects from reasoning-effort effects, which is what we did.

The released check file does not reproduce the paper's numbers

The headline valence figures land close (75.9% vs 77.0%, 0.63 vs 0.639). Modality moves the wrong way: the paper's 0.53 becomes 0.492, below the 0.5 that many teams use as a floor for keeping a label. The sample is 21 sentences smaller than the paper's 1,863, and the README states 1,842 in prose while reprinting the paper's per-domain table whose N column sums to 1,863. We cannot tell whether the file or the tables are the later version. Either way, a reader who downloads the corpus gets a validation file that disagrees with the published validation.

The per-domain ordering does not survive either. The paper names Central Banking the most consistent domain and Academia the least consistent on modality (0.46). In the released file Academia is among the most consistent on modality (0.530) and Central Banking sits at 0.487. Media valence drops from 0.66 to 0.575. With Ns of 101 to 467 per domain, differences of 0.05 are within sampling noise, so the honest reading is that the per-domain ranking in Table 10 carries no information a user should act on.

The extra-high reasoning run answers a question the paper leaves open. Raising GPT-5.5's reasoning effort does not bring it closer to DeepSeek: valence kappa 0.654, modality 0.471, and 21 polarity flips. The two GPT-5.5 settings agree with each other at 0.885 and 0.766. The gap between annotators is a provider effect, not a thinking-budget effect, so spending more on reasoning at annotation time would not have settled which labels are right.

The corpus annotator reads polarity into neutral talk

A kappa hides direction. Split the 1,842 valence pairs by which way they disagree and a lean appears.

DeepSeek-V4-Flash assigns positive or negative where GPT-5.5 says neutral 303 times; GPT-5.5 does it to DeepSeek-neutral sentences 100 times, a 3.0 to 1 ratio. On the sample DeepSeek marks 606 sentences negative (32.9%) against GPT-5.5's 525 (28.5%). Across the full corpus the DeepSeek labels come out 34.7% negative, 24.9% positive and 40.4% neutral. The paper's prompt instructs the model to "prefer neutral when uncertain"; the corpus annotator follows that instruction less than the validator does.

The true reversals, positive against negative, are few (20) and they are the kind of sentence where valence depends on stance rather than words. Two from the file: "The only reason the depression came to an end is because World War II started" (DeepSeek positive, GPT-5.5 negative) and "What we learned from Japan ... when the demography works against you ... you've got to have strong productivity" (same split). A human rater would need the interview context to decide either.

Modality has no comparable lean (215 DeepSeek-marked to GPT-neutral, 254 the other way). Its problem is plain disagreement on where "hedged" and "emphatic" begin, which is what the paper itself says: epistemic modality needs pragmatic inference and moves with each provider's decision boundary.

What it means for a team that wants to train or measure on these labels

• Treat PDC valence as DeepSeek-V4-Flash's opinion, not a measurement. Any finding of the form "central bankers are more negative than academics" inherits a 3.0 to 1 polar lean from one model. Re-run the analysis on GPT-5.5 labels for the 1,842-sentence sample before reporting it. If the sign changes, the finding belongs to the annotator.
• Do not use modality as a supervised target yet. Kappa 0.492 between two frontier models, and no human reference, means there is no defensible gold label to train toward.
• Ask for the check file with every LLM-labelled dataset, then recompute it. Here it took one join and a confusion table to find a 21-row sample mismatch and a modality kappa 0.04 lower than printed. Neither is fraud; both are what happens when tables are written before files are frozen.
• Reasoning effort is not an audit. The extra-high GPT-5.5 run agrees with the low-effort run at kappa 0.885 and with DeepSeek no better than before. Budget for a second provider or for humans, not for more thinking tokens from the same one.
• Check the rights before the labels. Judgement: the dataset card says the source material "was already in the public domain at the time of collection". Publicly viewable is not public domain. The 998 videos come from 558 channels, led by CNBC Television (37), Bloomberg Podcasts (34), Bloomberg Television (19) and CNN (18), and the released text is a transcript of their programming. CC BY-NC 4.0 already rules out commercial training; a commercial team would also need the broadcasters' permission, which the card does not claim.

The TSP taxonomy is the part worth copying. It is a clean answer to a real curation problem (is this video by the speaker or only about them) and it comes with a measured human reliability. The sentence labels are the part to hold at arm's length until the human gold set is published.

Check it yourself

About 45 MB, no account needed. Standard library only:

B=https://huggingface.co/datasets/ictchenbo/public-discourse-corpus/resolve/main
for f in main.jsonl gpt5_none_w6-80.0.01.jsonl gpt5_xhight_w6-80.0.01.jsonl; do curl -sLO "$B/$f"; done
python3 - <<'EOF'
import json, collections as C
k = lambda r: (r["video_id"], r["sentence_index"])
M = {k(r): r for r in map(json.loads, open("main.jsonl"))}
G = [json.loads(l) for l in open("gpt5_none_w6-80.0.01.jsonl")]
def kappa(p):
    n = len(p); po = sum(a == b for a, b in p) / n
    a, b = C.Counter(x for x, _ in p), C.Counter(y for _, y in p)
    pe = sum(a[x] * b[x] for x in a) / n / n
    return round(po, 3), round((po - pe) / (1 - pe), 3)
print(len(M), len(G))
for d in ("valence", "modality"):
    print(d, kappa([(M[k(r)][d], r[d]) for r in G]))
print("DeepSeek polar, GPT neutral:", sum(M[k(r)]["valence"] != "neutral" and r["valence"] == "neutral" for r in G))
print("GPT polar, DeepSeek neutral:", sum(M[k(r)]["valence"] == "neutral" and r["valence"] != "neutral" for r in G))
EOF
# expected: 186642 1842 / valence (0.770, 0.639) / modality (0.716, 0.492) / 303 / 100

• Swap in gpt5_xhight_w6-80.0.01.jsonl (the file name has the typo) for the extra-high run. Join it to the none file on the same key for the same-provider agreement.
• Per-domain: join main.jsonl speaker to speakers.jsonl domain. Our counts per domain were media 467, finance 410, geopolitics 320, academia 257, politics 249, central banking 101, technology 38.
• Channels and dates: videos.jsonl fields channel and upload_date; 531 of 998 videos were uploaded in 2026 and 256 in 2025.
• Construction code: github.com/ictchenbo/pdc-construction-pipeline; annotation tools: github.com/ictchenbo/pdc-annotator.

What would prove this wrong

Prediction, dated: when the authors publish the promised human gold set (about 400 sentences), human-vs-DeepSeek valence kappa will come in below 0.70, and humans will label more of those sentences neutral than DeepSeek-V4-Flash did. If humans instead side with DeepSeek on at least two thirds of the 303 polar-vs-neutral disagreements that fall in their sample, the lean we measured is GPT-5.5 under-reading polarity, not DeepSeek over-reading it, and the practical advice above flips. Check date: 30 June 2027, or the paper's v2, whichever comes first.

A second, cheaper test: a v2 of the dataset card should either restore a 1,863-row check file or correct Tables 9 and 10 to the 1,842-row numbers. If neither happens by the same date, treat all printed agreement statistics for this corpus as unverified.

Sources

1. Bo Chen (Institute of Computing Technology, CAS), The Public Discourse Corpus (PDC): A Speaker-Attributed Dataset for Valence and Epistemic Modality with Target Speaker Participation, arXiv:2609.20232v1, submitted 27 July 2026, announced September 2026. Sections 3.2.4, 3.4, 6.3, 7; Tables 4, 9, 10.
2. ictchenbo/public-discourse-corpus, Hugging Face dataset, created 27 July 2026, files main.jsonl, speakers.jsonl, videos.jsonl, gpt5_none_w6-80.0.01.jsonl, gpt5_xhight_w6-80.0.01.jsonl and README.md, retrieved 20 September 2026.
3. pdc-construction-pipeline and pdc-annotator, GitHub, linked from the paper.
4. Creative Commons, Attribution-NonCommercial 4.0 International, licence text.

Related BLOMEGA research: LLM annotators at kappa near zero · HH-RLHF preference label noise · Data annotation: latest research · Data provenance and chain of title

FAQ

Who labelled the Public Discourse Corpus?

All 186,642 sentence labels (valence and epistemic modality) come from one model, DeepSeek-V4-Flash, run zero-shot at temperature 0. Humans labelled only the video-level Target Speaker Participation step (2 annotators, 188 videos, kappa 0.616). A human gold set for sentences was described as in progress in arXiv:2609.20232v1 (submitted 27 July 2026).

How reliable are the PDC valence and modality labels?

Recomputed from the released 1,842-sentence check file, DeepSeek-V4-Flash and GPT-5.5 agree at kappa 0.639 on valence and 0.492 on modality. The paper reports 0.63 and 0.53 on 1,863 sentences. There is no human reference yet, so these are agreement figures, not accuracy.

Does more reasoning effort make LLM annotations agree better?

Not here. GPT-5.5 at extra-high reasoning agrees with DeepSeek at kappa 0.654 (valence) and 0.471 (modality), no better than at reasoning none, while the two GPT-5.5 settings agree with each other at 0.885 and 0.766. Provider choice moves labels more than reasoning budget.

Can the PDC be used to train a commercial model?

No. It is released under CC BY-NC 4.0, which excludes commercial use, and its text is transcribed from YouTube programming by 558 channels including CNBC, Bloomberg and CNN. The dataset card calls the source content public domain; being publicly viewable does not make it public domain.