VoiceNet-Emo's three experts say "present" 11.6% to 67.4% of the time on the same prompts, and the sampling bit scores 0.716

What LAION released on 25 September 2026

Christoph Schuhmann, Robert Kaczmarczyk and eight co-authors (LAION, Forschungszentrum Jülich, L3S, TU Darmstadt, DFKI and others) posted VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale (arXiv:2609.32016v1, cs.SD). It is a benchmark and a model release in one paper.

VoiceNet-Emo asks "is emotion E present in clip C?" over a 40-emotion taxonomy, on 7,988 pairs drawn from 3,944 in-the-wild clips of Emilia-Large (Section 3.3). Each pair gets a 0 / 1 / 2 rating (not, weakly, strongly present) from three experts, named user_0 to user_2. VoiceNet-Ext, labelled preliminary by the authors, scores 57 talking-style attributes over 18.5k pairs with 40,990 ratings from eight annotators, at Fleiss kappa -0.022.

Alongside: Emolia (Emilia-Large with 40 classifier emotion scores per clip, stated at 71.78M clips; the balanced subset's own card describes laion/Emolia as 80.5M samples, and we did not count shards to settle it), a 5.26M-clip balanced subset, MOSS-Audio-8B-Thinking annotations with 155.46M reasoning traces, and two voice-text models. VoiceCLAP-Large, a rank-16 LoRA on LCO-Embedding-Omni-7B, reaches 0.702 per-prompt balanced accuracy on VoiceNet-Emo (Table 1). The labels live in LAION-AI/emolia-bench, where the subsets are called emolia-emo and emolia-dim.

The authors are unusually candid about agreement: they call Ext chance-level, flag the classifier-preselection circularity, and warn that the model-versus-LOO comparison is not like-for-like. What follows uses their own released files to go one step further.

Do the released files match the paper, and what else do they show?

The counts match. Everything we recomputed agrees with the paper to the third decimal; the new rows are ours.

Paper numbers from arXiv:2609.32016v1. BLOMEGA rows computed on 30 September 2026 from annotations/emolia-emo/annotations.csv and analysis_outputs/emolia-emo/benchmark_labels.csv at LAION-AI/emolia-bench main (last commit 25 September 2026). Present means a rating of weakly or strongly present, the paper's binarisation.

On likely prompts the three experts are close, at 69.5%, 70.7% and 76.0% present. On the 3,986 unlikely prompts they are not the same instrument. user_0 rejects nearly nine in ten; user_2 accepts two in three. Their full-set present rates are 40.6%, 58.1% and 71.7%. Kappa 0.086 is not three experts each disagreeing a little. It is mostly one systematic gap in how readily each expert hears a weak trace of an emotion.

The pairwise numbers say the same thing. user_0 and user_1 reach binary kappa 0.231 (alpha 0.208 on that pair), while both pairs involving user_2 sit at 0.080 and 0.027. The 0/1/2 scale does not rescue it: ordinal alpha is 0.100, and on 46.8% of pairs one expert says "not present" and another "strongly present". We note, without claiming cause, that user_2 is the rater whose declared setup is external speakers; the other two used headphones or earbuds.

Why the sampling column beats every model

Every pair is sampled under one of five task types (Section 3.3): 4,000 affirmative pairs present the emotion the preselector ranked top, and 3,988 present its opposites or its lowest two ranks. That choice is written into benchmark_labels.csv as task_type and intended_present. Experts confirm 78.85% of affirmative pairs and reject about 63% of the rest, so the column is a strong label predictor before any audio model runs.

We scored it directly. Predicting "present" for affirmative and "absent" otherwise gives 0.716 balanced accuracy against the expert majority on the 7,986 pairs with two or more raters (true-positive rate 0.683, true-negative rate 0.749, raw accuracy 0.711). That is above VoiceCLAP-Large at 0.702 with per-prompt thresholds fitted on the test labels, and above its 0.682 held-out score. It uses no threshold at all. The repo's own summary.json carries the same 0.7163 under gemini_vs_human_consensus_balanced_accuracy; the paper does not report it.

Which model made that prior is itself unclear. The paper says the EmpathicInsight-Voice classifier preselected VoiceNet-Emo prompts (Section 3.3 and the AI use statement). The repo's analysis.py comments and report.md say the preselection prior was produced by gemini-3-flash and treat it as a synthetic fourth rater. We could not resolve which is right for the Emo subset from the public files.

The negative within-type agreement follows from the rater gap. Inside each unlikely task type, binary alpha is -0.083 to -0.093; inside affirmative it is 0.083. Once the sampling prior is held fixed, what varies across raters is mostly their threshold, and alpha, which pools marginals, counts that as disagreement. Our judgement: a model that ranks clips the way the preselector does will look strong on this benchmark whatever it hears, and VoiceCLAP was trained on Emolia-Balanced, which was sampled with the same EmpathicVoice classifier's logits (Section 3.2).

Does agreement predict which emotions are hard? Chance-corrected, yes

Appendix A.12 reports that per-emotion unanimity correlates 0.02 with VoiceCLAP-Large rho across the 40 emotions, and concludes that agreement does not predict difficulty. We reproduce the 0.02 (we get 0.018). The problem is the statistic. Raw unanimity rises with prevalence: its Spearman correlation with the per-emotion majority-present rate is 0.72. Thankfulness/Gratitude is 68% unanimous because 98.1% of its pairs are majority-present; its binary alpha is -0.060. Doubt is 55% unanimous at 96.0% prevalence, alpha -0.060.

Chance-corrected, the relation the paper looked for is there. Per-emotion binary alpha correlates 0.54 with VoiceCLAP-Large rho (Spearman, p = 0.0003, 40 emotions) and 0.37 with the LCO-Omni-7B base (p = 0.019). The four emotions with alpha above 0.20, Sadness (0.263), Pleasure/Ecstasy (0.208), Distress (0.205) and Impatience/Irritability (0.201), are among the model's strongest rows, at rho 0.52 to 0.63. Eleven of 40 emotions have alpha at or below zero; the median is 0.045.

This also qualifies the paper's agreement-stratified test (App. A.15), which found no concentration of VoiceCLAP's edge in higher-agreement categories. That test concerns the gap between two models, not absolute difficulty, so the two results can both hold. We have not rerun the stratified split with alpha.

What to do with VoiceNet-Emo if you evaluate or train on it

Report the task-type baseline next to any score. A model under 0.716 balanced accuracy on VoiceNet-Emo has not beaten one released column. Better still, report bal@pp within affirmative and within the unlikely types separately, where the prior carries no information. Never feed task_type or intended_present to a model under test; both are in the label file.

Model rater thresholds, do not average them away. With present rates of 40.6%, 58.1% and 71.7%, the majority label is roughly "would the middle rater say yes". A per-rater bias term, or scoring against each rater's labels and reporting the spread, keeps that visible. The same logic sits behind treating three annotators as a sample rather than as ground truth.

Weight or mask by chance-corrected agreement, not unanimity. The paper suggests users may mask weak emotions such as Sourness, Doubt and Interest. By alpha, the mask would be different: Intoxication (-0.084), Thankfulness, Doubt, Infatuation, Pride and Teasing are the least reliable, and Sourness sits at 0.043. Unanimity-based masking keeps high-prevalence emotions whose agreement is below chance.

Check the licence per clip before any commercial use. 2,168 of the 3,944 clips carry Emilia-style names (B00013_S03009 pattern), which the paper places in the CC-BY-NC-4.0 portion; the other 1,776 carry YouTube-ID names from Emilia-YODAS (CC-BY-4.0). This split is our inference from file names. The Hugging Face cards for laion/Emolia, VoiceNet/emolia and VoiceNet/emolia-thinking declare only cc-by-4.0, laion/emolia-balanced-5M-subset declares none and defers to Emolia, and the emolia-bench repo that hosts the benchmark mp3s is MIT. Our judgement: "permissively licensed" in the abstract overstates it for the majority of the benchmark audio.

Treat this as a lesson for your own protocol. Calibrate raters on shared anchor clips before production, log the listening setup, and monitor per-rater present rate per task type in the first few hundred items. Here the gap between 11.6% and 67.4% on unlikely prompts was visible in the log from the start; annotation ran from 30 March to 15 April 2026. See also how majority vote behaves when individual accuracy is low.

Check it yourself

Only two CSVs are needed, about 5.6 MB together, no audio. This script prints the deduplicated row and item counts, the per-rater present rates by sampling type, and the task-type baseline.

# files: github.com/LAION-AI/emolia-bench (main, last commit 2026-09-25)
#   annotations/emolia-emo/annotations.csv   fields: username, queried_emotion, task_type, file_name, rating, created_at
#   analysis_outputs/emolia-emo/benchmark_labels.csv   fields: n_raters, majority_present, intended_present
python3 - <<'PY'
import pandas as pd
B="https://raw.githubusercontent.com/LAION-AI/emolia-bench/main/"
a=pd.read_csv(B+"annotations/emolia-emo/annotations.csv")
a["p"]=a.rating.ne("not_present").astype(int)
k=["file_name","queried_emotion","task_type"]
a=a.sort_values("created_at").drop_duplicates(k+["username"],keep="last")
w=a.pivot_table(index=k,columns="username",values="p").dropna()
print(len(a),len(w))
aff=w.index.get_level_values("task_type")=="affirmative"
print(w[aff].mean().round(3).to_dict())
print(w[~aff].mean().round(3).to_dict())
b=pd.read_csv(B+"analysis_outputs/emolia-emo/benchmark_labels.csv")
b=b[b.n_raters>=2]; y,p=b.majority_present,b.intended_present
print(len(b),round((((p==1)&(y==1)).sum()/(y==1).sum()+((p==0)&(y==0)).sum()/(y==0).sum())/2,4))
PY
# 23958 7984
# {'user_0': 0.695, 'user_1': 0.707, 'user_2': 0.76}
# {'user_0': 0.116, 'user_1': 0.455, 'user_2': 0.674}
# 7986 0.7163

# licence tags as declared on the cards
curl -s https://huggingface.co/api/datasets/laion/Emolia | grep -o 'license:[a-z0-9.-]*'

For alpha, the coincidence-matrix form (Krippendorff 2011) on the 7,984 x 3 binary matrix gives 0.086, matching Fleiss as expected with complete data. The per-emotion correlation uses Table 7's VoiceCLAP-Large column; we typed it from the PDF text, so a transcription check against the released per-tag files is worth doing. We also confirmed the Ext counts from annotations/emolia-dim/annotations.csv: 40,990 rows, 18,532 non-LANG items, 16,166 with two or more ratings, 5,583 with exactly three, Fleiss kappa -0.0218, all as in App. A.14. The committed analysis_outputs/emolia-dim/summary.json is stale (34,097 rows, 1,366 complete items).

What would prove this wrong

Our reading is that VoiceNet-Emo's low kappa is dominated by rater threshold differences rather than by random perceptual noise, and that its headline metric is largely reachable from the sampling prior. Two releases could break that.

First, if LAION adds raters, for example a fourth and fifth expert on the same 7,984 pairs, and the new raters' present rates on unlikely prompts also spread by more than 30 points while their pairwise alpha with the closest existing rater stays above 0.2, the threshold story holds. If the new raters cluster tightly on present rate and still reach alpha below 0.1 with each other, the disagreement is perceptual and our explanation is wrong.

Second, a dated prediction: by 30 June 2027, any model reported above 0.75 bal@pp on VoiceNet-Emo will score no more than 0.62 bal@pp when evaluated separately within the unlikely task types, where the prior carries no information. A model that clears 0.65 within those types would show the benchmark measures hearing, not agreement with the sampler.

Sources

1. Schuhmann, C., Kaczmarczyk, R., et al. VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale. arXiv:2609.32016v1, 25 September 2026. Sections 3.2, 3.3, 4.1, Table 1, Table 7, App. A.12, A.14, A.15.
2. LAION-AI. emolia-bench (MIT), annotations and analysis outputs, last commit 25 September 2026. Accessed 30 September 2026.
3. LAION-AI. voicenet (taxonomy and guides, created 15 July 2026) and emotion-annotations.
4. Hugging Face dataset cards: laion/Emolia (last modified 20 November 2025), laion/emolia-balanced-5M-subset (19 April 2026), VoiceNet/emolia-thinking (13 July 2026). Accessed 30 September 2026.
5. Krippendorff, K. Computing Krippendorff's Alpha-Reliability. Annenberg School for Communication, University of Pennsylvania, 2011.
6. BLOMEGA. Treat the three annotators as a sample and eight LLMs tie.
7. BLOMEGA. Once individual accuracy drops below 50%, majority vote makes rare-event labels worse.