In the CuBEs release, a culture-blind judge matches a culture's majority 81.1% of the time and a fifth annotator from that culture 66.0%

What did the CuBEs paper claim on 2 October 2026?

Hoda Ayad and Tanu Mitra (University of Washington) and Abhishek Mukherji (Centific) posted CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges as arXiv:2610.02622v1 (cs.AI) on 2 October 2026. The code README describes it as an ACL submission under review. CuBE extends Anthropic's MIT-licensed Bloom pipeline so that scenario generation, user simulation and judging all receive a country and a social domain (Work, Health, Family).

Two experiments carry the argument. First, a judge-alignment study: claude-sonnet-4.5 wrote culture-localized vignettes for 11 alignment questions across six behaviors (self-preservation, delusional sycophancy, political bias, instructed long-horizon sabotage, self-preferential bias, implicit stereotyping), each with pre-written justifications for labels A (exhibits), B (partially) and C (does not). The paper reports 60 crowd annotators, 5 per country, 31 vignettes each, 1,860 labels, and four candidate judges scored against the human majority. claude-opus-4.5 reached quadratic-weighted kappa 0.633; gemini-3-flash 67.7% accuracy and kappa 0.631, and was chosen on cost. Within-culture annotator agreement (Table 6) runs from 0.543 in Nigeria to 0.103 in France, 0.280 overall.

Second, a scale-out: 13 target models, 6 behaviors, 12 countries plus a no-culture baseline, 9 rollouts per cell, scored 1 to 10 by gemini-3-flash. The headline gap is claude-sonnet-4.5 on political bias: 8.78 at baseline against a 12-culture mean of 2.72, a 6.06-point drop.

The study was funded through a data provider. Appendix C.4 says ethics review was done by a committee "within the data provider's Quality Management division", and C.3 says only a partial sample is public because the full dataset is proprietary. Annotation cost $3,022.50 in production plus $2,362.50 across two pilots, against $1,300 of API spend (Table 7).

What is actually in the release, and do its numbers match the paper?

The paper's footnote points to an anonymised repository that links an OSF project, mjkcr, created 26 May 2026, reachable through a view-only link. On 7 October 2026 it held two zip files. We downloaded both and recounted.

Release files: cube_judge_alignment.zip (427,804 bytes, sha256 d51af09b...a498) and cube_cross_cultural_evaluation.zip (17,718,164 bytes, sha256 c35eb3bb...df4f) from OSF project mjkcr, downloaded 7 October 2026. The OSF node lists no licence; the code repository is CC BY-NC 4.0 with Bloom-derived parts under MIT.

Where the release can check the paper, it mostly agrees. The Health-only political-bias gap for claude-sonnet-4.5 (+5.89) sits close to the paper's three-domain +6.06, and Figure 3 reproduces cell for cell. The within-culture kappas land near Table 6 without matching exactly; the paper does not state how it pooled pairs across a rotating pool of annotators, and our two weightings for Pakistan (0.252 unweighted, 0.407 weighted by shared items) bracket its 0.367. Table 9 lists 14 gaps with intervals clear of zero, one fewer than the text's "fifteen"; with 78 tests at the 5% level, 3.9 would be expected by chance, so the aggregate claim that baselines mislead holds either way.

The release also shows things the paper does not report. "5 annotators per country" is 5 labels per vignette drawn from 6 to 11 people, so per-annotator effects cannot be separated from per-vignette effects. In Nigeria, 21 of 157 label slots hold free-text comments or nothing, including one vignette where an annotator wrote that the scenario panel was blank; Nigeria is also the country with the highest agreement.

Is the culture-blind judge worse than a second human from the same culture?

The paper compares judges to the human majority and compares humans to each other, but never puts a single human and a judge against the same reference. We did. For each vignette and each annotator, the reference is the majority of the other valid labels on that vignette (ties skipped, 491 comparisons over 123 vignettes). The held-out annotator and each judge are scored against that identical reference.

gemini-3-flash, the judge the paper picked on cost, is 15.1 points above the held-out human, and claude-opus-4.5 is 10.8 points above with an interval of +2.9 to +18.7. claude-sonnet-4.5 (71.3%) and gpt-5.1 (67.6%) are level with the human within their intervals. No judge is reliably below the human in any of the four countries.

By country, the gap is largest where the humans agree least. In Pakistan the held-out human matches the others 59.3% of the time and gemini-3-flash 89.8% (+30.5, interval +21.8 to +40.7). In the United States, the most lenient released country, the human sits at 72.4%, gemini-3-flash at 80.3%, and claude-opus-4.5 slightly below the human at 70.9%; the US is the one country where three of four judges score under the held-out human, though none significantly. Our reading, offered as a judgement: the weak spot of these judges is not non-Western annotators.

Three caveats limit this. The four countries were picked by the authors to span Figure 3's strict-to-lenient range, not at random, and France (kappa 0.103) and India (0.132), the two lowest-agreement countries, are not released. Each vignette came with three label justifications written by claude-sonnet-4.5, which may make the task easier for LLM judges than for crowd workers. And the paper's own 12-country figure for gemini-3-flash, 67.7% accuracy against the full majority, is well below the 83.0% we get on these four, so the remaining eight countries are harder on average.

How does a Figure 3 cell get made, and where can it tip?

Figure 3 shows a modal label per country and question, then a per-country "average lean" with A=1, B=0.5, C=0. The paper does not say how the 10 to 15 labels behind a cell become one letter. We tried rules until one reproduced the published grid: take each vignette's majority, then the mode across vignettes, and break every tie toward the stricter label. That rule matches all 44 released cells and all four published leans (Pakistan 0.59, Philippines 0.45, Nigeria 0.27, United States 0.23).

Five of those 44 cells are ties. Pakistan's shutdown question (4 A, 4 B, 7 C over 15 labels) shows A in the figure; under a plain mode over all labels it is C. Pakistan's political-balance question (2 A, 6 B, 7 C) shows B; its statistical-stereotype question (3 A, 4 B, 3 C) shows A. The tie rule is a defensible choice, but it is a choice, and it moves the headline ranking.

Under plain pooled mode Pakistan's lean falls from 0.59 to 0.41 and ties the Philippines. On the mean of all labels, which uses every vote, Pakistan stays the strictest at 0.473, but the United States (0.305) and Nigeria (0.309) are level. The ordering survives for Pakistan; the size of the gaps does not.

Do the cross-cultural differences survive 10 to 15 labels per cell?

Two tests point in different directions, and both matter for anyone sizing a multi-country panel.

Question by question, the evidence among the four released countries is thin. For each of the 11 questions we ran a permutation test that shuffles country labels across vignettes (vignettes, not labels, are the unit, because the five labels on one vignette are not independent), 5,000 permutations each. Two questions come in under p = 0.05: political balance (p = 0.029) and stated stance versus assistance (p = 0.041). None survives a Bonferroni correction for 11 tests. The paper calls the human-label variation "significant" but reports no test on the human labels.

At the level of the whole Figure 3 grid, the pattern is beyond sampling noise. We built a culture-free null: pool the released vignettes per question, deal 12 synthetic countries the same number of vignettes at random, and aggregate with the paper's rule. Over 5,000 draws the null produces a mean of 2.0 questions showing all three labels across countries (90% range 0 to 3) and 4.4 unanimous questions (range 3 to 5). Figure 3 has 8 and 1. Something varies by country. Because every country saw its own vignettes, that something is annotator culture and vignette content together, and the released design cannot split them.

The judges do flatten that variation. On the same vignettes, the human majority's lean runs from 0.250 (Nigeria) to 0.444 (Pakistan), a spread of 0.194. claude-sonnet-4.5 spreads 0.055, gemini-3-flash 0.088, claude-opus-4.5 and gpt-5.1 0.145. Every judge is more lenient than the Pakistani majority (0.339 to 0.371). So the judge wins on per-item agreement because it tracks the dominant label, and loses on the thing CuBEs set out to measure, the country-level shift.

What should a team running multi-country annotation or LLM judges do with this?

Benchmark the judge against a held-out human, not against the majority. A judge's kappa against a five-person majority looks low or high with no anchor. Scoring one held-out annotator against the same leave-one-out reference costs nothing extra once you collect five labels per item, and here it turns a kappa of about 0.63, which the paper reads as judges still falling short of cultural nuance, into a judge that beats a typical in-country annotator item by item. It is the comparison a buyer needs to decide whether to replace a crowd layer with a judge.

Measure the country shift separately from item accuracy. The judges fail on lean spread, not on agreement. Report per-country lean (or label distribution) for humans and judge side by side, with an interval. A judge whose lean varies a third as much as the humans' will produce country comparisons that look calmer than reality, even at 80% item agreement.

Cross the design if you want a culture effect. CuBEs gives each country its own vignettes, which is the realistic choice for situated evaluation and the wrong one for attribution. A crossed block, for example 10 shared vignettes labeled by every country alongside the localized ones, separates "people read this differently" from "this vignette was harsher". It adds about a third to the label budget at the paper's scale.

Budget per cell, not per country. At 5 labels on 2 or 3 vignettes, 5 of 44 released cells were exact ties and our bootstrap kept the modal label in fewer than 80% of resamples for 15 of 44 cells. If cell-level verdicts matter, plan for 8 to 10 vignettes per cell or report label distributions instead of a single letter. At the paper's production rate of $1.63 per label (hourly rates from $5 in India to $15 in Germany and Canada), tripling vignettes on 11 questions in 12 countries adds roughly $6,000.

Validate labels at ingest. In the released Nigeria file, 13.4% of label slots are free text or empty, and one vignette was labelled while its scenario panel failed to load. A schema check that rejects non-A/B/C values at submission, plus a render check per task, would have caught both. Median time per label also varies more than threefold, from 79 seconds in the US to 254 in Pakistan, and in this sample fast labels agree with peers more often than slow ones (73.5% versus 64.1%), so time alone is a poor quality filter.

How can I reproduce the 81.1% versus 66.0% comparison?

One 428 KB zip, no API keys. The script joins each human task to its LLM-judge task by the text of option A, takes the majority of the other valid labels on each vignette, and scores the held-out human and each judge against it.

# OSF project mjkcr (view-only link from the paper's code README), checked 7 October 2026
curl -sL "https://osf.io/download/9ez2n/?view_only=667be584e83b47c9a5915be38a1fea85" -o ja.zip
shasum -a 256 ja.zip   # d51af09b809b49b58691997c7dae9921ae5fc0316db8761eb6a382e411a3f498
mkdir ja && unzip -q ja.zip -d ja
# fields: human_annotations.json  {country: {annotator_count, task_count, tasks:[{pod_title, options{A,B,C}, annotations:[{annotator, choice, lead_time_seconds}]}]}}
#         llm_annotations.json    {country: {tasks:[{data{option_a,...}, annotations:[{completed_by{username}, result:[{from_name:"vignette_label", value{choices}}]}]}]}}
python3 -I - <<'PY'
import json, collections as C
H = json.load(open("ja/judge_alignment/human_annotations.json"))
L = json.load(open("ja/judge_alignment/llm_annotations.json"))
def maj(l):
    c = C.Counter(l).most_common()
    return None if not c or (len(c) > 1 and c[0][1] == c[1][1]) else c[0][0]
hits = C.Counter(); n = 0
for k in H:
    lt = {t["data"]["option_a"].strip(): t for t in L[k]["tasks"]}
    for t in H[k]["tasks"]:
        x = lt.get(t["options"]["A"].strip())
        if not x: continue
        j = {a["completed_by"]["username"]: r["value"]["choices"][0]
             for a in x["annotations"] for r in a["result"] if r["from_name"] == "vignette_label"}
        labs = [(a["annotator"], a["choice"]) for a in t["annotations"] if a["choice"] in ("A","B","C")]
        for who, lab in labs:
            ref = maj([l for w, l in labs if w != who])
            if ref is None: continue
            n += 1; hits["held-out human"] += lab == ref
            for m, v in j.items(): hits[m] += v == ref
print(n, {m: round(h / n, 3) for m, h in hits.items()})
PY
# 491 {'held-out human': 0.66, 'claude-sonnet-4.5': 0.713, 'claude-opus-4.5': 0.768,
#      'gemini-3-flash-preview': 0.811, 'gpt-5.1': 0.676}

The code repository's file list answers at https://anonymous.4open.science/api/repo/CuBE-framework-B315/files/; the HTML page returns 401 to plain curl. The rollout file is cube_cross_cultural_evaluation.zip (download id 6a162252a387115ead82921b), whose scores.csv has fields behavior, culture, domain, target_model, variation, repetition, behavior_presence. Our Figure 3 grid was typed from the PDF image; the tie-rule reconstruction, permutation tests and culture-free null use the same two JSON files. The 14-of-78 count parses Table 9 from the arXiv v1 PDF.

What would prove this wrong?

Our claim is that, on CuBEs-style vignettes, a frontier judge matches an in-country majority at least as well as one more in-country annotator, while compressing between-country differences. Two outcomes would break it.

First, by 30 June 2027, if the authors or the data provider release labels for the remaining eight countries (or the camera-ready reports a single-human baseline), gemini-3-flash's leave-one-out agreement will be at or above the held-out human's in at least 9 of the 12 countries, and pooled over all 12 it will lead by 5 points or more. If the human leads pooled, or the judge trails in France and India by more than 10 points each, the "judge is not the weak link" reading is wrong for the low-agreement cultures that matter most.

Second, if a crossed study (the same vignettes labeled in several countries) finds country explains under 5% of label variance once vignette is a random effect, then Figure 3's diversity was mostly vignette content, and the paper's human-label finding needs restating.

Sources

1. Ayad, H., Mitra, T. (University of Washington), Mukherji, A. (Centific). CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges. arXiv:2610.02622v1, 2 October 2026. Sections 3.2, 4.1, 4.2, Figure 3, Tables 2, 4, 6, 7, 9, App. B.3, C.3, C.4.
2. CuBE authors. CuBE-framework (anonymised repository, CC BY-NC 4.0; Bloom-derived parts MIT). README accessed 7 October 2026.
3. CuBE authors. OSF project mjkcr, "CuBE", created 26 May 2026: cube_judge_alignment.zip and cube_cross_cultural_evaluation.zip. Downloaded 7 October 2026.
4. Gupta, I., Fronsdal, K., Sheshadri, A., et al. Bloom: an open source tool for automated behavioral evaluations, 2025. github.com/safety-research/bloom (MIT, last push 7 May 2026).
5. BLOMEGA. How many humans is an LLM judge panel worth?
6. BLOMEGA. LLM judges versus physicians.

FAQ

Are culture-blind LLM judges worse than human annotators from the same culture on CuBEs?

Not item by item in the released sample. On 491 comparisons across Pakistan, the Philippines, Nigeria and the United States, gemini-3-flash agrees with the majority of the other same-culture annotators 81.1% of the time and claude-opus-4.5 76.8%, against 66.0% for a held-out annotator from that culture. gpt-5.1 (67.6%) and claude-sonnet-4.5 (71.3%) are statistically level with the human. The judges do vary their strictness less across countries than the humans do.

Is the CuBEs human-labeled dataset publicly released?

Partly. As of 7 October 2026 an OSF view-only project (mjkcr, created 26 May 2026) holds judge-alignment labels for 4 of the 12 cultures: 622 of the paper's 1,860 human labels, plus LLM judge labels, and 468 cross-cultural rollout scores for the Health domain and 2 of 13 target models. The paper says the full dataset is proprietary to the data provider. The code is on anonymous.4open.science under CC BY-NC 4.0.

How many annotators per culture labeled each CuBEs item?

Five labels per vignette, from a pool of 6 to 11 distinct annotators per released country (the paper says 5 per country). Each of the 11 alignment questions has 2 or 3 culture-specific vignettes per country, so each cell in the paper's Figure 3 rests on 10 to 15 labels; 5 of the 44 released cells are ties that the figure breaks toward the stricter label.