Four frontier LLMs misjudge everyday norms in 90 societies. The average of the other 89 societies does better in every one

The average of 89 other societies predicts a market's norms better than a model asked about that market

The benchmark is the Global Study of Everyday Norms (GSEN): 25,422 respondents in 90 societies rated 150 scenarios, 15 behaviors such as crying, bargaining or kissing placed in 10 situations such as a funeral, a job interview or a bus, on a scale from -2.5 (extremely inappropriate) to +2.5 (Materials and Methods). Each model was asked, five times per cell, for the average rating respondents in a named society would give.

The paper scores within-society error with the level offset removed, so a model that rates everything a bit too harshly is not penalised for it. On that measure, which is ours to average (the paper reports only the range for GPT-5, 0.60 to 1.34), Claude Opus 4.6 averages 0.64, GPT-5.4 0.70, Gemini 3.1 Pro 0.70 and GPT-5 0.78. The leave-one-out world average scores 0.36. Restricting the average to other societies in the same GSEN region (West, Latin America, Asia, Africa) scores 0.30.

The gap is not close anywhere. The smallest margin in the 360 society-by-model pairs is Claude Opus 4.6 in Denmark, 0.067 points. Saudi Arabia is where the models come nearest as a group (mean of the four models 0.71 against 0.57 for the world average, ratio 1.25), because its norm profile is among the least typical in the survey. In Mexico the four-model mean is 0.74 against 0.22, more than three times the error.

To give the models every advantage, we also fitted one straight line per model mapping its answers onto the survey scale, using the survey itself (slopes 0.57 to 0.66). That in-sample rescaling cuts model error to 0.45 to 0.53, and the world average still wins in 80 to 83 of the 90 societies. Our judgement: a model's answer to "how acceptable is X in country Y" carries less information about country Y than the scenario's global average does.

What Eriksson, Vartanova and Strimling reported on 25 September 2026

arXiv:2609.30896, Large language models underestimate and partly misrepresent cultural variation in everyday norms, by Kimmo Eriksson, Irina Vartanova and Pontus Strimling (Institute for Futures Studies, Mälardalen, Uppsala and Linköping universities), was posted on 25 September 2026. It tests GPT-5 as an exploratory run preregistered on 11 September 2025 (osf.io/d7km9), then GPT-5.4, Claude Opus 4.6 and Gemini 3.1 Pro under a confirmatory preregistration filed on 13 April 2026 (osf.io/w3fxd).

Two headline results. Compression, the ratio of the models' between-society standard deviation to the survey's, averages 0.43 to 0.49 across scenarios, and 47% to 54% after correcting the survey for sampling noise (Table S1). Pattern accuracy, the correlation between model and survey across societies for one scenario, averages 0.23 to 0.33 (Table 1), and 49% to 66% of scenarios fall below r = 0.30. The authors find accuracy higher for behaviors respondents call vulgar, partial r 0.48 to 0.54, and only modest gains from prompting in the survey's local language: error fell by 0.02 to 0.11 points in 16 societies (Table S9).

The paper compares models with each other, with a noise ceiling (0.90, Supplementary Methods) and with individual respondents (Table S19). It uses the leave-one-out average of other societies only to measure how atypical a society is (Materials and Methods), never as a competing predictor. That prediction is the reference a localization team with access to the public GSEN file would actually use.

Every Table 1 number reproduces from the OSF deposit, and both survey baselines beat all four models

The deposit at osf.io/jzhqu (view-only link printed in the paper, node last modified 29 September 2026) holds data/analysis/cells.csv, 13,468 society-scenario survey means, and data/analysis/calls.csv, 64.9 MB with one row per model reply. The main arm has 67,340 calls per model, exactly 13,468 cells times five. Averaging the calls and applying the paper's definitions gives every Table 1 value to two decimals, the GPT-5 offset-adjusted range of 0.60 to 1.34 and the cell counts (median 54 ratings per cell, range 1 to 217).

All "ours" rows computed on 30 September 2026 from the OSF deposit. Regional average uses the GSEN four-region classification in society.csv. The world average's cross-cultural r is -1 because a leave-one-out mean falls exactly as the left-out value rises; it is not a meaningful pattern score. The regional average also compresses (0.50), since averages of neighbours are smoother than any one society.

Two rows speak directly to the question a localization lead asks. Naming the society in the prompt helps each model only a little: offset-adjusted error drops from 0.67 to 0.64 for Claude Opus 4.6 and from 0.80 to 0.70 for GPT-5.4, and the named prompt is better in 65 to 83 of the 90 societies. And adding a model's society-specific tilt (its estimate minus its own cross-society mean for that scenario) to the world average does not improve the world average in most societies. The tilt that the paper finds compressed to under half size also carries too little correct direction to help.

Where the models do carry signal is narrow, and the paper names it. Averaged over the four models, pattern accuracy for kissing, flirting and cursing is 0.60, 0.55 and 0.41, level with or above the regional average (0.57, 0.54, 0.26). For every other behavior the neighbours win, most sharply for crying (0.54 against 0.16), talking (0.50 against 0.19) and resting (0.50 against 0.16).

Most of the models' error is about the behavior, not the society

A two-way split of each model's squared error over the 13,468 cells shows why a baseline with no knowledge of the target can win. 44% (GPT-5.4) to 63% (GPT-5) of the error is a scenario-profile miss that is the same in every society: the model misjudges how acceptable crying at a funeral or bargaining in a restaurant is anywhere, and repeats that miss 90 times. The society-by-scenario part, the actual cultural signal the paper is about, is 20% to 30%.

The paper documents the ingredient: within a society, the models spread ratings across scenarios 1.33 (Claude Opus 4.6) to 1.58 (GPT-5) times wider than respondents do (Table S12, amplification), while squeezing differences between societies to under half. The models are too confident about which behaviors are acceptable in general and too timid about how societies differ. In the survey, 79.4% of the variance in cell means is between scenarios, so the world average captures most of what can be known about a cell before any culture enters.

That split is also why the regional average is the harder baseline. It keeps the scenario profile and adds a share of the society-by-scenario signal through geography, which reaches r = 0.41 without any model.

The preregistered tests pass as filed; the development figure has moved since arXiv v1

We read both registrations against the paper. The confirmatory record w3fxd states H1 (vulgar share among the three concern answers predicts per-scenario r, controlling for between-country SD, OLS with HC3 errors, p < .05) and H2 (mean compression below 1, 95% scenario bootstrap entirely below 1). The paper tests exactly these, and both pass for all three models (Table 1, Table S5). The authors disclose three deviations themselves: Gemini 3.1 Pro returned reasoning tokens on every call (median 294) although the registration says no model reasons; 84 failed GPT-5 calls were not retried although d7km9 promised three attempts; and H1 and H2 were formed from GPT-5 numbers (compression 0.58, r 0.209) that a parsing fault had computed from 15.6% of the calls, later corrected to 0.49 and 0.27.

One discrepancy the paper does not mention: the development result no longer matches the deposit. arXiv v1 says the Human Development Index is the 2021 value from the Human Development Report 2021/22, available for 85 societies, with error falling with HDI at r = -0.52 to -0.77 (Table S7) and -0.72 on the four-model average (Figure 3). The deposited paper_benchmark.qmd instead reads 2023 values from the HDR 2025 time series, and its society.csv has HDI for 86 societies. On that file we get r = -0.57 (GPT-5.4) to -0.80 (Gemini 3.1 Pro) and -0.76 on the average. Same direction, somewhat steeper. Our reading is that the deposit has moved ahead of v1, and a v2 will likely print these figures.

What this means for a localization team asking an LLM about market X

GSEN places Saudi Arabia, Turkey and the other Middle Eastern societies in Asia. Kissing items were not fielded in Saudi Arabia, so its profile has 140 scenarios.

• Do not use a model as the source for "is this acceptable in market X". For the 11 markets above, the best single model's error is 1.2 to 2.7 times the world average's. If a content review step asks an LLM whether a scene with shouting, crying or phone use will offend in a market, the answer is closer to the model's global prior than to that market.
• Use the public survey where it covers your case. The GSEN respondent file is CC BY 4.0 at osf.io/sh4rb. For the 150 scenarios it holds, measured means for the target society beat any estimate, and neighbouring-society means beat every model tested.
• Where you must use a model, ask about vulgarity-type content and discount the rest. Kissing, flirting and swearing are the only behaviors where the models match the regional average. For everyday conduct (talking, eating, reading, resting) their society ranking is barely above chance, r = 0.16 to 0.26 (our four-model means).
• Expect local-language prompting to help a little. The paper's 16-society test cut error by 0.02 to 0.11 points and removed 22% to 69% of the compression shortfall (Table S9). That does not close a gap of 0.28 to 0.42 points to the world average.
• Budget for native reviewers in the markets where models fail worst. Four-model error is highest in Africa (0.78 to 0.97 offset-adjusted, Table S20), and the 11 worst societies in our four-model ranking are nine African societies plus Peru and Bolivia, led by the Central African Republic at 1.18. Those are the markets where a model's cultural prior is least connected to local judgement.

Our judgement on scope: the benchmark is 150 scenarios from mostly student samples (mean student share 70%, mean age 28), and it measures appropriateness, not offence at specific imagery or wording. The comparison with the world average is also partly a calibration advantage, because it uses the same instrument. Neither caveat helps the models much: the student-only means correlate with full-sample means at r = 0.98, and even an in-sample rescaling of every model leaves the world average ahead in 80 or more societies. For how LLM judgements fail in the neighbouring task of translation scoring, see LLM judges rank machines above human translators.

Check it yourself

The script reads three files from the deposit's view-only link (about 65 MB) and prints each model's offset-adjusted error, how many of the 90 societies the world average wins, and the cross-cultural r that should match Table 1. It ran in under 30 seconds on our machine.

pip install pandas numpy
cat > gsen_baseline.py <<'PY'
import pandas as pd, numpy as np
VO = "?view_only=4c217744f7564e8c8eef583e3ca6fd13"
cells = pd.read_csv("https://osf.io/download/6abaef6a58537b8039a2771d/" + VO)   # 13,468 society x scenario survey means
calls = pd.read_csv("https://osf.io/download/6abaef7fb640e7210250b7b5/" + VO,  # 65 MB, one row per model reply
                    usecols=["model", "grid", "arm", "country", "scenario", "rating"])
soc = pd.read_csv("https://osf.io/download/6abaef5e8ea6e0639239da1c/" + VO)
calls = calls[(calls.grid == "cross") & (calls.arm == "main")]
llm = calls.groupby(["model", "country", "scenario"]).rating.mean().unstack("model")
d = cells.set_index(["country", "scenario"]).join(llm).reset_index()
d["region"] = d.country.map(soc.set_index("country").region)
g = d.groupby("scenario").approp            # world average of the OTHER societies
d["world_avg"] = (g.transform("sum") - d.approp) / (g.transform("count") - 1)
g = d.groupby(["scenario", "region"]).approp  # same, other societies in the same GSEN region
d["region_avg"] = (g.transform("sum") - d.approp) / (g.transform("count") - 1)
def adj_mae(col):  # offset-adjusted within-society MAE, as in the paper's Methods
    e = (d[col] - d.approp).groupby(d.country)
    return e.apply(lambda v: (v - v.mean()).abs().mean())
def xc_r(col):     # Table 1 cross-cultural r; a constant column scores 0, as in the paper
    return np.mean([np.nan_to_num(x[col].corr(x.approp)) for _, x in d.dropna(subset=[col]).groupby("scenario")])
base = adj_mae("world_avg")
for m in list(llm.columns) + ["world_avg", "region_avg"]:
    a = adj_mae(m)
    print(f"{m:16s} adj MAE {a.mean():.2f}  societies where world_avg is lower: {(base < a).sum()}/90  cross-cultural r {xc_r(m):.2f}")
PY
python3 gsen_baseline.py

Expected output: Claude Opus 4.6 0.64, GPT-5 0.78, GPT-5.4 0.70, Gemini 3.1 Pro 0.70, each with "world_avg is lower: 90/90", and cross-cultural r 0.23, 0.27, 0.24, 0.33 (Table 1). world_avg prints 0.36, region_avg 0.30 with r 0.41. The model labels come from the model column (GPT-5 appears as GPT-5 (parent)). For the HDI check, correlate the per-society Claude or Gemini error with society.csv column hdi (2023 values, 86 non-missing).

What would prove this wrong

Our claim is that on GSEN-style norm questions, a prediction built from other societies' survey means beats current frontier models asked about the target society, and that this holds because the models' error is mostly a scenario-level miscalibration rather than missing cultural knowledge alone.

It would be wrong if the public release of the deposit, promised on acceptance, changes calls.csv or cells.csv so that any of the four models beats the world average in more than 5 of the 90 societies. We also predict that by 30 June 2027, no model run with the paper's elicitation prompt on the full 90 by 150 grid, without retrieving GSEN data, will reach a mean offset-adjusted error below 0.36 or a mean cross-cultural r above 0.41. A published run that does either, with its calls deposited, falsifies the prediction.

Sources

1. Kimmo Eriksson, Irina Vartanova, Pontus Strimling. Large language models underestimate and partly misrepresent cultural variation in everyday norms. arXiv:2609.30896v1, 25 September 2026. Table 1, Materials and Methods, Supplementary Methods, Tables S1, S3, S5, S7, S9, S12, S19, S20, Figure 3.
2. OSF deposit osf.io/jzhqu (view-only key in the paper), files data/analysis/cells.csv, calls.csv, society.csv, code/paper_benchmark.qmd, README.md. Node modified 29 September 2026, downloaded 30 September 2026.
3. Confirmatory preregistration osf.io/w3fxd, registered 13 April 2026; exploratory GPT-5 preregistration osf.io/d7km9, registered 11 September 2025.
4. Eriksson, Strimling, Vartanova, Simpson, Persson et al. Everyday norms have become more permissive over time and vary across cultures. Communications Psychology 3, 66 (2025). GSEN respondent data, CC BY 4.0: osf.io/sh4rb.
5. BLOMEGA recomputation, 30 September 2026: Table 1 reproduction, world and regional leave-one-out baselines, no-society and hybrid comparisons, in-sample rescaling, error decomposition, per-behavior and per-market tables, HDI correlation on the deposited society.csv.

FAQ

Can a localization team ask an LLM whether a behavior is acceptable in a given country?

Not as a primary source. On the GSEN benchmark of 150 everyday scenarios in 90 societies, GPT-5, GPT-5.4, Claude Opus 4.6 and Gemini 3.1 Pro estimate between-society differences at 43% to 49% of their measured size and correlate with the society pattern at only r = 0.23 to 0.33 per scenario (arXiv:2609.30896, Table 1). Our recomputation from the OSF deposit finds the average of the other 89 societies' survey ratings has lower offset-adjusted error than every model in all 90 societies.

What baseline beats the LLMs on the GSEN norms benchmark?

Predicting each society with the mean rating of the other 89 societies for the same scenario, which uses no information about the target. Its offset-adjusted within-society error is 0.36 scale points against 0.64 (Claude Opus 4.6), 0.70 (GPT-5.4, Gemini 3.1 Pro) and 0.78 (GPT-5). Averaging only the other societies in the same GSEN region gives 0.30 error and a cross-cultural r of 0.41, above the best model's 0.33. Both are BLOMEGA computations from the deposited cells.csv and calls.csv.

For which behaviors are LLM norm estimates closest to useful?

Kissing, flirting and cursing, the behaviors respondents most often call vulgar. Averaged over the four models, cross-cultural r is 0.60, 0.55 and 0.41 for these, level with or above the regional baseline (0.57, 0.54, 0.26). For the other 12 behaviors the regional average is higher, for example crying 0.54 against 0.16 and talking 0.50 against 0.19 (our computation).