A 500M-parameter monolingual model beats 70B LLMs on 14 of 101 languages
MultiBLiMP 1.0, published in TACL 2026, scored 42 LLMs on 128,321 grammatical minimal pairs across 101 languages. Llama3-70B hits 99.0% accuracy on German and 11.1% on Tupinambá, where its 8-billion-parameter sibling scores 0.0%. Model size explains most of the pattern: a regression across all 42 models puts the size effect at β=4.30 with R²=0.936. But a monolingual model with roughly a hundredth of the parameters, trained on up to 1GB of text in a single language, beats both Llama3-70B and Gemma3-27B outright on 14 of the 101 languages. Scale is not the only variable, and on the languages that matter most it is not even the dominant one.
What the benchmark is, and what changed
MultiBLiMP 1.0 is a massively multilingual benchmark of linguistic minimal pairs, built by Jumelet, Weissweiler, Nivre and Bisazza and published in Transactions of the Association for Computational Linguistics, volume 14, pages 193 to 216 (DOI 10.1162/tacl.a.600), with an earlier version posted to arXiv on 2 April 2025 and revised 18 July 2025.
A minimal pair is two sentences that differ in exactly one place: a correctly agreeing verb or adjective, and an incorrectly agreeing one. The model is scored on whether it assigns higher probability to the grammatical sentence. The pairs are generated automatically, not written by hand, drawing on Universal Dependencies 2.15 (168 languages, 296 treebanks) and UniMorph v4.0 (183 languages). Before balancing, the pipeline produced roughly 1.4 million candidate pairs; after capping at 100 items per agreement condition per language, the released set is 128,321 pairs across 101 languages, covering two types of subject agreement.
The paper evaluated 42 models: base and chat variants of Llama3 from 1B to 70B parameters (plus Tulu3), Aya-Expanse at 8B and 32B, Gemma3 from 4B to 27B, Qwen3 from 0.6B to 14B, OLMo2 from 1B to 32B, and EuroLLM at 1.7B and 9B, alongside Goldfish, a family of small monolingual models covering 70 of the 101 languages. We flag one internal inconsistency rather than silently resolve it: the paper's own Section 8 refers to "17 LMs" in one discussion while the model list and Table 3 describe 42, and Goldfish is described as 500-million-parameter models in the dataset section but listed as 125M in at least one table render we checked. Treat both figures as reported, not reconciled, until you have the camera-ready PDF in front of you.
What the scores actually show
Table 1 pulls the overall accuracy and the low/mid/high-resource breakdown for the largest model in each family, plus Goldfish, from the paper's Table 3. "Low/mid/high" are the paper's own resource tiers, based on how much text in that language existed in the models' pretraining corpora.
| Model | Params | Overall | Low-resource | Mid-resource | High-resource |
|---|---|---|---|---|---|
| Llama3-70B, base | 70B | 90.2 | 81.1 | 93.9 | 97.8 |
| Gemma3-27B, base | 27B | 90.2 | 78.3 | 96.3 | 98.0 |
| Llama3-70B, chat | 70B | 89.3 | 80.3 | 93.1 | 96.9 |
| Aya-Expanse, chat | 32B | 86.4 | 75.7 | 89.4 | 97.7 |
| Llama3-8B, base | 8B | 86.9 | 77.2 | 90.3 | 96.2 |
| EuroLLM-9B, base | 9B | 85.8 | 72.6 | 92.0 | 95.7 |
| Gemma3-27B, chat | 27B | 85.8 | 73.1 | 92.3 | 94.4 |
| Qwen3-14B, chat | 14B | 85.3 | 74.1 | 89.9 | 94.8 |
| OLMo2-32B, base | 32B | 82.7 | 74.4 | 84.6 | 92.5 |
| OLMo2-32B, chat | 32B | 81.5 | 72.5 | 83.6 | 91.9 |
| Goldfish, monolingual | ~500M (reported 125M in one table) | 93.8 | 88.0 | 95.6 | 95.9 |
Scale explains most of the aggregate pattern. A regression run across all 42 models found a significant positive effect of model size (β=4.30, t=15.88) and a significant negative effect of post-training, i.e. instruction-tuned chat variants scoring lower than their base counterparts (β=−3.29, t=−6.47), with an R² of 0.936 for the overall model. Llama3-70B significantly outperformed Llama3-8B on 48 of the 101 languages, with zero languages where the smaller model won significantly.
But scale is not the whole story on the hardest languages. Goldfish, a family of monolingual models at a small fraction of Llama3-70B's parameter count, each trained on up to 1GB of text in one language, significantly outperformed both Llama3-70B and Gemma3-27B on 14 of the 101 languages, concentrated in the low-resource tier where Goldfish's 88.0% average beats every multilingual model in Table 1. On Erzya specifically, Goldfish's 73.7% beats Llama3-70B's 56.9% by almost 17 points, with a model that was never trained on anything but Erzya.
Why a smaller model wins on the languages that matter most
The mechanism is capacity allocation, not raw ability. A 70-billion-parameter multilingual model splits its training data and its representational capacity across every language it was trained on, and that split is proportional to how much text existed for each language in the first place, which is exactly what makes a language "low-resource." A 500-million-parameter model trained on nothing but Erzya spends its entire capacity on one language's grammar, with no competition from English, German or the other 99 languages crowding the gradient.
This also explains why the benchmark itself has a structural tilt. Universal Dependencies, the treebank resource MultiBLiMP draws from, has more treebanks and more annotated text for Indo-European languages than for any other family, and the two agreement phenomena MultiBLiMP tests are themselves more common in Indo-European grammar. The paper's authors note this directly: the test is not fully language-neutral, which means the low scores on Tupinambá and Karo are a floor estimate of the real gap, not a ceiling.
What this means if you are sourcing multilingual training data
Buying a bigger multilingual model does not substitute for sourcing more data in a specific target language. The regression shows size matters on average, but the Goldfish result shows that for any single low-resource language you care about, dedicated per-language data can outscore a 140x larger general model. If your product needs Erzya, Karo or Tupinambá specifically, a larger frontier model is not the lever to pull.
Instruction tuning has a cost on raw grammatical competence. Every chat variant in Table 1 scores lower than its base counterpart: Llama3-70B chat loses 0.9 points to base, Gemma3-27B chat loses 4.4, OLMo2-32B chat loses 1.2. If your localization pipeline depends on a chat-tuned model's grammatical judgment in a low-resource language, test the base model too; the paper's regression says post-training is a net negative on this specific capability, consistently.
Treat "low-resource" as a data gap, not a capability ceiling. The variance between Llama3-8B (0.0% on Tupinambá) and Llama3-70B (11.1%) shows that even an 8.75x parameter increase inside the same family barely moves the needle on the most starved languages. The thing missing is text in that language, not FLOPs.
Check it yourself
# 1. The paper and its data
# TACL 2026, vol 14, pp 193-216: https://aclanthology.org/2026.tacl-1.10
# arXiv preprint (v2, 18 Jul 2025): https://arxiv.org/abs/2504.02768
# 2. Reproduce the resource-tier gap from Table 1 above
python3 - <<'PY'
rows = { # model: (overall, low, mid, high)
"Llama3-70B base": (90.2, 81.1, 93.9, 97.8),
"Llama3-8B base": (86.9, 77.2, 90.3, 96.2),
"Goldfish mono": (93.8, 88.0, 95.6, 95.9),
}
for name, (overall, low, mid, high) in rows.items():
print(f"{name:18s} low-to-high gap: {high-low:.1f} pts overall: {overall}")
PY
# 3. Pull the underlying resources yourself
# Universal Dependencies 2.15: https://universaldependencies.org/
# UniMorph v4.0: https://unimorph.github.io/
# Goldfish models: https://huggingface.co/goldfish-models
# 4. Check the model-count discrepancy we flagged (42 vs "17 LMs" in Section 8,
# and Goldfish listed at both ~500M and 125M) against the camera-ready PDF
# rather than trusting this summary or any other secondary source.
What would prove this wrong
The claim under test is that, for the lowest-resource languages in MultiBLiMP 1.0, a small model trained only on that language can outperform a much larger general-purpose multilingual model, and that this gap will not close by simply scaling the general model further. This is wrong if, by 1 January 2028, a multilingual model release, tested on the same MultiBLiMP pairs or a direct successor benchmark, closes the gap to within 2 points of the matching Goldfish monolingual score on at least 10 of the 14 languages where Goldfish currently wins, without adding dedicated per-language training data beyond what a standard multilingual web-scrape pipeline would collect.
A second prediction, marked as judgment: BLOMEGA expects the next MultiBLiMP-style benchmark cycle to show the gap narrowing fastest for languages where a provider has run a dedicated, consented data-collection effort (voice, text or both) rather than for languages that simply got more web-scrape coverage by chance. If the opposite is true, that reading was wrong.
FAQ
How do you evaluate an AI model's quality across languages?
MultiBLiMP 1.0, published in TACL 2026, uses minimal pairs: a grammatical sentence and an ungrammatical one differing only in subject-verb or subject-adjective agreement, generated automatically from Universal Dependencies 2.15 and UniMorph v4.0. The model is scored on whether it assigns higher probability to the grammatical sentence, across 128,321 pairs in 101 languages. It avoids human annotators and avoids testing translation quality directly, but it only tests grammatical competence, not meaning or fluency.
Does a bigger LLM always do better on low-resource languages?
No. MultiBLiMP 1.0 found that scale explains most variance in a regression across 42 models (size coefficient 4.30, t=15.88, R-squared 0.936), and Llama3-70B beat Llama3-8B on 48 of 101 languages with no languages going the other way significantly. But monolingual Goldfish models, each around 500 million parameters trained on up to 1GB of text in a single language, significantly outperformed both Llama3-70B and Gemma3-27B on 14 of the 101 languages, almost entirely the lowest-resource ones. Scale wins on average; dedicated per-language data wins where the gap is worst.
Why did Llama3-70B score 0% on some languages?
On Tupinambá, an extinct-to-endangered Brazilian language, Llama3-8B scored 0.0% accuracy and Llama3-70B scored 11.1%, against 99.0% to 99.4% for both models on English and German. The paper attributes the broader pattern to language frequency in training data and to a structural bias in the benchmark itself: Universal Dependencies and the agreement phenomena it tests are both overrepresented in Indo-European languages, so even the test format favors languages the models have seen more of.
Sources
- Jumelet, Weissweiler, Nivre, Bisazza, MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs, Transactions of the Association for Computational Linguistics, vol. 14, pp. 193–216, DOI 10.1162/tacl.a.600, 2026.
- Same authors, arXiv:2504.02768, v1 posted 2 April 2025, v2 revised 18 July 2025. Model list, Table 3 scores, regression statistics, and the Section 8 "17 LMs" reference flagged above.
- Universal Dependencies 2.15 and UniMorph v4.0, the underlying linguistic resources the benchmark's minimal pairs are generated from.
Related BLOMEGA guides: Consented AI training data providers.