A 30-trillion-token corpus buys Basque a 160-million-parameter model
HPLT 3.0 is the largest openly published multilingual pretraining collection: 30 trillion sub-word tokens across close to 200 language-script combinations. Its Table 1 prints the per-language counts, and English holds 16T of them while Galician holds 3.1B and Basque 3.2B. Convert those counts through the Chinchilla ratio of 20 tokens per parameter and you get the largest model each language can compute-optimally support: 800B parameters for English, 160M for Basque. At the data intensity Meta actually used for Llama 3, 1,875 tokens per parameter, Basque buys 1.7M parameters. That ceiling, not vendor effort, is what sets quality in your smaller locales.
What got published, and when
Three releases since late 2024 made per-language accounting possible. Before them, "supports 100 languages" was an unfalsifiable claim because nobody printed the denominator.
8 December 2024. Hugging Face released FineWeb2, over 1,000 language-script subsets built from 96 Common Crawl snapshots spanning summer 2013 to April 2024. The dataset card lists the configs by language-script pair but does not print per-language token counts, so you have to count them yourself.
13 March 2025. The HPLT consortium posted An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (arXiv:2503.10267, ACL 2025 main proceedings): 8T tokens of monolingual text covering 193 languages, plus 380M parallel sentence pairs covering 51 languages. Note the second number. Parallel data, the kind that trains translation directly, exists for roughly a quarter of the languages the monolingual side covers.
2 November 2025. Stephan Oepen and 31 co-authors posted HPLT 3.0 (arXiv:2511.01066, revised to v3 on 19 April 2026). 30T sub-word tokens, close to 200 language-script combinations, 57 encoder-decoder models, and the table this article is built on.
4 December 2025. The v3 revision of EMMA-500 (arXiv:2409.17892) documented the MaLA corpus: 939 languages, 824M documents, 74.255 billion whitespace-delimited tokens, average document length 90.12 tokens. It also states the cut-off that matters most in this whole article. Of the 939 languages, 546 have more than 100k tokens, and those 546 are what EMMA-500 was trained on. The other 393 were collected and then left out.
How many tokens does each language actually have?
Table 1 of HPLT 3.0 prints document counts, token counts, average document length and token share for English, for the multilingual remainder, and for nine individual languages. The chart below is those nine plus English, on a log axis, because a linear axis renders eight of the ten bars as a single line.
The nine non-English rows in that chart sum to 11.86% of the non-English portion of the corpus, using the shares printed in the same table. Nine languages, close to 200 in the collection, and just under an eighth of everything that is not English.
Now the conversion. Hoffmann and colleagues (arXiv:2203.15556, 29 March 2022) established that for a fixed compute budget, model size and training tokens should scale together, with Chinchilla at 70B parameters trained on four times the data of Gopher at 280B and beating it. The ratio commonly read off that result is about 20 tokens per parameter. Meta's own Llama 3 announcement (18 April 2024) states the Chinchilla-optimal budget for an 8B model is around 200B tokens, which is 25 to 1, and then reports training both the 8B and 70B models on up to 15T tokens because performance kept improving log-linearly. For Llama 3 8B that is 1,875 tokens per parameter, 94 times the Chinchilla ratio.
So there are two ceilings, and both are computed the same way: divide the available tokens by the ratio. The Chinchilla column below is the generous reading. The Llama 3 column is what a lab building a competitive model in 2026 actually consumes.
| Language | Documents | Tokens | Ceiling at 20 tok/param | Ceiling at 1,875 tok/param | Source |
|---|---|---|---|---|---|
| English | 18B | 16T | 800B params | 8.53B params | arXiv:2511.01066 Table 1 |
| All non-English combined | 11B | 13T | 650B params | 6.93B params | arXiv:2511.01066 Table 1 |
| Spanish | 725M | 658B | 32.9B params | 351M params | arXiv:2511.01066 Table 1 |
| French | 603M | 584B | 29.2B params | 311M params | arXiv:2511.01066 Table 1 |
| Czech | 107M | 126B | 6.30B params | 67.2M params | arXiv:2511.01066 Table 1 |
| Ukrainian | 80M | 81B | 4.05B params | 43.2M params | arXiv:2511.01066 Table 1 |
| Finnish | 49M | 73B | 3.65B params | 38.9M params | arXiv:2511.01066 Table 1 |
| Norwegian | 37M | 52B | 2.60B params | 27.7M params | arXiv:2511.01066 Table 1 |
| Catalan | see note | 22B | 1.10B params | 11.7M params | arXiv:2511.01066 Table 1 |
| Basque | 3.2M | 3.2B | 160M params | 1.71M params | arXiv:2511.01066 Table 1 |
| Galician | 4.0M | 3.1B | 155M params | 1.65M params | arXiv:2511.01066 Table 1 |
| MaLA corpus, all 939 languages | 824M | 74.255B (whitespace) | 3.71B params | 39.6M params | arXiv:2409.17892 Table 1 |
| A MaLA tail language at the 100k cut-off | not reported | 100k (whitespace) | 5,000 params | 53 params | arXiv:2409.17892 |
Note on Catalan: the row we read prints 2.6M documents, 22B tokens and an average document length of 853. Those three do not multiply out, and 22B divided by 853 implies roughly 26M documents. We flag the inconsistency rather than pick a side, and we use only the token count, which is what the ceiling depends on.
Note on units: HPLT counts sub-word tokens, MaLA counts whitespace-delimited tokens. A whitespace token is worth more than one sub-word token, so the MaLA rows understate in HPLT's units, probably by a factor of two to three for the scripts involved. We did not convert. Treat the MaLA ceilings as a floor, and note that the direction of the error makes the tail look better than it is on the model-size axis while making tokenization worse on the cost axis.
Two numbers out of that table are worth saying on their own. The entire MaLA corpus, 939 languages, everything anyone has managed to assemble across the long tail, is 74.255B tokens. Ukrainian alone in HPLT 3.0 is 81B. Every low-resource language on earth, pooled, is smaller than Ukrainian. And English at 16T is roughly 215 times that pooled corpus.
Why the ceiling is a ceiling and not a budget line
The reason this is not fixable with money inside the current pipeline is that every step downstream of the corpus is multiplicative on what the corpus contains. The diagram traces one locale through it.
The corpus builders drew the line themselves, and where they drew it is the most useful operational number in either paper. HPLT 3.0 states its team did not train models on languages with fewer than roughly 0.25M documents. MaLA collected 939 languages and trained on 546. Both groups are funded, motivated and public-good oriented, and both stopped at almost exactly the same place. If the people assembling the data will not train on a language, a commercial lab optimising for benchmark scores certainly will not.
English dominance is not uniform across corpora, which matters when a vendor tells you which corpus they used. Table 1 of HPLT 3.0 prints the English token share for four collections side by side.
| Corpus | Languages claimed | Total tokens | Per-language counts published? | Parallel data | Source |
|---|---|---|---|---|---|
| HPLT 3.0 | close to 200 language-script combinations | 30T sub-word | Yes, for English plus nine named languages in Table 1 | mined and MT-synthesised, count not stated in the abstract | arXiv:2511.01066 (2 Nov 2025) |
| HPLT v2 | 193 monolingual | 8T | not in the abstract | 380M sentence pairs, 51 languages | arXiv:2503.10267 (13 Mar 2025) |
| FineWeb2 | over 1,000 language-script subsets | n>1T, roughly 3T words across ~20TB | No, the card lists configs without token counts | none | HuggingFaceFW/fineweb-2 (8 Dec 2024) |
| MADLAD-400 1.0 | roughly 400 | 4.4T (1.7T English, 2.7T other) | not reported here | none | as printed in arXiv:2511.01066 Table 1 |
| MaLA corpus | 939 collected, 546 used for training | 74.255B whitespace | Yes, by threshold: 546 above 100k, ~300 above 1M | none | arXiv:2409.17892 v3 (4 Dec 2025) |
Read the fourth column first. Two of the five let you check a specific locale, and one of those two only by threshold. "Over 1,000 languages" and "939 languages" are both accurate and both compatible with a locale holding less text than a paperback.
What to do with this when picking locales
Put a token column next to the market-size column. Locale selection decks rank by addressable revenue and support cost. Add available in-language tokens from a published corpus, because that column sets a ceiling the other two cannot buy past. Spanish at 658B and Galician at 3.1B are both Iberian, both official, and 212 times apart on the only axis that determines model quality.
Treat 20B tokens as the practical line for an in-language model. Below 20B you cannot compute-optimally train even a 1B-parameter model in that language alone. Catalan at 22B is just over it. Basque and Galician are 6.9 times under it. That does not mean you cannot ship those locales, it means what you ship there will be transfer from a larger language, and you should budget review accordingly rather than assume parity.
Ask vendors which corpus, not how many languages. "1,000+ languages" and "193 languages" describe FineWeb2 and HPLT v2 respectively, and both are true statements about datasets whose tails contain almost nothing. The question that separates vendors is the per-language token count for your specific locales, and whether that count includes machine-translated text. HPLT v2's parallel side covers 51 languages against 193 monolingual. Ask which side your locale is on.
Assume synthetic data is already in the tail. Neither of these corpora claims to filter machine-translated web pages out of low-resource subsets, and the cheapest way to produce Galician web text at volume has been machine translation for years. That is the compounding problem we described in the LLM judge note: the evaluator that would catch it is measurably worst at exactly that comparison.
Where more data has to come from, it is commissioned. This is our position rather than a finding: below the corpus line, the only remaining supply is people who speak the language being paid to produce text and speech in it, with consent and chain of title recorded. Crawling has already returned what it can return. The numbers in Table 1 are what a decade of crawling produced.
Check it yourself
You can rebuild the ceiling column for any locale in about twenty minutes. The corpora are public and the arithmetic is one division.
# 1. per-language row counts and byte sizes straight from the HPLT 3.0 release
# (browse the dataset tree, each language-script pair is its own directory)
curl -s "https://huggingface.co/api/datasets/HPLT/HPLT3.0" | python3 -m json.tool | head -40
# 2. count tokens yourself for a locale in FineWeb2
pip install datasets transformers
python3 - <<'PY'
from datasets import load_dataset
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
for cfg in ["glg_Latn", "eus_Latn", "cat_Latn", "spa_Latn"]:
d = load_dataset("HuggingFaceFW/fineweb-2", cfg, split="train", streaming=True)
n = sum(len(tok(r["text"])["input_ids"]) for _, r in zip(range(2000), d))
print(cfg, "tokens in first 2000 docs:", n)
PY
# 3. the ceiling. T = tokens available in the language.
# chinchilla_ceiling = T / 20
# llama3_ceiling = T / 1875
python3 -c "T=3.1e9; print('20:1 ->', T/20/1e6, 'M params; 1875:1 ->', T/1875/1e6, 'M params')"
# 20:1 -> 155.0 M params; 1875:1 -> 1.653 M params
Three things are worth writing down per locale: the raw token count, the fraction of documents whose URL is a known machine-translation-heavy domain, and the ratio between sub-word tokens and whitespace words, which tells you how much of your inference budget the tokenizer eats in that script. The primary tables are Table 1 of HPLT 3.0, Table 1 of the EMMA-500 paper for the MaLA corpus, and the FineWeb2 dataset card for the config list.
What would prove this wrong
The claim under test is that available in-language token volume, not vendor capability, sets the quality ceiling for a locale, and that the ceiling for languages like Basque and Galician sits three orders of magnitude below English. It is wrong if, by 1 September 2027, a documented public corpus release gives at least 20 languages that hold under 10B tokens in HPLT 3.0 more than 100B tokens each of text that is neither machine-translated nor model-generated, with the provenance breakdown published per language. Today the two smallest languages in Table 1 sit at 3.1B and 3.2B, and the largest single-language jump between HPLT 2.0 and HPLT 3.0 was under one order of magnitude.
A second prediction, marked as judgement rather than finding: the next round of low-resource corpora will grow mostly through synthetic and machine-translated text, and the papers will report total token counts without a provenance split. If a 2027 release for a language currently under 10B tokens reports a 30-times increase and does not publish what fraction is model-generated, the increase is not the kind of data this article is about.
FAQ
Where do you get training data for low-resource languages?
From web corpora first, and then you run out. The MaLA corpus is the widest public attempt: 939 languages, 824M documents, 74.255B whitespace-delimited tokens. Only 546 of those languages hold more than 100k tokens and only about 300 hold more than 1 million, so 393 sit at or under 100k each. The English subset of HPLT 3.0 alone is roughly 215 times the entire 939-language corpus. Past that line the supply is commissioned in-language collection, licensed archives and consented recording.
How much text does a language need to train a model?
About 20 tokens per parameter at the compute-optimal point (Hoffmann et al., arXiv:2203.15556, 29 March 2022), and far more in practice. Meta states the Chinchilla-optimal budget for an 8B model is around 200B tokens and trained Llama 3 8B on 15T instead, roughly 1,875 tokens per parameter. Applied to HPLT 3.0, Spanish at 658B tokens supports 32.9B parameters at the Chinchilla ratio and 351M at Llama 3's; Galician at 3.1B supports 155M and 1.65M.
Which languages should an AI company localize first?
Rank candidates by available in-language token volume next to market size. In HPLT 3.0 the spread among European languages alone runs from Spanish at 658B to Galician at 3.1B, a factor of 212. A locale under roughly 20B tokens cannot compute-optimally support even a 1B-parameter in-language model, so output there comes from cross-lingual transfer and needs a review budget that reflects it.
Is English still the majority of AI training data in 2026?
Yes in every large public corpus, though the share varies. Table 1 of HPLT 3.0 puts English at 55% of HPLT 3.0, 78% of FineWeb 1.4.0/2.1.0, 38% of MADLAD-400 1.0 and 35% of HPLT 2.0. Percentage is the misleading figure: MADLAD-400 is only 38% English because it holds 1.7T English tokens against FineWeb's 17T.
Does a bigger model fix a small corpus?
No, it inverts the problem. Scale is what consumes tokens. At Chinchilla's 20 to 1, Galician's 3.1B tokens support 155M parameters; at Llama 3's 1,875 to 1, they support 1.65M. The more data-intensive training practice becomes, the smaller the in-language model a fixed corpus can support, which is why the gap between English and the tail has widened rather than closed since 2022.
Sources
- S. Oepen et al. (32 authors), HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models, arXiv:2511.01066, 2 November 2025, revised 19 April 2026. Table 1 supplies every token and document count used here, the 55% English share, the comparison rows for FineWeb, HPLT 2.0 and MADLAD-400, and the statement that no models were trained on languages below roughly 0.25M documents.
- HPLT consortium, An Expanded Massive Multilingual Dataset for High-Performance Language Technologies, arXiv:2503.10267, 13 March 2025, ACL 2025 main proceedings. HPLT v2: 8T tokens across 193 languages monolingual, 380M sentence pairs across 51 languages parallel.
- S. Ji, Z. Li, J. Paavola, P. Lin, P. Chen, D. O'Brien, H. Luo, H. Schütze, J. Tiedemann, B. Haddow, EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models, arXiv:2409.17892, v3 4 December 2025. The MaLA corpus: 939 languages, 824M documents, 74,255M tokens, 90.12 average document length, 546 languages above 100k tokens and more than 300 above 1M.
- J. Hoffmann et al., Training Compute-Optimal Large Language Models, arXiv:2203.15556, 29 March 2022. Chinchilla at 70B parameters on four times Gopher's data, 67.5% MMLU; the result the 20 tokens per parameter ratio is read from.
- Meta AI, Introducing Meta Llama 3, 18 April 2024. States the Chinchilla-optimal budget for an 8B model is around 200B tokens and that Llama 3 8B and 70B were trained on up to 15T tokens with log-linear improvement continuing, which gives the 1,875 tokens per parameter figure used in Table 1.
- Hugging Face, FineWeb2 dataset card, released 8 December 2024. Over 1,000 language-script subsets from 96 Common Crawl snapshots covering summer 2013 to April 2024. Per-language token counts are not printed on the card.
Related BLOMEGA guides: Your multilingual LLM judge prefers the machine translation · Localization is the new default · Consented AI training data providers · How to license off-the-shelf AI training datasets.