EPFL's English-to-multilingual quality filter beats its best baseline by 0.08 points at 8B, and its cultural check turns on 151 questions

The adapted filter ties FineWeb2-HQ, and no downstream number exists below the top 20 languages

The paper (arXiv:2610.11585v1) is honest in its own words: it claims the method "maintains" baseline performance, not that it beats it. Our recount of Table 2 agrees with that wording and makes the size of "maintains" concrete. Every reported average reproduces exactly as the mean of its nine benchmark columns (17 of 17 rows, difference under 0.00005), so the arithmetic is clean. What the arithmetic shows is a tie: at 8B the mDCLM filter averages 0.5344 against 0.5336 for FWHQ+FW2HQ+, a gap of 0.08 points, while one binomial standard error on OpenBookQA alone, a 500-question test inside that average, is about 2.2 points.

The low-resource claim is the part a corpus team should read most carefully. The title promise is "over 100 languages". The downstream LLM benchmarks cover English plus the top 20 FineWeb 2 languages at 1B, and at 3B and 8B they report averages over "all available languages" with no per-language row. Languages ranked 21 to 100 get a proxy test on synthetic text in at most 5 languages, and languages beyond 100 get an LLM-judge correlation in 7 languages, shown only as scatter plots with no coefficient printed. Our judgement: the method is cheap and licence-clean, and its effect on a Kazakh or Assamese pretraining mix is not yet measured by anyone.

Two things in that chart deserve a plain statement. First, at 3B and 8B the runs are cooldowns from existing checkpoints (200B and 9.5T tokens), so the filter only shapes the last 50B or 63B tokens; a shrinking gain over unfiltered data is expected from that design and does not prove the filter matters less at scale. Second, at 1B the six classifier variants trained on different language sets span 0.65 points (Table 5), more than three times the 0.20 point edge over FWHQ+FW2HQ+. The paper reports one seed per configuration and no confidence intervals.

What EPFL published on 8 October 2026, and what it released

arXiv:2610.11585, Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection, by Vinko Sabolčec, Bettina Messmer, Yassine Turki and Martin Jaggi (EPFL), version 1, submitted 8 October 2026 09:33 UTC, categories cs.CL and cs.AI. It comes from the group behind FineWeb2-HQ and the Apertus models, and it uses Apertus architecture and checkpoints for every run.

The recipe has three steps (Section 3). Take a random 0.1% sample of the top 100 FineWeb 2 language-script pairs ranked by bytes, translate it into English with Qwen3-32B (reasoning off, a prompt that keeps typos and formatting), which produced about 5M documents for about 750 H100 GPU hours. Score the English translations with two existing English filters: the FineWeb-edu classifier (a 0 to 5 educational score) and the DCLM fastText classifier (a probability). Then train an MLP with two hidden layers of 3,072 units on frozen, mean-pooled embeddings of the original, untranslated text, with L1 loss on the FineWeb-edu score or on the logit of the DCLM probability. The training set is about 6M samples, about 1M of them English. At filtering time no translation is needed: the head scores native text directly, and the top 10% per language is kept.

The release is one Hugging Face model repo, epfml/FineWeb2-HQ-PlusPlus-Classifier, created 7 October 2026, last modified 9 October 2026 07:39 UTC, Apache-2.0, ungated, 0 downloads when we checked on 9 October 2026. It holds two files, mfwedu.pt (47,226,137 bytes) and mdclm.pt (47,226,125 bytes). We found no code repository under the epfml GitHub organisation and no Hugging Face dataset holding the 5M translations or the 6M training labels; the epfml dataset list on 9 October 2026 shows FineWeb2-HQ, FineWeb2-HQ-Plus, FineWeb2-HQ-QA, FineWeb-HQ and FineWeb2-embedded only.

Which of the paper's numbers hold up when recounted?

Paper values from arXiv:2610.11585v1. Subtractions, counts, inferred denominators and file-size arithmetic are BLOMEGA computations, 9 October 2026.

The CulturalBench row is the one we would flag to anyone citing the "no cultural bias" result. Table 4 reports four-decimal accuracies; the smallest question counts that reproduce all four rows in each country column are 59, 53, 14 and 25. Read as question counts, the mFW-edu filter answers 34 of 59 Chinese items against 32 for FWHQ+FW2HQ+, 38 of 53 Japanese items against 42, 8 of 14 French items against 9, and 10 of 25 Brazilian items against 9. The France gap of 7.15 points is one question. The paper's conclusion ("does not introduce a bias") is consistent with this, but so is the opposite: with 14 items, one binomial standard error at 50% accuracy is 13.4 points.

The INCLUDE check in Table 3 has the same shape: monolingual 1B models in four languages, mFW-edu ahead on Japanese (0.3653 against 0.3453), French and Portuguese, behind on Chinese (0.3358 against 0.3578). The average edge is 0.71 points. INCLUDE question counts per language are not printed, so we did not infer them.

How an English score becomes a Kazakh label

The design choice that matters is the direction of translation. The authors translate non-English web text into English for labelling, then train on the original text, so the classifier sees the real distribution of each language's web rather than translationese. The cost is that every training label is only as good as Qwen3-32B's reading of that language, and the paper reports no translation quality measure. Qwen3 lists 119 supported languages and dialects; that list bounds which languages can receive labels at all, and anything past it relies on mmBERT's shared embedding space alone.

Figure 2 of the paper also shows that languages left out of the classifier's training get lower absolute scores, even when the classifier still separates synthetic encyclopedia text from web text. That shift is harmless in the paper's own setup because selection keeps the top 10% per language, so only the ranking inside each language matters. It is harmful if a team applies one global threshold (say, "keep score 3 and above", the usual FineWeb-edu habit) across languages: out-of-distribution languages would lose more documents than in-distribution ones for the same underlying quality.

The released head matches the paper's description. Two hidden layers of 3,072 on a 768-wide input give 11,805,697 parameters, which in fp32 is 47,222,788 bytes; the files are 47,226,137 and 47,226,125 bytes, the difference being PyTorch serialisation overhead. The model card quickstart says to apply softmax to the DCLM output while its code applies a sigmoid; the sigmoid is the correct inverse of the logit target in Section 3.

What this means for teams building low-resource corpora

• Use it as a cheap first filter with a clean licence chain, then measure your own languages. FineWeb-edu classifier (Apache-2.0), DCLM fastText (MIT), Qwen3-32B (Apache-2.0), mmBERT-base (MIT) and the released heads (Apache-2.0) contain nothing that blocks commercial corpus work. What is missing is evidence that the filter improves a model in your target language. Our judgement: budget a small per-language check before trusting it.
• Rank within each language, never one global cutoff. The paper's own Figure 2 shows lower absolute scores outside the training languages. Keep the top fraction per language, as the authors do (top 10%).
• Check that your language is on the card, and what that means. The card lists 101 entries but 99 distinct codes; Serbian and Northern Uzbek appear twice, presumably once per script. The list also contains gmh (Middle High German) and nrm (Norman), which tells you "top 100" means top 100 by FineWeb 2's language-ID byte counts, not by speaker population. A language outside the list gets no translated labels at all.
• The translation step is your weakest link, and you can test it. No MT quality figure is reported for the 5M translations. For a target language, a few hundred native-speaker judgements of Qwen3-32B translations tell you whether the English classifier was ever scoring the right content. That is human data work, and it is cheap compared with the 750 H100 hours the translation took.
• Plan for the encoder, not the head. The 47 MB head is trivial; mmBERT-base at 1.23 GB has to embed every document, so filtering cost scales with an encoder pass over the corpus. The quickstart assumes CUDA.
• Treat the bias result as untested at useful power. 151 CulturalBench questions across four countries cannot detect a moderate cultural skew. If your use case is cultural or regional content, run your own regional benchmark on a filtered versus unfiltered sample.

For a related check on how cross-lingual automatic judgements can drift from human ones, see our note on multilingual LLM judges and translationese.

Check it yourself

Paper text and the release state:

curl -sL https://arxiv.org/pdf/2610.11585v1 -o p.pdf
python3 -c "import pypdf;print('\n'.join(p.extract_text() for p in pypdf.PdfReader('p.pdf').pages))" > p.txt
curl -s "https://huggingface.co/api/models/epfml/FineWeb2-HQ-PlusPlus-Classifier?blobs=true" \
  | python3 -c "import json,sys;d=json.load(sys.stdin);print(d['createdAt'],d['lastModified'],d['downloads'],d['cardData']['license']);l=d['cardData']['language'];print(len(l),len(set(l)));[print(s['rfilename'],s.get('size')) for s in d['siblings']]"
# 9 Oct 2026: 2026-10-07T07:20:16 2026-10-09T07:39:42 0 apache-2.0 / 101 99 / mdclm.pt 47226125, mfwedu.pt 47226137
curl -s "https://huggingface.co/api/models/jhu-clsp/mmBERT-base?blobs=true" | grep -o '"pytorch_model.bin","size":[0-9]*'
curl -s "https://huggingface.co/api/datasets?author=epfml&limit=100" | grep -o '"id":"[^"]*"'

The table recount: copy the nine benchmark columns of Tables 2, 4 and 5 into a script (ours is headline.py, about 80 lines) and run it. The core of the CulturalBench inference is:

python3 - <<'PY'
T4={"China":[.4915,.5424,.5763,.5254],"Japan":[.6792,.7925,.7170,.7358],
    "France":[.6429,.6429,.5714,.5714],"Brazil":[.4000,.3600,.4000,.3600]}
for c,v in T4.items():
    n=next(n for n in range(5,200) if all(abs(round(x*n)/n-x)<5e-5 for x in v))
    print(c,n,[round(x*n) for x in v])   # rows: pre-cooldown, FWHQ+FW2HQ+, mFW-edu, mFW-eduEng
PY
# China 59 [29, 32, 34, 31]; Japan 53 [36, 42, 38, 39]; France 14 [9, 9, 8, 8]; Brazil 25 [10, 9, 10, 9]

Expected from headline.py: all 17 recomputed averages equal the reported ones; best-adapted minus FWHQ+FW2HQ+ of +0.20, +0.22, +0.08; minus FW+FW2 of +2.89, +1.28, +0.59; Table 5 spread 0.65. The Appendix D title count (380 listed, 367 distinct) comes from titles.py p.txt. Table 2 is on page 7 of the PDF, Table 4 on page 8, Table 5 on page 21.

We did not score sample texts with the released classifier. Doing so needs the 1.23 GB mmBERT-base encoder plus PyTorch and Transformers, which exceeded our 1 GB per-download limit on the shared machine. A reader with the encoder can run the card's quickstart on a passage and its translation (switch .cuda() to CPU) and compare the two scores directly; that is the cheapest cross-lingual consistency test available.

What would prove this wrong

Our claim is that the adapted classifier performs the same as FineWeb2-HQ's multilingual classifier within noise, and that its benefit for languages outside the top 20 is unmeasured.

It would be wrong if, by 30 June 2027, a revision of arXiv:2610.11585 or an independent replication reports per-language downstream results for at least five languages ranked 21 to 100 in FineWeb 2, with the adapted filter ahead of unfiltered FineWeb 2 by 1 point or more of average accuracy at 3B or larger, or ahead of FWHQ+FW2HQ+ by more than 1 point with seeds or confidence intervals reported. We predict neither will appear by that date, and that any per-language results that do appear will show gains under 1 point against FWHQ+FW2HQ+.

Sources

1. Vinko Sabolčec, Bettina Messmer, Yassine Turki, Martin Jaggi (EPFL). Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection. arXiv:2610.11585v1, 8 October 2026. Sections 3 to 5, Tables 1 to 5, Figures 1 to 3, Appendices A to F.
2. epfml/FineWeb2-HQ-PlusPlus-Classifier on Hugging Face, created 7 October 2026, checked via the HF API on 9 October 2026.
3. jhu-clsp/mmBERT-base on Hugging Face (Marone et al., 2025), MIT, checked 9 October 2026.
4. HuggingFaceFW/fineweb-edu-classifier (Apache-2.0) and mlfoundations/fasttext-oh-eli5 (MIT), the two English labellers, checked 9 October 2026.
5. Qwen/Qwen3-32B, the translation model, Apache-2.0, checked 9 October 2026.
6. epfml/FineWeb2-HQ and the epfml dataset list on Hugging Face (Messmer et al., 2026), checked 9 October 2026.
7. BLOMEGA computations, 9 October 2026: averages and differences from Tables 2 and 5, benchmark win counts, CulturalBench denominators from Table 4, Appendix D title count, MLP parameter arithmetic against HF file sizes.

FAQ

Does an English quality classifier transfer to low-resource languages for pretraining data filtering?

Partly shown. In arXiv:2610.11585 the only downstream LLM benchmarks cover English plus the top 20 FineWeb 2 languages (1B runs) or an average over available benchmark languages (3B, 8B) with no per-language results. For lower-resource languages the evidence is proxy only: classifier scores on synthetic encyclopedia text versus web text in 8 languages, and score correlation with three LLM judges in 7 languages beyond the top 100, where 2 of 21 judge-language pairs showed no positive correlation.

How much better is the adapted mFW-edu or mDCLM classifier than FineWeb2-HQ?

By the paper's Table 2, the better of the two adapted classifiers beats the strongest baseline (FWHQ+FW2HQ+) by 0.20 points of average accuracy at 1B, 0.22 at 3B and 0.08 at 8B. The paper calls this comparable performance, which is the fair reading. One binomial standard error on OpenBookQA alone (500 questions) is about 2.2 points.

Can I run the released classifier on CPU?

The two MLP heads on Hugging Face (epfml/FineWeb2-HQ-PlusPlus-Classifier, Apache-2.0) are 47.2 MB each, but they need mean-pooled embeddings from jhu-clsp/mmBERT-base, whose weights file is 1.23 GB. The model card's quickstart calls .cuda(), so a CPU run means editing it. We did not score texts ourselves because the encoder exceeded our 1 GB download cap on 9 October 2026.