The Polish-Silesian system "outperforming" Google wins 4 of 8 score cells, and copying the Polish input beats half the field on FLORES

The Polish-Silesian rule-based system beats every outside system in 4 of 8 score cells, and copying the Polish input beats 4 of 8 outside systems on FLORES

NASK's paper, posted to arXiv on 1 October 2026, is titled as a system "outperforming open-source and commercial models". Its own tables support that for Polish to Silesian on SiLTT and BOUQuET only. The rule-based system loses to Google Translate on FLORES (14.9 against 17.7 BLEU, 48.3 against 48.6 chrF) and to GPT-5.4 and Google on Silesian to Polish. That makes 4 wins in 8 BLEU and chrF cells by our count. In our own run on 5 October 2026, passing the Polish FLORES-200 devtest sentences through unchanged as "Silesian" scored 6.1 BLEU and 36.7 chrF, above four of the eight external systems the paper compares against. The winning rule-based system has no release linked from the paper; the test set and the fine-tuned model are gated behind manual approval.

The figure shows chrF per condition. The wins are large where they happen: on BOUQuET Polish to Silesian the rule-based system leads Google Translate by 14.9 BLEU and 11.6 chrF. On SiLTT the lead over GPT-5.4 is 2.9 BLEU and 2.8 chrF. The losses are smaller on FLORES (2.8 BLEU, 0.3 chrF behind Google) and larger in the reverse direction (9.4 BLEU and 5.8 chrF behind GPT-5.4). Our judgement: the abstract states the result accurately ("consistently strongest on SiLTT and BOUQuET"); the title does not carry that qualifier.

What NASK published on 1 October 2026

arXiv:2610.01082v1, Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models, by Grzegorz Kulik, Mikołaj Pokrywka and Wojciech Kusa (NASK National Research Institute, Poland) and Adam Jatowt (University of Innsbruck), submitted 1 October 2026 and marked "Accepted at EMNLP 2026 Findings". It is a 10-page paper with one version as of 5 October 2026.

Three contributions. First, SiLTT (Silesian Literary Translation Testset): 1,237 Polish-Silesian sentence pairs from literary and cultural texts including "Cinderella", "Rapunzel", "Janko the Musician" and "A Christmas Carol", aligned with LaBSE and kept only above cosine 0.6 (Section 5.2). Second, a rule-based Apertium system with 31,757 lexical translation pairs and 2,349 grammatical rules drawn from dialect dictionaries and theses going back to before 1904 (Section 4.1). Third, TranslateGemma-4B fine-tuned with QLoRA on CBT-22k, 22,783 Silesian sentences from Wachtyrz.eu, Slazag.pl and older dialect texts, translated into Polish by the rule-based system and then post-edited by a professional translator (Section 5.1).

The comparison set: PLLuM-12B-nc-chat, Bielik-PL-11B-v3.0-IT, MADLAD-400-10B-MT, NLLB-200 3.3B, NLLB 54B MoE, base TranslateGemma-4B, GPT-5.4 (temperature 1.0, reasoning off, Polish prompt) and the Google Cloud Translation API (queried April and May 2026). Metrics are sacreBLEU BLEU and chrF only; the authors exclude COMET and LLM judges as unreliable for this pair (Section 6.2). Silesian context: 467,145 residents of Poland declared it a home language in the 2021 census, and its orthography was standardised only in 2009.

The paper's numbers, our recomputation and the release state

Paper values from arXiv:2610.01082v1. Margins, copy-source scores and release checks are BLOMEGA computations, 5 October 2026, scripts listed in the check section.

We could not score the copy-source baseline on SiLTT or BOUQuET. SiLTT requires manual approval from NASK-PIB, and BOUQuET (facebook/bouquet) is gated with automatic approval behind a Hugging Face login; we had neither credential on 5 October 2026. FLORES-200 is public as a tarball, and the paper's FLORES set has 1,012 pairs, the size of the devtest split. Our sacreBLEU signatures (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0 for BLEU, nc:6|nw:0|space:no for chrF) match the paper's footnotes 5 and 6 character for character.

On FLORES, copy-source at 36.7 chrF sits above Bielik-PL-11B (35.1), NLLB-200 3.3B (34.8), base TranslateGemma-4B (32.1) and PLLuM-12B (12.1), and 0.5 to 0.8 below MADLAD-400 (37.2) and NLLB 54B MoE (37.5). On BLEU, copy-source at 6.1 beats PLLuM (0.1), base TranslateGemma (5.1) and Bielik (5.7). The rule-based system adds 8.8 BLEU and 11.6 chrF over doing nothing; Google adds 11.6 and 11.9. None of the 1,012 FLORES Polish and Silesian lines are identical, so the 36.7 comes from shared characters, not from untouched sentences.

Two internal points a reader can check. The HF dataset ships a SiLTT_filtered split of 1,219 rows next to the 1,237-row split, a difference of 18 pairs that the paper does not mention; Section 5.2 says pairs under the 0.6 threshold "were excluded from the final dataset", so which split produced Table 2 is not stated. And Table 5 gives NLLB-200 3.3B 18.0 BLEU but 32.5 chrF on Silesian to Polish, lower chrF than our copy-source run in the same direction on FLORES (36.2), with the paper noting NLLB "failed to translate Silesian-specific words, leaving them unchanged". Different test sets, so this is a flag for a re-run, not a contradiction.

Why a dictionary system wins on SiLTT and BOUQuET and not on FLORES

The paper's own explanation for Google's FLORES lead is possible training exposure, since FLORES "has been publicly available for a long time" (Section 6.3.1). It offers no test of that. A second mechanism sits in the method section, and our reading is that it deserves equal weight. The rule-based system's dictionary encodes one orthography and one lexical register, drawn from literary and dialect sources. SiLTT is built from literary translations. CBT-22k, the fine-tuning data, has its Polish side produced by the same rule-based system before post-editing. So the two "ours" systems share a lineage with each other, and the system with the highest score shares a genre with the test set it wins on most clearly.

For a closely related pair, chrF also rewards spelling convention heavily. Silesian written with Polish orthography (what the paper says most speakers do, Appendix A) and Silesian written in the 2009 ślabikŏrz convention differ in characters even where the word is the same. A system that outputs the convention a test set uses gets credit across many character n-grams at once. Our copy-source run shows how much shared material there is to begin with: 36.7 chrF on FLORES before any translation happens. The paper's qualitative table (Table 4) supports this reading from the other side: Google's errors there are retained Polish words and person agreement broken by pivoting through English, the kinds of errors that cost the most character overlap against a consistently spelled reference.

None of this makes the result wrong. A 14.9 BLEU lead on BOUQuET, a community-translated multicentric benchmark that NASK did not build, is large. What it changes is the reading: "precision over scale" is shown for a standardised, literary-adjacent register of Silesian, and the paper's own Limitations section says spoken and informal Silesian is underrepresented.

What this means for teams localizing into close dialects and regional languages

• Score a copy-source baseline before any model comparison. For Polish to Silesian it is 36.7 chrF on FLORES-200 devtest. Any system near that number is adding little; four of the paper's eight external systems are at or below it. The same check applies to Catalan and Valencian, Bavarian, Limburgish or Ladin.
• Report per direction and per benchmark, not a single verdict. The rule-based system's lead flips sign between SiLTT/BOUQuET and FLORES, and between Polish to Silesian and Silesian to Polish. A vendor claim of "beats Google" for a dialect pair should come with the cell table.
• Fix the orthography in the brief. Since chrF credit depends on spelling convention, a localization spec for Silesian (or any variety with competing orthographies) should name the convention, and test references must be in it. Otherwise the metric measures which convention a system learned.
• About 22,800 post-edited pairs moved a 4B model past GPT-5.4 on two sets. TranslateGemma-FT on 22,783 curated pairs reached 26.3 BLEU on BOUQuET against 7.4 for the base model, while adding 78,107 noisier NLLB-pivot and HPLT back-translated pairs dropped it to 6.2. Our judgement: this is the clearest data-quality result in the paper, and the one most relevant to anyone buying or commissioning dialect data.
• Plan for licences and gates. SiLTT is CC BY-NC 4.0 and gated; the adapter carries the Gemma licence and needs the Google base model. Neither is usable for a commercial evaluation pipeline without clearance.

For a related case where a trivial baseline reframed a dialect result, see our note on dialectal Arabic error spans and a trivial baseline, and on character-copy baselines for Chinese varieties in the Teochew Hanzi copy baseline.

Check it yourself

Copy-source baseline on FLORES-200 (public tarball, 25,585,843 bytes, sha256 b8b0b767...f6 on 5 October 2026):

pip install sacrebleu==2.6.0
curl -sL https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz -o flores200.tgz
tar xzf flores200.tgz flores200_dataset/devtest/szl_Latn.devtest flores200_dataset/devtest/pol_Latn.devtest
sacrebleu flores200_dataset/devtest/szl_Latn.devtest -i flores200_dataset/devtest/pol_Latn.devtest -m bleu chrf -b
# expected: 6.1 (BLEU), 36.7 (chrF)

Release state (Hugging Face and GitHub APIs):

curl -s https://huggingface.co/api/datasets/NASK-PIB/SiLTT | python3 -m json.tool | grep -E '"(gated|license|num_examples|createdAt)"'
curl -s https://huggingface.co/api/models/NASK-PIB/translategemma-4b-it-pol-szl-qlora | grep -o '"gated":"[a-z]*"'
curl -s -o /dev/null -w "%{http_code}\n" https://huggingface.co/datasets/NASK-PIB/SiLTT/resolve/main/data/SiLTT-00000-of-00001.parquet   # 401
git clone https://github.com/apertium/apertium-pol-szl.git && cd apertium-pol-szl
grep -c '<e[ >]' apertium-pol-szl.pol-szl.dix      # 22358
grep -c '<rule[ >]' apertium-pol-szl.pol-szl.t1x   # 33 (t2x and t3x: 1 each)

The win count comes from Tables 2, 3 and 5 (pages 4 and 10 of the PDF): for each of the four conditions take the rule-based BLEU and chrF and subtract the highest value among the non-NASK systems. Four of the eight differences are positive (SiLTT and BOUQuET Polish to Silesian), four negative (FLORES and Silesian to Polish). If you hold SiLTT access, the next step is sacrebleu ref.szl -i src.pol -m bleu chrf on both splits; we did not have it.

What would prove this wrong

Our claim is that the rule-based lead is specific to standardised, literary-register Silesian and does not hold on FLORES or in the Silesian to Polish direction, and that the system cannot yet be independently re-run.

It would be wrong if, by 30 June 2027, NASK or anyone else publishes the 31,757-pair, 2,349-rule system (or its outputs) and a third-party evaluation on a Silesian test set not built from literary sources, such as spoken transcripts or social media, shows it ahead of Google Translate and GPT-5.4 on both BLEU and chrF. We predict that by that date the public apertium-pol-szl repository will still be behind the paper's dictionary size, and that the SiLTT copy-source chrF, once someone with access runs it, will land within 10 points of the weakest neural systems in Table 2 (21.3 to 26.8 chrF).

Sources

1. Grzegorz Kulik, Mikołaj Pokrywka, Adam Jatowt, Wojciech Kusa. Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models. arXiv:2610.01082v1, 1 October 2026. Sections 4 to 7, Tables 1 to 5, Appendices A to D.
2. NASK-PIB/SiLTT on Hugging Face, created 26 August 2026, last modified 1 October 2026, gated manual, CC BY-NC 4.0; checked 5 October 2026.
3. NASK-PIB/translategemma-4b-it-pol-szl-qlora, QLoRA adapter, gated manual, licence gemma; checked 5 October 2026.
4. apertium/apertium-pol-szl, GPL-3.0, created 28 May 2018 by Grzegorz Kulik, last commit 20 July 2021, 12 commits; checked 5 October 2026.
5. NLLB Team. FLORES-200 dataset tarball, szl_Latn and pol_Latn dev and devtest; downloaded 5 October 2026.
6. facebook/bouquet on Hugging Face, CC BY 4.0, gated auto, Polish-Silesian sentence-level files listed; checked 5 October 2026.
7. Matt Post. A Call for Clarity in Reporting BLEU Scores. arXiv:1804.08771 (sacreBLEU).
8. BLOMEGA computations, 5 October 2026: copy-source BLEU and chrF on FLORES-200, rule-based and released-model margins from Tables 2, 3 and 5, release checks.

FAQ

Does the NASK Polish-Silesian system really beat Google Translate and GPT-5.4?

In 4 of 8 BLEU and chrF cells. The rule-based system leads on SiLTT (10.3 / 34.5 against GPT-5.4 at 7.4 / 31.7) and on BOUQuET Polish to Silesian (36.7 / 61.0 against Google at 21.8 / 49.4). It trails Google on FLORES (14.9 / 48.3 against 17.7 / 48.6) and GPT-5.4 on BOUQuET Silesian to Polish (31.7 / 57.4 against 41.1 / 63.2). Figures from Tables 2, 3 and 5 of arXiv:2610.01082; margins are our arithmetic.

Can I download SiLTT and the NASK Silesian model?

Only after approval. On 5 October 2026 both NASK-PIB/SiLTT (CC BY-NC 4.0, splits of 1,237 and 1,219 rows) and the TranslateGemma QLoRA adapter (Gemma licence) were gated with manual approval on Hugging Face, and anonymous file downloads returned HTTP 401. The rule-based system that tops the tables has no linked release; the public apertium-pol-szl pair was last changed on 20 July 2021 and is smaller than the paper's system.

How strong is a do-nothing baseline for Polish to Silesian?

On FLORES-200 devtest, copying the Polish sentences unchanged scores 6.1 BLEU and 36.7 chrF with the paper's sacreBLEU 2.6.0 settings (our run, 5 October 2026). That is above Bielik-PL-11B (35.1 chrF), NLLB-200 3.3B (34.8), base TranslateGemma-4B (32.1) and PLLuM-12B (12.1) in the paper's Table 3. We could not run it on SiLTT or BOUQuET because both are gated.