Machine-translating Alpaca into 50 European languages raised ROUGE-L and cut instruction-following 29.8%

Lab note · 19 September 2026 · BLOMEGA

Fine-tuning four small models on Alpaca machine-translated into 50 European languages made them worse at following instructions: European-IFEval accuracy fell from 0.309 to 0.217, 29.8% relative, while the reference-similarity metrics everyone reports went up (Aya ROUGE-L 0.124 to 0.203). Rebuilding the same data so each example still tests the same task raised accuracy to 0.349, 12.9% above the untuned models. The finding comes from Telefónica and UPC's EuroAlpaca, published 4 September 2026.

Telefónica and UPC published a task-preserving localisation of Alpaca on 4 September 2026

4 September 2026. Aleix Sant and Jordi Luque (Scientific Research, Telefónica Innovación Digital) and Carlos Escolano (Universitat Politècnica de Catalunya) posted EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages, arXiv:2609.05043. It releases two resources: EuroAlpaca, the 51,760-example Alpaca Cleaned set localised into 50 European languages and regional varieties (21 high-, 15 medium- and 14 low-resource by MADLAD-400 token counts), and European-IFEval, a harmonised verifiable instruction-following benchmark for the same 50 languages.

The experiment is unusually clean: 4 models × 50 languages × 3 training conditions = 600 LoRA runs, each on 50,760 examples, with everything but the training data held fixed. That design is what lets the paper say the difference is the data.

Direct machine translation raised similarity scores and lowered instruction-following in every model

Table 1. The same four 3 to 4B models, fine-tuned per language with LoRA on 50,760 examples under each condition. ROUGE-L and F-BERT are reference similarity on the Aya Evaluation Suite (36 languages). European-IFEval is strict accuracy on verifiable instructions (50 languages). Direct MT (bold) is NLLB-3.3B over every field. Source: Sant, Luque and Escolano, arXiv:2609.05043, Table 4.
Model	Training data	Aya ROUGE-L	Aya F-BERT	European-IFEval acc.	Source
Tiny-Aya-Global	Off-the-shelf	0.103	0.824	0.421	Table 4
	Direct MT	0.228	0.872	0.295	Table 4
	Task-Preserved	0.232	0.874	0.446	Table 4
	EuroAlpaca	0.238	0.877	0.499	Table 4
Phi-4-Mini-Instruct	Off-the-shelf	0.091	0.827	0.258	Table 4
	Direct MT	0.198	0.857	0.192	Table 4
	Task-Preserved	0.200	0.859	0.262	Table 4
	EuroAlpaca	0.219	0.867	0.291	Table 4
Ministral-3-3B-Instruct	Off-the-shelf	0.118	0.829	0.253	Table 4
	Direct MT	0.191	0.851	0.187	Table 4
	Task-Preserved	0.193	0.855	0.277	Table 4
	EuroAlpaca	0.217	0.867	0.296	Table 4
Llama-3.2-3B-Instruct	Off-the-shelf	0.186	0.860	0.305	Table 4
	Direct MT	0.196	0.858	0.196	Table 4
	Task-Preserved	0.201	0.860	0.263	Table 4
	EuroAlpaca	0.218	0.869	0.310	Table 4
Macro mean (4 models)	Off-the-shelf	0.124	0.835	0.309	Table 4
	Direct MT	0.203	0.860	0.217	Table 4
	Task-Preserved	0.207	0.862	0.312	Table 4
	EuroAlpaca	0.223	0.870	0.349	Table 4

Same training volume, same models, same benchmarks. Reference similarity rises under direct MT; verifiable instruction-following falls below the untuned baseline. Source: arXiv:2609.05043, Table 4.

The drop is not an average hiding a split. Direct MT lowered European-IFEval accuracy for all four models, by 0.066 to 0.126 absolute, and Tiny-Aya-Global, the most multilingual of the four, lost the most (0.421 to 0.295). EuroAlpaca beat Direct MT on instruction following in all 200 model-language pairs. The intermediate Task-Preserved condition, before the final validation pass, beat it in 199 of 200 and recovered roughly to the untuned level (0.312 vs 0.309). Validation added a further 0.037.

The one comparison EuroAlpaca loses is worth stating. Against Bactrian-X, which pairs translated prompts with responses generated natively in the target language, EuroAlpaca scores lower on European-IFEval on their 20 shared languages, 0.385 against 0.401, while winning on Aya. The authors suggest the generated responses are the reason. It is the only external resource in the paper that beats it on instruction following.

Translation silently deletes the thing the example was testing

An English example that asks the model to correct "You was late to the party" stops being a grammar task once a translator fixes the error on the way into Spanish. "Write exactly ten words" does not survive translation with its count intact. A rhyme stops rhyming. A classification label translated into Polish no longer matches the checker. The paper's router classifies every example before anything is translated.

The EuroAlpaca routing, with the counts for the 51,760 source examples, and the reasons the 11,088 rewritten examples could not be translated. Source: arXiv:2609.05043, Figure 1, Table 1 and Table 9.

Field policies matter as much as routing. Of the 40,672 examples that pass the translation gate, the router preserved the input field unchanged in 33,344 cases (empty inputs count as preserved) and the output field in 10,493 (paper Table 10). Those are the code blocks, JSON keys, class labels and source-language texts in translation tasks that a uniform "translate every field" job corrupts.

We checked how much of this a keyword filter would catch. Running simple patterns (grammar, spelling, rhyme, letter counts, "N words", translate, code, classify) over the 51,760 Alpaca Cleaned instructions flags 4,828 examples, 9.3%. The paper's LLM router sent 23,814, 46.0%, to field-preserving or rewrite handling. The grammar pattern alone finds 640 instructions; the router rewrote 4,694 for English grammar or spelling dependence. Keyword triage undercounts task-breaking examples by about five times. The script is below.

Per-language gain of EuroAlpaca over Direct MT on European-IFEval, grouped by the paper's resource tiers. The per-language values are the paper's Figure 6; the tier grouping and means are our arithmetic. Source: arXiv:2609.05043, Figure 6 and Table 13.

Averaged by tier, the gain is 15.3 points for high-resource languages, 10.9 for medium and 12.5 for low, or 11.3 without Asturian. The pipeline depends on Gemma-4-31B to decide, rewrite and validate, and the authors say plainly that for Latgalian, Lombard or Friulian its quality "is bounded by the LLM's pre-existing exposure". The method helps everywhere and helps least in the languages that most need synthetic data.

If you localise training or evaluation data, route it before you translate it

Do not judge translated instruction data by reference similarity. ROUGE-L went up under direct MT for all four models (F-BERT for three of four) while instruction-following went down for all four. A data vendor reporting only similarity to references is reporting the metric this paper shows can move in the wrong direction.

Expect about half of a general instruction set to need non-uniform handling. 54.0% of Alpaca translated cleanly; 24.6% needed at least one field preserved and 21.4% needed the task rebuilt. For a code-heavy or classification-heavy set the preserve share will be higher. Budget human review for the rewrite bucket: those are the examples where a target-language speaker has to invent an equivalent error, rhyme or idiom.

Freeze the literals. Labels, JSON keys, code, URLs, file paths and the source text inside translation tasks must be copied, not translated. That rule is cheap, deterministic, and on its own accounts for a quarter of the corpus.

For low-resource varieties, the router is the weak link. A pipeline whose decisions come from a 31B model inherits that model's gaps. Native-speaker review of the rewrite bucket in those languages is where human effort buys the most, and where consented, paid contributors are the only reliable source.

A judgement, marked as one: the Bactrian-X row suggests the next step is not better translation of English responses but responses written or generated natively in the target language. Task-preserving localisation fixes what translation breaks; it does not add what English data never had.

Check it yourself

# 1. How much of Alpaca a keyword filter would flag (BLOMEGA's check, 19 Sep 2026)
curl -sL -o alpaca.json https://raw.githubusercontent.com/gururise/AlpacaDataCleaned/main/alpaca_data_cleaned.json
python3 - <<'EOF'
import json, re
d = json.load(open("alpaca.json")); print(len(d))            # 51760
pats = [r"\b(grammar|grammatical|spelling|misspell|punctuat)", r"\brhym",
        r"\b(letters?|characters?)\b.*\b(start|begin|end|contain|count|number)|\b(starts?|begins?) with the letter",
        r"\b(\d+|one|two|three|four|five|six|seven|eight|nine|ten|twenty|fifty|hundred)[- ]words?\b",
        r"\btranslat", r"\b(python|javascript|java|sql|html|function|code)\b", r"\b(classify|categori[sz]e|label)\b"]
n = sum(any(re.search(p, x["instruction"], re.I) for p in pats) for x in d)
print(n, round(100*n/len(d), 1))                          # 4828 9.3
EOF
# the paper's LLM router: 12,726 + 11,088 = 23,814 (46.0%)

# 2. The multilingual IFEval source used for languages without a curated release
#    https://huggingface.co/datasets/danish-foundation-models/multi-ifeval

# 3. Relative change, from Table 4 macro means
python3 -c "print(round((0.217/0.309-1)*100,1), round((0.349/0.309-1)*100,1))"   # -29.8 12.9

The paper links two repositories, github.com/Telefonica-Scientific-Research/EuroAlpaca and github.com/Telefonica-Scientific-Research/european-ifeval. Both returned HTTP 404 on 19 September 2026, and neither dataset appeared in a Hugging Face Hub search. Until they are public, the resource itself cannot be downloaded; every number above is checkable against the paper's tables, and the per-language results for all 200 model-language pairs are in its Appendix C.

What would prove this wrong

The claim is that uniformly machine-translated instruction data lowers verifiable instruction-following below the untuned model, and that routing plus task-level rewriting reverses it. The paper reports a single training seed per run. It is wrong if, by 30 June 2027, a replication with at least three seeds, on at least 10 of the 50 languages and at least two of the four models, finds Direct MT within 0.02 of the off-the-shelf European-IFEval accuracy. It is also weakened if the same replication, using a router other than Gemma-4, finds EuroAlpaca-style data beating Direct MT by under 5 points on average, against the 13.2 reported.

Sources

Aleix Sant, Jordi Luque (Telefónica Innovación Digital) and Carlos Escolano (UPC), EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages, arXiv:2609.05043v1, 4 September 2026. Table 1 (handling regimes), Tables 2 and 3 (failure taxonomy), Table 4 (main results), Table 5 (Okapi, MITS, Bactrian-X), Table 9 (rewrite reasons), Table 10 (field policies), Table 13 (resource tiers), Figure 6 (per-language gains), Limitations.
gururise/AlpacaDataCleaned on GitHub, the 51,760-example source set.
danish-foundation-models/multi-ifeval, one of the three source tiers for European-IFEval.
Jeffrey Zhou et al., Instruction-Following Evaluation for Large Language Models (IFEval), 2023, the checker design European-IFEval harmonises.

FAQ

Can you machine-translate instruction-tuning data into other languages?

Only for about half of it. In EuroAlpaca (arXiv:2609.05043, 4 September 2026), fine-tuning four models on Alpaca translated field by field into 50 European languages cut European-IFEval accuracy by 29.8% relative to the untuned models, even though ROUGE-L and BERTScore rose. A router found 54.0% of examples safe to translate directly, 24.6% needed fields preserved and 21.4% needed the task rebuilt.

What is task-preserving localisation?

Building a target-language instruction, input and output triple that keeps the original task, its constraints, difficulty and answer relation, even when that means not translating literally: inventing an equivalent grammar error, rhyme or idiom, or copying labels, code and JSON keys unchanged. EuroAlpaca's version raised instruction-following 12.9% above the untuned baseline.

Which instruction examples break when translated?

In Alpaca, the 11,088 examples EuroAlpaca had to rewrite were mostly English grammar or spelling tasks (42.3%), letter or character constraints (21.4%), language-specific tasks (10.5%), word or line counts (9.5%), rhymes (5.8%), idioms (3.7%) and wordplay (3.5%).

Do keyword filters find the examples that break under translation?

No. A keyword filter over the 51,760 Alpaca Cleaned instructions flags 4,828 (9.3%), against 23,814 (46.0%) routed for special handling by EuroAlpaca's LLM router. BLOMEGA ran the filter on 19 September 2026.

Does task-preserving localisation help low-resource languages?

Yes, but less. Grouping the paper's per-language gains by its resource tiers, EuroAlpaca beats direct MT by 15.3 points on average in high-resource languages, 10.9 in medium and 12.5 in low (11.3 without Asturian). Georgian gained least at 6.5 points.