Agent skill libraries have zero Hindi or Swahili skills, and quality filters pick skills worse than random

Lab note · 20 September 2026 · BLOMEGA

Huawei's M-SQE paper, posted 16 September 2026, audited about 84,700 community agent skills and found zero written in Hindi or Swahili. We rescanned the largest source, 61,650 skills, and the Hindi and Swahili zeros hold. The paper's second finding is the one to act on: three multilingual data-quality scorers, DEITA, M-DaQ and JQL, picked skills that agents succeeded with 51.6 to 53.5% of the time, below the 55.1% of a random pick. Huawei's own two-view scorer reached 69.1%.

What Huawei published on 16 September 2026

arXiv:2609.18445, "M-SQE: Multilingual Skill Quality Estimation for Enhancing Language Equality in Agentic Skill Use", has 12 authors from Huawei (Yilun Liu, Shimin Tao, Daimeng Wei and colleagues). Code, skill pools, tasks and checkers are released at github.com/lunyiliu/M-SQE.

An agent skill is a procedural document, typically a SKILL.md file, that an agent retrieves to learn how to call an API, fill a form or follow a convention. The paper's argument: skill libraries are the new localization surface. A Hindi-speaking user's agent retrieves an English skill, or a machine-translated one, or one an LLM made up, and task success depends on which.

The evaluation has 411 tasks: 94 general tasks in French, Hindi, Japanese, Korean, Swahili and Chinese; 265 tool-use utterances in seven languages over 55 functions (smart home, calendar, media) from a 52-language function-calling benchmark; and 52 cultural short-answer tasks across six regions. Each task draws from a pool mixing ecological-style skills, machine-translated skills and self-generated skills, the latter two produced by Qwen3.5-9B. Gemini-3-Flash scores, routes and solves.

Hindi and Swahili have no native agent skills; Japanese has more than the paper counted

The paper's audit uses Unicode script detection with a 25-character floor, then checks flagged repositories by hand. Script detection separates Chinese, Japanese and Korean cleanly. It cannot see French or Swahili at all, since both use Latin script, so those counts rest on the paper's targeted keyword searches. We downloaded the largest named source and ran our own detector over every skill body.

Table 1. Native-language agent skills. Paper column: arXiv:2609.18445 Table 1, audit dated July 2026 over about 84,700 un-deduplicated entries from "over a dozen" indexes. Our column: the largest of those sources, huggingface.co/datasets/LittleDinoC/agent-skills (61,650 skills, 8,809 GitHub owners), scanned 20 September 2026. A skill counts when at least 30% of the letters in its body are in the target script (Chinese, Japanese, Korean, Russian, Hindi) or, for Latin-script languages, when target-language function words make up over 10% of its words and outnumber English ones 1.5 to 1.
Language	Paper, all sources	Our scan, one source	Source
English	~79,700	not counted directly	Table 1: total minus non-English
Chinese	~4,000	2,712 (917 authors); 4,600 pass the paper's 25-character screen	Table 1 / our scan
French	~700	18 (10 authors), stopword test	Table 1 / our scan
Korean	~200	296 (65 authors)	Table 1 / our scan
Japanese	~80	353 (87 authors)	Table 1 / our scan
Spanish	not reported	150 (24 authors), stopword test	our scan
Russian	not reported	144 (18 authors)	our scan
Portuguese	not reported	40 (16 authors), stopword test	our scan
Hindi	0	0; 5 skills contain 25+ Devanagari characters, all English bodies (3 from sarvamai/skills)	Table 1 / our scan
Swahili	0	0, stopword test	Table 1 / our scan

Paper counts against ours. The zero rows reproduce exactly. Japanese and Korean exceed the paper's all-source totals in one source alone, which suggests the paper's hand check discarded many genuine skills; French is far lower in our source, which could be our conservative stopword test or French content concentrated in other sources. Source: arXiv:2609.18445 Table 1; our scan.

Two arithmetic notes. The three sources the paper names, LittleDinoC (61,650 on its dataset card; the paper says about 61,000), huzey/claude-skills (22,862) and wshobson/agents (175), sum to 84,687, which is essentially the whole 84,700 attributed to "over a dozen" indexes. And the total is un-deduplicated: huzey's set is crawled from skills.sh, which indexes GitHub repositories that LittleDinoC also crawls, so the English denominator is smaller than 79,700 by an amount nobody has measured. Neither note changes the conclusion. English dominates, and for Hindi and Swahili there is nothing native to retrieve.

The zero is not for lack of Indian-language AI companies publishing skills. Three of the five skills in our scan with more than 25 Devanagari characters come from sarvamai/skills, and all three are English documents with Hindi examples inside.

Relevance picks the skill that looks right; M-SQE also checks whether it runs

Average task success over nine settings. Red bars are multilingual instruction-data quality scorers; they judge a skill's language quality, not whether it carries the slots, schema or cultural fact the task needs. Source: arXiv:2609.18445, Table 3.

The paper's explanation for the quality scorers' failure is in its dimensions. DEITA, M-DaQ and JQL were built to filter instruction-tuning data, so they reward fluent, well-formed text. A fluent machine-translated smart-home skill that omits the device-location slot still fails the call. The paper's Lunar New Year example: a self-generated skill names the right festival and states the wrong customary red-envelope amount.

The M-SQE pipeline and its tool-use pool. The Theory view is scored on six dimensions a language-service team with an average of over six years' experience wrote: correctness, completeness, executability, cross-lingual faithfulness, localization and context efficiency. Source: arXiv:2609.18445, Sections 4 and 5, Tables 2 and 3, Figures 4 to 6.

Three results survive scrutiny. The gain is largest at small budgets (+9.7 pp over retrieve-only at top-1, +3.4 at top-10), which is where production agents run because each skill costs context. It holds when the router's domain labels are randomly corrupted on up to 90% of queries (+4.5 pp at 90%). And its paired sign-flip tests against retrieve-only are significant in every domain, retriever and budget slice (Table 6, smallest gain +2.1 pp with SkillFlow, p = 0.0079).

One result deserves a discount. Gemini-3-Flash is scorer, router and solver in the main setting. Swapping the backbone keeps M-SQE ahead by 3.5 to 9.0 pp, but a scorer that shares a solver's preferences has an easier job than one that does not.

For anyone shipping an agent outside English: localize the skill layer, and score it for execution

Your agent's skill library is part of your localization scope. If the product works in Hindi or Swahili, there is no community skill to fall back on. The choice is between English skills (cross-lingual retrieval, with the accuracy and recall loss the paper cites), machine-translated skills, or skills written natively. Budget for the last one in the markets you care about.

Do not reuse your training-data quality filter as a skill filter. In this paper, three published multilingual quality scorers were worse than random. Judge a skill by whether its function schema, required slots and values match the task, and treat a skill below a hard threshold as unusable, which is what M-SQE's tool-use rule (Theory score of at least 65 before relevance counts) does.

Keep the budget small and the check strict. Retrieval plus a usability score at top-1 beat retrieve-only by 9.7 pp. Handing the agent ten candidates and hoping it picks well gave the smallest gain.

Machine translation is where the cultural errors enter. The Theory view caps a skill at 40 if a correctness red line fails and at 80 if a basic dimension such as localization fails. Those caps exist because a single wrong fact, a wrong amount or a wrong convention, cannot be averaged out by fluent prose. That is a native-reviewer job.

Check it yourself

# Our scan of the largest source (274 MB parquet)
curl -sL -o dino.parquet \
  https://huggingface.co/datasets/LittleDinoC/agent-skills/resolve/main/total_skill_data.parquet
python3 - <<'EOF'
import pyarrow.parquet as pq, re, collections
t = pq.read_table("dino.parquet", columns=["owner", "content"])
S = {"hi": r"[\u0900-\u097F]", "ko": r"[\uAC00-\uD7AF\u1100-\u11FF]", "kana": r"[\u3040-\u30FF]",
     "zh": r"[\u4E00-\u9FFF]", "ru": r"[\u0400-\u04FF]"}
out = collections.Counter()
for c in t["content"].to_pylist():
    c = c or ""; n = len(re.findall(r"[^\W\d_]", c)) or 1
    k = {l: len(re.findall(p, c)) for l, p in S.items()}
    if k["kana"] >= 10 and k["kana"] + k["zh"] >= .3 * n: out["ja"] += 1
    elif k["ko"] >= .3 * n: out["ko"] += 1
    elif k["zh"] >= .3 * n: out["zh"] += 1
    elif k["hi"] >= .3 * n: out["hi"] += 1
    elif k["ru"] >= .3 * n: out["ru"] += 1
print(t.num_rows, dict(out))
EOF
# printed on 20 Sep 2026: 61650 {'zh': 2712, 'ja': 353, 'ko': 296, 'ru': 144}   (no 'hi' key: zero)

The Table 3 numbers are on page 6 of https://arxiv.org/pdf/2609.18445; the audit protocol is Appendix A. The released pools and checkers in the GitHub repository let you rerun any selector, including your own, against the same 411 tasks. The Swahili and Latin-script counts in our table use a stopword test that is not in the block above; it is conservative, so treat our French, Spanish and Portuguese numbers as floors.

What would prove this wrong

The claim is that the public agent-skill ecosystem has effectively no native Hindi or Swahili content, and that generic data-quality scoring is the wrong tool for choosing skills. The first part is wrong if a rescan of a public skill index of at least 50,000 entries, taken before 30 June 2027, finds 50 or more skills whose bodies are majority Devanagari or Swahili. We will rerun the scan above against LittleDinoC and huzey/claude-skills on that date.

The second part is wrong if an independent evaluation on the released M-SQE tasks shows DEITA, M-DaQ or JQL beating random selection by more than 3 pp with a solver other than Gemini-3-Flash.

Sources

Yilun Liu, Shimin Tao, Minggui He et al. (Huawei), M-SQE: Multilingual Skill Quality Estimation for Enhancing Language Equality in Agentic Skill Use, arXiv:2609.18445v1, 16 September 2026. Tables 1 to 4 and 6, Figures 3 to 6, Appendix A.
lunyiliu, M-SQE code and evaluation data, GitHub, accessed 20 September 2026.
LittleDinoC, agent-skills dataset, Hugging Face, 61,650 skills, last modified 23 May 2026, MIT licence. Scanned 20 September 2026.
huzey, claude-skills dataset, Hugging Face, 22,862 SKILL.md files, last modified 6 April 2026.
wshobson, agents, GitHub, the 175-skill repository the paper cites.

FAQ

How many agent skills are written in Hindi or Swahili?

None found. Huawei's M-SQE audit (arXiv:2609.18445, July 2026) of about 84,700 community skill entries found zero native Hindi or Swahili skills, and our scan of the largest source, 61,650 skills in LittleDinoC/agent-skills, also found zero on 20 September 2026.

What share of community agent skills are in English?

The paper estimates about 79,700 of about 84,700 entries, roughly 94%, with Chinese next at about 4,000. The total is not de-duplicated across sources, so the exact share is uncertain.

Can I use a multilingual data-quality scorer to choose agent skills?

The evidence says no. In M-SQE's Table 3, DEITA, M-DaQ and JQL selected skills with 52.5%, 53.5% and 51.6% average task success, below a random pick of retrieved skills at 55.1%. Scorers that check executability and task fit did better.

What is M-SQE?

A post-retrieval scorer from Huawei that rates each candidate agent skill on intrinsic quality (Theory view: correctness, completeness, executability, cross-lingual faithfulness, localization, context efficiency) and task fit (Action view), and combines them with a domain-specific rule. It reached 69.1% average task success against 62.8% for retrieval alone.

Does machine-translating English skills solve agent localization?

Partly. Machine-translated and self-generated skills fill the gap but carry wrong-language fragments, missing slots and cultural errors. M-SQE's biggest gains, +12.9 pp on Hindi and +5.6 pp on Swahili tool use, came from filtering those out.