mu-bench's best 11.9% UER is 6.7% outside Mandarin, and its leaderboard scores the same six APIs up to 7.6 points worse

Why does mu-bench's best 11.9% hide a 32.9% Mandarin error rate?

UER is macro-averaged: each of the five locales counts one fifth, whatever its size (Section 3.3, Equation 1). Google Chirp-3 batch scores 3.2% in en-US, 7.3% in es-MX, 9.5% in tr-TR, 6.7% in vi-VN and 32.9% in zh-CN (Table 5). The Mandarin cell supplies 32.9 of the 59.6 points that sum to 11.9 x 5, which is 55% of the headline (our arithmetic). Without Mandarin the same provider averages 6.7%. For every one of the nine rows in Table 5 Mandarin is the worst cell, which is the paper's own "Mandarin is hardest for all six" claim, and it contributes between 32.7% (Grok) and 55.2% (Google batch) of each row's macro score (our arithmetic).

We then read the public leaderboard the paper points to. It is not the paper's scoring. The leaderboard was launched on 20 April 2026 and grades with gpt-4.1-2025-04-14, scoring each aligned word error, while the paper grades whole utterances with gpt-6-astra. The repository README says the two are "not directly comparable". The numbers show how far apart they are: the same six batch APIs score 0.1 to 7.6 points worse on the leaderboard, and two pairs of providers change order.

What Sierra and UC Berkeley published on 25 September 2026

arXiv:2609.32082, mu-bench: A Multilingual Utterance Transcription Benchmark, by Andrea Li (UC Berkeley) and Soham Ray (Sierra AI), was submitted on 25 September 2026 and is formatted for ICASSP 2027. Native speakers role-played four customer personas on phone calls to an AI banking agent that collects a name, email or phone number, checks card status, raises limits, disputes charges and issues a case code. Only the caller side is released: 4,270 utterances from 250 calls, 50 per locale, 8 kHz mono, 5.08 hours (Table 2).

The paper's contribution is the metric. UER asks an LLM judge whether a normalized transcript derails the meaning or misspells any part of a form-field value. On 1,847 rows rated by native speakers it agrees with humans at Cohen's kappa 0.779 (95% CI 0.717 to 0.835), against 0.533 for "any difference after normalization" and 0.098 for that test on raw text (Table 3). A reference-guided LLM normalizer fixes format mismatches such as "C N 6 5 3" against "cn653" (1,496 provider-utterance pairs) and "Ashley dot Brown at email dot com" against "[email protected]" (1,141 pairs, Table 6). The leaderboard, dataset and blog post went live on 20 April 2026; Sierra CEO Bret Taylor announced it that day.

The release matches the paper's counts; the leaderboard does not match its scores

We listed all 4,274 files in the Hugging Face repo through the public tree API (the audio is gated, the file list is not), cloned the GitHub branch icassp-2027, and recomputed every leaderboard UER from the 9 x 4,270 per-utterance JSON files it ships. All nine recomputations match scores.json to four decimals.

Hugging Face repo sierra-research/mu-bench (last modified 20 April 2026) and GitHub sierra-research/mu-bench branch icassp-2027 at a204586, both read 30 September 2026. Leaderboard values from the same commit's results/.

Two things follow. The paper's two closest calls (Azure against ElevenLabs, Grok against OpenAI) flip under a different judge, which is consistent with the paper's own finding that those pairs are not reliably separated. And the Azure gap sits almost entirely outside English: 2.8% against 3.1% in en-US, then 13.2 and 11.7 extra points in Turkish and Vietnamese. Which pipeline is closer to a human is not something the release lets anyone test, because the 1,847 human ratings are not published.

A second check the paper did not run: does WER alone pick the same winner? On the leaderboard's nine full entries, Gemini 3.5 Transcribe has the lowest macro WER (7.97% against Chirp-3's 8.02%) while Chirp-3 has the lowest UER (14.1% against 14.9%). Across the 36 provider pairs, WER and UER order 32 the same way and 4 the other way. In en-US they disagree on 16 of the 33 untied pairs, because every provider sits between 3.1% and 6.2% UER there (our Kendall counts from lb.json).

What makes a Mandarin email address fail half the time?

The leaderboard files let us split the error by what the caller was saying, which the paper reports only as one input-collection bucket (1,706 utterances, Section 5.4). We tagged each gold transcript with a regex: email address, case code (letters then digits, as in "C N 7 3 4"), a number with four or more digits, or everything else. This is narrower than the paper's definition, which also counts names, addresses, dates and amounts, so our "everything else" still holds some form-field turns.

Email addresses are the worst field in four of five locales: 56.1% in Mandarin, 50.7% in Vietnamese, 28.4% in Spanish. Turkish is the exception, with numbers at 33.9% on 145 utterances, mostly the phone number "Artı 90 123 0 90 45 67", which providers write as digits, as words ("doksan yüz yirmi üç"), or with a plus sign. The headline is still carried by ordinary speech: it holds 76.8% of English, 75.1% of Vietnamese and 68.5% of Mandarin significant errors, because it is 65% to 78% of the utterances. Vietnamese ordinary speech alone runs at 38.4%.

The Mandarin email case has a structural cause. The gold writes the local part in Chinese characters ("美玲 at email dot com。"), and the annotators spelled the other persona's jūnlíng-sounding name four ways: 君玲 41 times, 郡凛 10, 君灵 3, 峻岭 2. Seven of the nine APIs write the local part in Latin letters on 111 to 137 of the 140 email turns ("meiling", "junlin", "mailing", "dreamling"). No audio can tell a system which characters, or which romanization, the reference expects. Whether "[email protected]" is a significant error then depends on the judge, and the judge does not answer the same way twice.

Across all locales, 4,572 exact gold-and-transcript pairs occur at least twice (15,663 rows). 164 of those groups get split verdicts, with 232 rows taking the minority verdict: about 1.5% of repeated rows (our count). Turkish (53 groups) and Mandarin (59) hold most of them. When we group by the normalized strings the judge actually receives, 129 of 2,314 recurring inputs still split, so normalizer drift does not explain it. Our reading, a judgement: calls to a hosted model at temperature 0 with a seed are not fully deterministic, and each split lands on a borderline form-field case. The paper reports 0.3% of verdicts changing between seeds for its own judge (Section 5.2). The leaderboard pipeline has no equivalent number.

How wide is one language's error bar with 50 calls?

The paper bootstraps whole conversations for the macro ranking (Section 5.3.1) but prints no interval on a single locale cell. We ran the same conversation bootstrap (4,000 resamples) on the leaderboard verdicts. Non-English cells carry 95% intervals of plus or minus 2.6 to 4.2 points, 1.4 to 2.1 times the variance a naive per-utterance binomial gives, because each call contributes a median of 14 to 17 utterances from one speaker on one phone. Chirp-3's Mandarin cell is 29.5% [26.0, 33.2].

That width decides real comparisons. In Mandarin, Gemini 3.5 Transcribe beats Chirp-3 by 2.5 points with a paired interval of [-0.8, 5.8]: not separated. In Vietnamese, ElevenLabs beats Azure by 3.0 points [-1.0, 7.2]: not separated. In Turkish, the 11.6-point ElevenLabs lead over Azure [7.2, 15.6] is real. Our computation; the paired numbers use each resample's same calls for both providers.

What this means for teams choosing ASR for a multilingual voice agent

• Read the locale column, never the macro. On paper numbers, Chirp-3 batch is the best batch API in es-MX, tr-TR and vi-VN and a 32.9% UER system in Mandarin. Azure batch is best in English (2.8%) and in Mandarin (31.2%). A single macro number mixes those into 11.9% and 17.0%.
• Pick the leaderboard or the paper, and say which. They disagree by up to 13.2 points in one cell. The leaderboard is the only one with released per-utterance verdicts, covers newer APIs (Gemini 3.5 Transcribe, Qwen3-ASR on Baseten and Modal) and runs a judge whose prompt is not public. The paper has the calibrated judge and no per-utterance output.
• Test your own form fields in your own script. Mandarin and Vietnamese email turns fail on half of all attempts, and part of that is a reference convention (characters for a Latin-alphabet address). If your agent reads back an email, collect the local part letter by letter or confirm it, and score that turn separately.
• Budget at least 50 calls per language, and expect plus or minus 3 to 4 points anyway. Differences under about 4 points in one locale need a paired test before they justify switching vendors.
• Watch for judge drift in any LLM-graded ASR metric. Score a fixed set of transcript pairs twice and count flips before trusting a leaderboard move under 2 points. For how LLM judges misbehave on translation, see six LLM judges that scored a machine above the human reference; for real-world ASR outside English, the 51% word error rate on Moroccan YouTube speech.

None of this undercuts the dataset. The counts reproduce exactly, the calls are consented role-play with invented personal data (Section 8), and phone-channel form-field speech in five languages was not public before April. Our judgement is narrower: the paper's ranking and the leaderboard's ranking are two measurements of the same audio, and a buyer should treat the gap between them as the uncertainty.

Check it yourself

Counts against Table 2, from the public file list (no Hugging Face access needed):

python3 - <<'PY'
import json, re, urllib.request, collections
url = "https://huggingface.co/api/datasets/sierra-research/mu-bench/tree/main?recursive=true"
files = []
while url:
    r = urllib.request.urlopen(url); files += json.load(r)
    m = re.search(r'<([^>]+)>;\s*rel="next"', r.headers.get("Link") or ""); url = m and m[1]
n, conv, byt = collections.Counter(), collections.defaultdict(set), 0
for f in files:
    m = re.match(r'(\w\w-\w\w)/conv-(\d+)-turn-\d+\.wav', f["path"])
    if m: n[m[1]] += 1; conv[m[1]].add(m[2]); byt += f["size"] - 44
print(dict(n), sum(n.values()), sum(len(v) for v in conv.values()), round(byt / 16000 / 3600, 3))
PY
# expect en-US 817, es-MX 792, tr-TR 846, vi-VN 975, zh-CN 840; 4270; 250; 5.079

Leaderboard UER recomputed from the released verdicts, and the split-verdict count:

git clone --depth 1 -b icassp-2027 https://github.com/sierra-research/mu-bench.git && cd mu-bench
python3 - <<'PY'
import json, glob, collections
L = ["en-US", "es-MX", "tr-TR", "vi-VN", "zh-CN"]; groups = collections.defaultdict(list)
for p in ["google-chirp3", "azure", "elevenlabs-scribe-v2", "grok", "openai-gpt4o-mini-transcribe", "deepgram-nova3",
          "gemini-3-5-transcriber", "baseten-qwen3-asr", "modal-qwen3-asr"]:
    per = []
    for l in L:
        v = [json.load(open(f)) for f in glob.glob(f"results/{p}/details/{l}/*.json")]
        per.append(sum(d["significantWer"] > 0 for d in v) / len(v))
        for d in v:
            if d["gold"] != (d["predicted"] or ""): groups[(l, d["gold"], d["predicted"] or "")].append(d["significantWer"] > 0)
    print(p, round(100 * sum(per) / 5, 2), [round(100 * x, 1) for x in per])
rep = {k: v for k, v in groups.items() if len(v) > 1}
print(len(rep), sum(len(set(v)) > 1 for v in rep.values()))
print(sorted(groups[("zh-CN", "美玲 at email dot com。", "[email protected]")]).count(True), len(groups[("zh-CN", "美玲 at email dot com。", "[email protected]")]))
PY
# expect google-chirp3 14.13 ... azure 24.55 [3.1, 16.4, 30.7, 33.8, 38.7]; 4572 164; 40 59

Paper numbers: Table 5 (per-locale UER), Section 5.3.1 (p-values), Section 5.4 (input collection). The judge block with model, seed and prompt hashes is in every results/<provider>/scores.json.

What would prove this wrong

Our claims: the paper's macro UER is dominated by one locale, the leaderboard and paper grade the same APIs differently enough to reorder close pairs, and the leaderboard judge splits verdicts on about 3.6% of recurring transcript pairs.

The split-verdict claim is wrong if rescoring the released submissions/normalized files twice with the pinned leaderboard pipeline changes fewer than 0.5% of verdicts. The ranking claim is testable when Sierra rescores the leaderboard with the ICASSP pipeline. We predict that, if that happens by 31 March 2027, Azure batch will land within 1 point of its paper 17.0% and at least 5 points below today's 24.6%, and Mandarin email turns will still show UER above 40% averaged over the six providers unless the gold switches to a romanized local part. A rescored Mandarin email UER under 25% with the current references would falsify our reading that the reference convention drives it.

Sources

1. Andrea Li, Soham Ray. mu-bench: A Multilingual Utterance Transcription Benchmark. arXiv:2609.32082, submitted 25 September 2026. Abstract, Sections 3.3, 4.3, 4.4, 5.2 to 5.5, 8, Tables 2 to 6.
2. sierra-research/mu-bench on GitHub, branch icassp-2027, commit a204586 (22 September 2026): README, manifest.json, scoring/, results/*/scores.json and results/*/details/. Read 30 September 2026.
3. sierra-research/mu-bench on Hugging Face: gated dataset card (CC BY-NC 4.0) and public file tree, last modified 20 April 2026. Read 30 September 2026.
4. mu-bench leaderboard, Sierra Research, bundled data file read 30 September 2026.
5. Sierra. μ-Bench: an open multilingual transcription benchmark, 20 April 2026.
6. Bret Taylor. Launch post on X, 20 April 2026.
7. BLOMEGA analysis, 30 September 2026: file counts, UER recomputation, field-type tagging, split-verdict grouping, conversation bootstrap, WER against UER rank counts, macro-share arithmetic on Table 5.

FAQ

Which ASR API is best on mu-bench?

In the paper (Table 5), Google Chirp-3 batch at 11.9% macro UER, but 32.9 of its 59.6 summed locale points come from Mandarin; outside Mandarin it averages 6.7%. Azure batch is best in English (2.8%) and Mandarin (31.2%). On the public leaderboard, scored with a different judge, Chirp-3 leads at 14.1% and Gemini 3.5 Transcribe, which the paper does not test, is second at 14.9%; in Mandarin the two are not separated (difference 2.5 points, 95% interval -0.8 to 5.8, our bootstrap).

Why do the mu-bench paper and leaderboard show different numbers?

The leaderboard launched on 20 April 2026 and grades each aligned word error with gpt-4.1-2025-04-14; its prompt file is not public. The paper grades whole utterances with gpt-6-astra and publishes its prompts in scoring/prompts_icassp.py. The README says the two are not directly comparable. The same six APIs score 0.1 to 7.6 points worse on the leaderboard, Azure's Turkish cell moves from 17.5% to 30.7%, and Azure/ElevenLabs and Grok/OpenAI swap order.

Which caller inputs do speech recognizers get wrong most often on mu-bench?

On the leaderboard's per-utterance verdicts for the six paper providers, email addresses fail most in four of five locales: 56.1% of Mandarin and 50.7% of Vietnamese email turns have a meaning-changing error, against 7.1% in English. Turkish phone numbers fail 33.9% of the time. Mandarin references write the email name in Chinese characters while most APIs write Latin letters, and the judge scored the identical pair 美玲 at email dot com against [email protected] as an error 40 times and not an error 19 times.