The best speech recognizer gets 51% of words wrong on Moroccan YouTube speech

What GigaSpeechBench measured, and when

27 June 2026. A 38-author team from SpeechColab, Shanghai Jiao Tong University, Tsinghua, NTU, UIUC, Alibaba and others posted GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark. The current version is v3, dated 21 July 2026. The benchmark totals 680 hours of manually transcribed speech in five modules: low-resource languages, six Chinese dialects, six English accents, 12 vertical domains, and child and older-adult speech. It is released under CC BY 4.0 on Hugging Face, ungated, with evaluation code on GitHub.

The low-resource module is the one that matters for localization. It covers Arabic speech from seven regions (Iraq, Algeria, the UAE, Egypt, Morocco, Saudi Arabia, Syria), five Southeast Asian languages (Indonesian, Malay, Filipino, Vietnamese, Thai), plus Japanese and Korean. Three design choices separate it from FLEURS and Common Voice, which are read speech. The audio comes from YouTube and includes multi-speaker conversation and noisy rooms. All of it was published within the year before collection, which the authors say limits overlap with existing training data. And the reference transcripts were written by paid annotators through a professional annotation company, with Chinese and English translations added for 11 of the languages so speech translation can be scored too.

The authors ran nine commercial APIs (Microsoft Azure Speech, Google Chirp 3, OpenAI GPT-4o Transcribe, Gemini 3.0 Flash, ElevenLabs Scribe v2, Qwen3-ASR-Flash, Qwen3.5-Omni-Plus and two ByteDance Seed-ASR versions) and open models including Whisper Large v3, NVIDIA NeMo Canary, Meta OmniASR-LLM-3B, Qwen3-ASR 1.7B, FunASR and Dolphin. Table 2, the low-resource table, has 16 rows once Deepgram Nova 3 is included. The same paper also reports FLEURS and Common Voice scores for most of these systems, which is what makes a like-for-like comparison possible.

How far does each system fall from FLEURS to real speech?

Take one system and hold everything else fixed. ElevenLabs Scribe v2, which ElevenLabs lists at $0.22 per hour of audio, has the best FLEURS average of the seven systems scored on both test sets: 5.38% across eight shared languages. On GigaSpeechBench the same eight languages average 24.90%. Japanese character error rate goes from 2.41% to 29.95%, a factor of 12.4. Malay goes from 3.92% to 38.52%, a factor of 9.8.

Scribe v2 is not an outlier. Gemini 3.0 Flash averages 5.82% on FLEURS and 28.80% on the real speech. Google Chirp 3 goes from 6.21% to 24.91%. Whisper Large v3 goes from 7.81% to 34.65%. On average every system tested on both sets gets at least 2.38 times worse, and Azure is the one with the smallest drop.

The degradation is not uniform, and that is the part that should change procurement. The two systems with the worst and third-best FLEURS averages swap places with the systems at the bottom of the real-speech table.

The rank calculation is ours. Differences in rank between FLEURS and GigaSpeechBench are 0 for Scribe v2, Whisper and OmniASR, 2 for Gemini and Chirp 3, and 4 for GPT-4o Transcribe and Azure. That gives a sum of squared differences of 40 and a Spearman coefficient of 1 minus 240 over 336, which is 0.29. Seven systems is a small sample, so read it as "weak" rather than as a precise figure.

Three things in that table are not in the paper's own discussion.

Arabic is a regional problem, not a language problem. The best system's error rate spans 13.76% (Syria) to 51.34% (Morocco) inside one language label. A vendor quoting a single "Arabic" accuracy figure is averaging across a 3.7-fold spread. On Common Voice Arabic, Table 3 of the same paper puts Scribe v2 at 10.14%; its seven-region GigaSpeechBench Arabic average is 41.11%.

No single vendor wins. FunASR-Realtime has the best Southeast Asian average (16.85%) and Japanese and Korean scores, then averages 55.11% on Arabic, twelfth of 15 systems with Arabic results. Qwen3.5-Omni-Plus has the best Arabic average (32.80%). GPT-4o Transcribe is the best system in the table on UAE speech (26.26%) and the fourth worst of the 15 systems with Egyptian results (64.23%).

An open 1,600-language model is not a long-tail solution out of the box. Meta's OmniASR-LLM-3B averages 42.49% across the eight shared languages and 65.52% on Morocco. Meta positioned the family as transcription for more than 1,600 languages when it launched in November 2025.

Where a transcription error goes in a dubbing pipeline

A cascaded AI dub is transcription, then translation, then synthesis, then a mix. Each stage consumes the previous stage's output as ground truth. A transcription error is not diluted downstream. It is translated fluently and then spoken in a confident, well-timed voice.

The speech translation numbers in Table 10 back that up without any cascade assumption. Qwen3.5-Omni-Plus averages 49.87 chrF++ into English across 11 subsets and Gemini 3 Flash Preview 49.82. Microsoft's Azure translation service averages 40.05 and SeamlessM4T v2 Large 34.19. For the UAE subset the best score is 41.54, and SeamlessM4T v2 gets 24.30. Moroccan speech is the hardest subset for recognition and the third hardest for translation into English, after the UAE (41.54) and Malay (44.89) subsets.

What to change if you dub or caption long-tail locales

Stop accepting FLEURS numbers in vendor evaluations. A 0.29 rank correlation means a FLEURS leaderboard tells you little about which vendor to buy for real source video. Ask for a score on held-out audio that looks like yours: recent, conversational, noisy, and from the specific country. GigaSpeechBench's own filter for recency is "within the past year", and that is a reasonable default for a private test set too.

Pick the recognizer per locale, not per contract. On this data the best system for Saudi and Syrian speech is not the best for Malay, Thai or Japanese, and the best for UAE speech is second worst on Egypt. A pipeline that routes each locale to its own recognizer beats one vendor across the board. The cost is integration work and a per-locale test set, both of which are cheaper than re-dubbing.

Budget human transcription review for North African Arabic. At 44% to 51% best-case WER on Algerian and Moroccan speech, automatic transcription is a draft. Our judgement: for those two subsets, native-speaker correction of the transcript should be priced as a standard line item, not as an exception. For Saudi and Syrian speech, at 13.76% to 16.56%, spot-checking is a defensible compromise.

Treat "supports N languages" as a coverage claim, not an accuracy claim. OmniASR-LLM-3B is part of a family marketed for 1,600+ languages and averages 42.49% on eight widely spoken ones in this test. Coverage and accuracy are separate columns in any vendor comparison.

The fix is in-language, in-domain data. The authors attribute the gap to the difference between read benchmark speech and recent real-world audio. The only lever a buyer controls is adaptation data that matches the target: transcribed, consented recordings from the region and register you are dubbing. That is our reading of the result, not a claim in the paper.

Check it yourself

The dataset is ungated and the per-system hypothesis files are published next to the audio, so the central numbers can be recomputed without calling any API.

# 1. pull the Moroccan subset and every system's published output
pip install -U "huggingface_hub[cli]"
huggingface-cli download speechcolab/GigaSpeechBench --repo-type dataset \
  --include "Low-Resource-Languages/data/MAR/*" "Low-Resource-Languages/results/*" \
  --local-dir gsb

# 2. get the scoring code, including the per-region text normalizers
git clone https://github.com/SpeechColab/GigaSpeechBench
ls GigaSpeechBench/text_norm      # MAR.py, DZA.py, EGY.py, ... one per subset
cat GigaSpeechBench/run_ASR.sh     # the exact normalize + compute_wer invocation

# 3. inspect the reference file: segments carry text, text_en, text_zh,
#    speaker, gender, age_group and emotion; audios carry duration
python3 - <<'PY'
import json
d = json.load(open("gsb/Low-Resource-Languages/data/MAR/metadata.json"))
segs = sum(len(a["segments"]) for a in d["audios"])
hours = sum(a.get("duration", 0) for a in d["audios"]) / 3600
print(len(d["audios"]), "recordings,", segs, "segments,", round(hours, 1), "hours")
PY

# 4. the rank correlation in this note, from Tables 2 and 4
python3 - <<'PY'
fleurs = {"Scribe v2":5.38,"Gemini 3.0 Flash":5.82,"GPT-4o Transcribe":5.94,"Chirp 3":6.21,
          "Whisper Large v3":7.81,"OmniASR 3B":10.29,"Azure":10.59}
wild   = {"Scribe v2":24.90,"Chirp 3":24.91,"Azure":25.21,"Gemini 3.0 Flash":28.80,
          "Whisper Large v3":34.65,"OmniASR 3B":42.49,"GPT-4o Transcribe":44.59}
rank = lambda d: {k:i+1 for i,k in enumerate(sorted(d, key=d.get))}
rf, rw = rank(fleurs), rank(wild)
n = len(rf); d2 = sum((rf[k]-rw[k])**2 for k in rf)
print("spearman", round(1 - 6*d2/(n*(n*n-1)), 2))   # 0.29
PY

The eight-language averages in the slope chart are plain means of the Table 2 and Table 4 cells for Egyptian Arabic, Indonesian, Malay, Filipino, Vietnamese, Thai, Japanese and Korean. The repository README reports its leaderboard with a filter that drops segments of 0.5 seconds or less, and its numbers match Table 2.

What would prove this wrong

The claim under test is that read-speech benchmarks do not predict which recognizer performs best on real source video in long-tail locales. It is wrong if, by 30 June 2027, a published re-run of GigaSpeechBench and FLEURS covering at least seven common systems and the same eight languages shows a Spearman rank correlation of 0.7 or higher. Today it is 0.29.

A second prediction, marked as judgement: no system on the public GigaSpeechBench leaderboard will score below 30% WER on the Moroccan subset by 30 June 2027, using the repository's own normalizer. The best today is 51.34%. Getting under 30% needs regional training data that does not exist at scale in public corpora, and a year is short for that to change. If a system posts under 30% by that date, this reading was wrong.

FAQ

How accurate is speech recognition on Arabic dialects in 2026?

Not accurate enough to feed a dub without human correction. On GigaSpeechBench the best of 16 systems scores 51.34% WER on Arabic speech from Morocco, 44.22% on Algeria, 37.12% on Egypt, 28.54% on Iraq and 26.26% on the UAE. Saudi Arabia (16.56%) and Syria (13.76%) are the only Arabic subsets where the best system stays under 20%. The best seven-subset Arabic average is Qwen3.5-Omni-Plus at 32.80%.

Do FLEURS scores predict how an ASR system performs on real speech?

Poorly. Across seven systems scored on both test sets for the same eight languages, the Spearman rank correlation is 0.29. GPT-4o Transcribe ranks third on FLEURS at 5.94% and last on real speech at 44.59%. Azure ranks last on FLEURS at 10.59% and third on real speech at 25.21%.

Which speech recognition API is best for Southeast Asian languages?

On GigaSpeechBench's YouTube speech, FunASR-Realtime has the best Southeast Asian average at 16.85% WER, followed by Qwen3.5-Omni-Plus at 19.61% and Chirp 3 at 20.87%. ElevenLabs Scribe v2 averages 22.60% and Azure 22.68%. Malay is the hardest of the five, with a best score of 25.20%.

Why does AI dubbing quality drop in low-resource locales?

Because transcription, the first stage, is where real-world speech breaks, and every later stage inherits the errors. Scribe v2 moves from 2.94% WER on FLEURS Indonesian to 22.91% on recent YouTube Indonesian, and from 2.41% to 29.95% CER on Japanese. Translation and synthesis downstream cannot recover words the recognizer never heard.

Sources

1. Tu, Yang, Wang et al., GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark, arXiv:2606.28884, submitted 27 June 2026, v3 21 July 2026, HTML. Table 2 (low-resource WER/CER, 16 systems), Table 3 (Common Voice), Table 4 (FLEURS), Table 10 (speech translation into English), sections 3 and 4 on collection, recency and annotation, ethics statement on Creative Commons sourcing and paid annotators. CC BY 4.0.
2. speechcolab/GigaSpeechBench on Hugging Face, ungated, last modified 9 July 2026, read 15 September 2026. Per-subset metadata.json, audio.tar.gz and per-system results/*.json and results_trans/*.json.
3. SpeechColab/GigaSpeechBench on GitHub, read 15 September 2026. README leaderboard with regional averages (Qwen3.5-Omni-Plus 32.80% Arabic, FunASR-Realtime 16.85% Southeast Asian, 17.68% East Asian), run_ASR.sh, scripts/compute_wer.py, text_norm/.
4. Omnilingual ASR Team, Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages, arXiv:2511.09690, 12 November 2025, and the AI at Meta launch video, 10 November 2025.
5. ElevenLabs API pricing, Scribe v2 at $0.22 per hour, as recorded in BLOMEGA's dubbed-minute cost note on 10 September 2026.
6. Spearman rank correlation, eight-language averages and FLEURS-to-real ratios: BLOMEGA calculations from the tables above, 15 September 2026. Code in "Check it yourself".

Related BLOMEGA guides: A dubbed minute costs $0.33 to $9.00 · A 30-trillion-token corpus buys Basque a 160-million-parameter model · Your multilingual LLM judge prefers the machine translation · Consented AI training data providers.