Pseudo-dialect TTS adds 0.86 BLEU on Japanese dialects, and plain text distance predicts which dialects fail

LLM-written dialect text adds 0.86 BLEU on Japanese, and a text-only distance score predicts which dialects fail

On 7 October 2026 SB Intuitions posted arXiv:2610.09321, which trains speech language models on "pseudo-dialect" speech: standard sentences rewritten into a dialect by an LLM, then read aloud by an ordinary standard-language TTS voice, with zero real dialect audio. Read against its own Table II, the dialect part of the recipe is the smallest part of the gain: on 20 Japanese dialects it adds 0.86 BLEU over a same-size synthetic standard-speech control, while doubling the data with synthetic standard speech adds 1.17 and a standard-text intermediate target adds 2.02. In German, synthetic volume alone accounts for 9.20 of the 10.10 BLEU gained.

Our own measurement, from the public CPJD test transcripts: the character edit distance between each dialect transcript and its standard sentence explains 80% of the variance in the paper's per-dialect baseline BLEU (Pearson r = -0.89, 20 dialects). The same check found that 21.1% of the 5,250 test transcripts are identical to the standard sentence once punctuation is stripped, and that one CPJD speaker is mapped to the wrong parallel set in the corpus metadata. As of 8 October 2026 no code, model or generated data from the paper is public, and the Japanese TTS it used is licensed CC BY-NC 4.0.

The paper's comparison is fair in one important way: the synthetic standard-speech control has the same 1:1 volume and the same TTS as the pseudo-dialect set, so the 0.86 Japanese and 0.90 German margins are attributable to the dialect rewriting. Both are reported as significant by paired bootstrap (p below 0.001). Our judgement is that the abstract's framing, which leads with these margins, undersells how much of the total gain comes from cheaper steps a team can take first: adding synthetic standard speech, and asking the model to emit a standard-language paraphrase before translating.

What SB Intuitions posted on 7 October 2026, and what it trained on

arXiv:2610.09321 v1, by Shunsuke Mitsumori (SB Intuitions and Waseda University), Tomoya Mizumoto and Yusuke Fujita (SB Intuitions, Tokyo), was submitted on 7 October 2026 at 02:25 UTC under cs.CL and eess.AS, CC BY 4.0, with the comment "Accepted to IEEE SLT 2026". It is 7 pages with 7 tables. It builds on the ASRU 2025 paper by Mizumoto, Fujita and colleagues that tested Japanese dialect understanding in speech and text LLMs, and it reuses that paper's dialect-to-English translation protocol.

The model is a frozen Whisper large-v3 encoder and a frozen LLM (Llama-3.1-Swallow-8B-Instruct-v0.3 for Japanese, Llama-3.1-8B-Instruct for German and Chinese), joined by an 18.4M-parameter convolutional projector that is the only part trained. Training sets are standard-language ASR corpora: ReazonSpeech large v2 (about 2.6M utterances) plus Speech-BSD and CoVoST2 for Japanese, the German part of Multilingual LibriSpeech (about 0.4M), and 3M utterances of WenetSpeech. English labels for these were machine-generated by Qwen2.5-32B-Instruct.

The augmentation rewrites every standard transcript into one randomly chosen target dialect with gpt-oss-120b (20 Japanese, 8 German, 8 Chinese dialects), then synthesises it with Tsukasa-Speech using JVS reference voices for Japanese, and "Qwen-TTS" for German and Chinese. Tests use real dialect speech never seen in training: CPJD (Japanese, 20 dialects), SwissDial (8 Swiss German dialects) and KeSpeech (8 Mandarin subdialects), scored with SacreBLEU against English references that GPT-4o generated from each corpus's standard-language parallel text.

Which of the paper's numbers hold up when recomputed from its tables?

We typed Tables II, III, IV and VII into a script and re-derived every count the text states. The per-dialect claims hold. The averages and the attribution of gains need care.

Paper values from arXiv:2610.09321v1. Counts, sign tests, decompositions, correlations and release checks are BLOMEGA computations, 8 October 2026.

The averages first. Every "Avg." in Tables III and IV sits above the plain mean of its 20 dialect columns, by 0.06 to 0.15 BLEU. That pattern fits a corpus-level BLEU over all 5,250 test utterances rather than an average of per-dialect scores, and corpus BLEU weights Nara double because CPJD has two Nara speakers (500 utterances). Our judgement: not an error, but the column is not the average its label suggests, so a reader comparing a subset of dialects should recompute rather than reuse it.

The Chinese result is the clearest attribution gap. The +4.70 BLEU (11.67 to 16.37) is credited to the framework, but pseudo-dialect audio alone moved Chinese by -0.05, and every multi-task model was trained only on the pseudo-dialect-augmented set. No Standard+English model was trained on synthetic standard speech or on real data alone, so the paper cannot say whether pseudo-dialect audio contributes anything to the 16.37. The authors explain the null audio effect themselves: Chinese characters do not change with dialect pronunciation, so a standard TTS reads the rewritten text with standard Mandarin readings.

The cascade row is a free baseline the paper already contains. A fully fine-tuned Whisper feeding a LoRA-tuned Swallow scores 29.78 with no synthetic data at all, 1.52 BLEU above the best frozen SLM. The authors say the two architectures should not be compared on absolute score because their trainable parameters differ (1,585.4M against 18.4M), which is fair. For a team choosing a production stack, though, it says that fine-tuning the components you already have is worth more than this augmentation on the frozen design, and that synthetic standard speech did nothing for the cascade (-0.13) while pseudo-dialect speech still added 0.79.

How pseudo-dialect speech is built, and what each step inherits

The pipeline has three generative steps, and the only dialect knowledge in it is what gpt-oss-120b knows about writing each dialect. The TTS model never heard dialect speech, so the audio has standard prosody and accent, which the paper calls "non-native-sounding dialect speech". Its own failure analysis names both limits: gpt-oss-120b never produced the Kanazawa verb "gameru" (to pick up) anywhere in the training data, and the Tsugaru pronunciation of "aka" (red) defeated both the baseline and the proposed model because no training audio carries Tsugaru phonetics.

That split predicts where the method should help: dialects whose difference from the standard is mostly lexical and grammatical, which an LLM can write, rather than phonetic, which a standard voice cannot produce. The text distance we measured is a direct proxy for the lexical part. It correlates with the baseline at r = -0.89, and with the pseudo-dialect gain over synthetic standard speech only weakly (r = +0.30). Split into groups, the five most text-distant dialects (Morokata, Tsugaru, Akita, Kyoto, Miyazaki) gain +1.50 BLEU from pseudo-dialect audio, the other fifteen +0.76. The standard-text target gains about the same in both groups (+2.09 and +1.97), so it behaves like a general translation aid, not a dialect fix.

Seen this way, the Japanese baseline is mostly a function of how much text a dialect rewrites. Morokata, with 0.407 distance and no sentence left identical to the standard, scores 10.60; Saitama, where 78% of sentences equal the standard text, scores 28.66. A one-variable line (BLEU = 31.51 - 41.56 x distance) misses the reported baseline by 1.82 BLEU on average. Because each dialect is read by one speaker (Nara by two), speaker and dialect are confounded in every per-dialect column, and the distance correlation is the only part of this analysis that does not depend on who happened to record.

What this means for teams building dialect speech data or dubbing in dialect

• Do the cheap steps first. On Japanese, synthetic standard speech (+1.17) and a standard-language intermediate target (+2.02) together are worth 3.7 times the dialect rewrite (+0.86). On German, synthetic standard speech alone was +9.20. Our judgement: a team with a dialect gap should test both before building a dialect text generator.
• Score dialect distance from text before you collect audio. The CPJD transcripts are 517 KB and the distance takes seconds to compute. It ranked the 20 dialects by difficulty with Spearman -0.87 against the paper's baseline. For a localization team planning a collection, the same metric applied to a few hundred parallel sentences per variety is a cheap way to decide which varieties need real recordings.
• Pseudo-dialect audio cannot teach accent. The method's own failure case (Tsugaru "aka") and the Chinese null result both point to phonetics as the remaining gap. That is the part only real dialect speakers can supply, and it is where paid, consented collection still has to go. The paper's conclusion says the same: dialect-faithful TTS "under low-resource constraints" is future work.
• Check the licences before you copy the recipe. Tsukasa-Speech is CC BY-NC 4.0; JVS audio is free for research and non-commercial use with a separate commercial licence on request; ReazonSpeech is CDLA-Sharing-1.0 restricted to Japanese Copyright Act Article 30-4 analysis; WenetSpeech is non-commercial; KeSpeech forbids commercial use, adaptation and redistribution. The Japanese branch as published cannot ship in a commercial model without swapping the TTS and voices. gpt-oss-120b, Qwen2.5-32B, Whisper and Multilingual LibriSpeech are permissive.
• Audit LLM-written dialect text. Nothing in the paper measures how accurate gpt-oss-120b's dialect rewriting is; the authors list evaluation protocols for dialect text generation as future work. Before any pseudo-dialect corpus is used for dubbing scripts or ASR training, a native reviewer per variety should grade a sample, which is the same human-in-the-loop check our note on dialectal Arabic error spans argued for.
• If you reuse CPJD, fix one metadata row. speaker-info.csv in the text release lists speaker M009 (Nara) against common_set2.txt, but the transcript parallels common_set1.txt (edit distance 0.145 against 1.179). Pairing by the metadata would give every M009 utterance the wrong reference. The paper's Nara score (26.67) looks normal, so the authors probably paired correctly.

For how hours of real speech translate into BLEU when you can collect them, see our note on low-resource speech translation hours against BLEU.

Check it yourself

Everything here reproduces from three short scripts (headline.py, dialect_distance.py, check_release.sh, available from [email protected]) and two public downloads. The table recomputation needs no data:

python3 headline.py            # Tables II, III, IV, VII recomputed; claims 15/20 and 19/20; gain decomposition
# expected: "beats both: 15", "std+en > single: 19 losers: ['Morokata']",
# "Japanese volume +1.17 | dialect text +0.86 | output format +2.02 | total +4.05 | dialect share 21%"

The distance analysis needs the CPJD text release (CC BY-SA 4.0, 517,379 bytes on 8 October 2026). Extract it into its own directory and pass the path in:

mkdir -p dl && curl -sL -o dl/cpjd.zip https://ss-takashi.sakura.ne.jp/corpus/cpjd/cpjd_text-only_ver1.zip
(cd dl && unzip -q cpjd.zip)
python3 -I dialect_distance.py dl/cpjd_text-only_ver1
# expected: METADATA MISMATCH: M009 (nara-ben) listed as common_set2.txt, transcript matches common_set1.txt
#           Pearson dist vs no-aug BLEU r=-0.89 ; R^2 0.80 ; identical share 21.1%

The release and licence state, one call each:

curl -s "https://export.arxiv.org/api/query?id_list=2610.09321" | grep -E "published|comment"
curl -s "https://api.github.com/search/repositories?q=2610.09321"          # total_count 0 on 8 Oct 2026
curl -s "https://api.github.com/orgs/sbintuitions/repos?per_page=100"      # 18 repos, none for this paper
curl -s "https://huggingface.co/api/models?author=sbintuitions&limit=500"   # 34 models, none for this paper
curl -s https://huggingface.co/api/models/Respair/Tsukasa_Speech | grep -o 'license:[a-z0-9.-]*'   # cc-by-nc-4.0

Table values to type in are on PDF page 4 (Tables II to IV) and page 6 (Table VII). The full script set, headline.py, dialect_distance.py and check_release.sh, with their saved outputs, is the basis of every number marked "our" above.

What would prove this wrong

Our claim has two parts: that the dialect rewrite is a small, real gain next to synthetic volume and a standard-text target, and that it helps most where dialects differ from the standard in text rather than in sound.

It would be wrong if, by 30 June 2027, a replication or the authors' journal version shows pseudo-dialect audio beating a same-size synthetic standard control by 2.0 BLEU or more on average on a Japanese dialect test set with at least three speakers per dialect, or shows the gain on the five most text-distant CPJD dialects no larger than on the rest. A Standard+English model trained without pseudo-dialect audio that scores clearly below 16.37 on KeSpeech (by more than 1 BLEU) would show the Chinese gain does depend on the augmentation, and we would correct the attribution above. We also predict no public code or pseudo-dialect data for this paper by 31 January 2027; a release before then disproves that part.

Sources

1. Shunsuke Mitsumori, Tomoya Mizumoto, Yusuke Fujita (SB Intuitions; Waseda University). Synthetic pseudo-dialect augmentation for speech language models, arXiv:2610.09321v1 (full title on arXiv), submitted 7 October 2026, accepted to IEEE SLT 2026. Sections II to VI, Tables II, III, IV, V, VII.
2. Shinnosuke Takamichi, Hiroshi Saruwatari. CPJD corpus page and text-only release ver1 (2023-04-18), CC BY-SA 4.0, downloaded 8 October 2026.
3. Respair/Tsukasa_Speech on Hugging Face, licence cc-by-nc-4.0, read 8 October 2026.
4. JVS corpus terms of use: audio for academic and non-commercial research, commercial use by separate agreement, read 8 October 2026.
5. reazon-research/reazonspeech dataset card: CDLA-Sharing-1.0, solely for Japanese Copyright Act Article 30-4, read 8 October 2026.
6. wenet-e2e/wenetspeech dataset card (non-commercial, CC BY 4.0) and KeSpeech licence (non-commercial, no adaptations, no distribution), read 8 October 2026.
7. Hugging Face API licence tags for gpt-oss-120b, Qwen2.5-32B-Instruct, whisper-large-v3 (all Apache 2.0) and Multilingual LibriSpeech (CC BY 4.0), read 8 October 2026.
8. SB Intuitions on GitHub (18 repositories) and on Hugging Face (34 models, 21 datasets), searched 8 October 2026; no artefact for arXiv:2610.09321.
9. BLOMEGA computations, 8 October 2026: recomputed averages and counts from Tables III and IV, sign tests, gain decomposition from Table II, CPJD text distance and its correlation with Table III, identical-sentence share, CPJD metadata check.

FAQ

Does pseudo-dialect TTS augmentation work without any real dialect speech?

Yes, modestly. On 20 Japanese dialects in CPJD it raised dialect-to-English BLEU from 25.38 (same-size synthetic standard speech) to 26.24, and on Swiss German from 31.57 to 32.47, both p below 0.001 in the paper (Table II). It beat both baselines in 15 of 20 Japanese dialects. Larger gains came from synthetic standard speech (+1.17 Japanese, +9.20 German) and from predicting a standard-language sentence before the English translation (+2.02 Japanese). By our computation the gain is about twice as large on the five dialects whose text differs most from standard Japanese (+1.50 against +0.76).

Why did pseudo-dialect audio not help Chinese?

Pseudo-dialect audio moved Chinese BLEU on KeSpeech from 11.72 to 11.67, no gain. The authors attribute it to the writing system: Chinese dialect text uses the same characters as Mandarin, so a standard TTS reads it with Mandarin pronunciation and produces no dialectal audio. The +4.70 Chinese gain (to 16.37) came from the standard-text intermediate target, and because every multi-task model was trained on the pseudo-dialect set, the paper does not show whether that gain needs the augmentation at all.

Can I use this recipe to build a commercial dialect speech model?

Not as published. The Japanese TTS, Tsukasa-Speech, is CC BY-NC 4.0; JVS reference audio is free only for research and non-commercial use; ReazonSpeech is limited to Japanese Copyright Act Article 30-4 analysis; WenetSpeech and KeSpeech are non-commercial. gpt-oss-120b, Qwen2.5-32B and Whisper are Apache 2.0. No code, weights or generated data were public on 8 October 2026, so a commercial team would rebuild the pipeline with a commercially licensed TTS and real-speech corpora, and the accent gap would still need real dialect recordings.