Dubbed anime characters break speaker verification, and the encoder KyaraBench ships to fix it is non-commercial

Speaker verification error multiplies 2.66 times when a dub swaps the actor under one character

KyaraBench asks one question two ways. Do two clips belong to the same fictional character, when the performer changes (the original Japanese voice against the English dub), and when the performer stays the same but voices two different characters in two shows? Speaker verification is built to answer the opposite question, so it fails in both directions. In Table 3 of arXiv:2610.06013, ECAPA-TDNN trained on VoxCeleb scores 0.138 EER on same-language trials (MONO) and 0.367 on cross-dub trials (XLING), a factor of 2.66 by our division. ReDimNet-b2 goes from 0.156 to 0.383 (2.46 times), WavLM-base+ SV from 0.201 to 0.314.

KyaraEmbed, a 20.8M-parameter ECAPA fine-tuned for 1,500 steps, holds MONO at 0.155 and brings XLING down to 0.215, 31.5% below the best zero-shot speaker model (WavLM, 0.314) and 20.7% below the anime-domain model it starts from (0.271). On cross-dub retrieval against a 53-character gallery it reaches R@1 0.325, which is 17 times the 1-in-53 chance rate of 0.019. The same-actor trap (MONO-VA) costs ECAPA 0.167 EER over MONO and KyaraEmbed 0.052.

Two shortcut rows matter for anyone tempted by a cheaper fix. Pitch statistics alone score 0.418 on XLING and fall to chance (0.477) when impostors are pitch-matched to within two semitones. The audio tower of Qwen3-Omni, a general multimodal model, is worse than pitch on MONO (0.387). Our judgement: neither a voice "type" heuristic nor a general audio LLM embedding is a usable character check today.

What Spellbrush published on 5 October 2026, and what actually landed in the release

arXiv:2610.06013v1, Character Identity is not Speaker Identity: KyaraBench and KyaraEmbed for Character Verification, by Joonyong Park and Jerry Li (Spellbrush, CA, USA), was submitted on 5 October 2026 as a 5-page paper. It builds the benchmark from Anim-400K (Cai, Liu and Chan, January 2024), a corpus of aligned Japanese and English anime dub pairs whose speaker labels are episode-local and name no characters.

Construction (Section 3.1, Table 1): every clip is restored with Sidon, episode-level speaker groups are clustered into show-level identities with a public anime speaker embedder, a YOLOv8 anime-face detector plus the CCIP character image encoder names clusters from the video, Whisper and UTMOS gates drop noisy clips, and a human listening audit keeps clips whose voice matches the pictured character. The conditions are MONO (1,778 positives, 5,327 impostors), XLING (646 / 1,812), MONO-VA (309 shared-actor impostors), two four-clip bundled variants XLING-K and XLING-KP, and a 424-query retrieval task. A human reference comes from 48 native Japanese listeners on 320 verification trials and 30 listeners on 200 same-actor trials: 78 participants in all.

The paper says the benchmark manifest, evaluation code and model "are publicly available". On 6 October 2026 github.com/sizigi/kyarabench returned HTTP 200 with 2 commits (v1.0 at 2026-10-01T02:26Z, a licence fix at 15:20Z the same day), 27 tree entries, 1 star, 0 forks. The checkpoint is at huggingface.co/spellbrush/kyaraembed: kyaraembed_v1.0.pt, 83,125,959 bytes, which at 4 bytes per float32 weight is 20.78M parameters and matches the stated 20.8M. No audio ships, by design.

The paper's numbers checked against its own manifest, licences and tables

Paper values from arXiv:2610.06013v1. Counts, sums and release checks are BLOMEGA computations from the GitHub and Hugging Face APIs, 6 October 2026.

The strongest check passes. The paper says about 79% of positives in each bundled condition contain at least one aligned Japanese to English segment pair (the same line, spoken once by each actor), while single-clip XLING excludes them. Counting directly from the shipped manifest gives 79.1% and 79.2%, and exactly 0 for XLING. That matters because an aligned pair shares script, timing and emotion, which helps any model. The paper reports KyaraEmbed at 0.173 on XLING-K positives without such a pair and 0.155 with one. Our reading: the single-clip XLING column is the cleaner cross-performer number, and the bundled columns are partly a same-line test.

The funnel does not match. Paper Table 1 reports 1,828 utterances and 3,656 clips after the quality gates, and the audit tallies in Section 3.1 sum to exactly 1,828. The README and the manifest's own meta.funnel both say 1,772 and 3,544, 56 utterances fewer. Both agree on the 986 released. Either the release was regenerated after the paper's counts were frozen or one of them is a typo; the paper does not say which.

The licence moved after release. Commit 4cf0090 (v1.0, 1 October 2026, 02:26 UTC) shipped a LICENSE-DATA stating that "the model checkpoint ... is MIT licensed separately". Commit 6453727, 12 hours 54 minutes later, rewrote that line to CC BY-NC 4.0 and added LICENSE-MODEL, which explains the reason: Anim-400K is research-only and the tri-jek corpus is non-commercial. The Hugging Face card already carried cc-by-nc-4.0. Anyone who copied the repository in that window has a contradictory licence pair.

How a voice cluster becomes a benchmark character, and which parts you can download

The construction is aggressive. Of 9,399 show-level clusters, 85 survive, 0.9%. Of 436,887 aligned utterances, 986 are released, 0.23%. The human audit is the step a localization data team should study: even after face detection, portrait matching with 75% agreement, a cast list and automatic pitch and two-speaker filters, listeners rejected 772 of 1,828 clips (42.2%) as not the pictured character, and 70 more as undecidable. The paper reports that this audit roughly halves MONO error, while leaving XLING within its interval.

One circularity is worth naming as a judgement. The clustering embedder (footnote 2) and the two-speaker repair embedder (two_speaker_repair.json, field embedder: "anime_speaker_embedding variant=char") are the same public model that KyaraEmbed is fine-tuned from. Its model card describes the char variant as "trained to guess character voices, not voice actors; eager to distinguish speakers (even two characters with the same voice actor)". So the "anime domain FT" row is a character-supervised model, the benchmark's candidate identities were proposed by it, and the human audit filters purity but cannot add identities that model never grouped. Our reading of Section 3.1 is that XLING is the least exposed to this, because the English side of each identity enters through the corpus's line alignment rather than through voice similarity.

Model selection also ran on the benchmark. Section 5.4 says KyaraEmbed was chosen among 31 fine-tuned configurations "based on its joint ranking on XLING, MONO-VA, and R@1 on KyaraBench"; there is no held-out development split. In Table 5 the λ=1.0 variant beats the chosen λ=0.3 on MONO (0.151 vs 0.155), MONO-VA (0.205 vs 0.207) and XLING-K (0.156 vs 0.159), ties XLING at 0.215, and loses on XLING-KP by 0.001 and on R@1 (0.297 vs 0.325). The choice rests mostly on retrieval. Training data, the paper states, shares no clips, episodes, shows or identities with the benchmark; 64 of the 85 benchmark identities come from Anim-400K's own train split, 12 from val, 9 from test.

What character identity diverging from actor identity means for dubbing and voice data teams

KyaraEmbed's same-actor penalty is 0.020 against 0.087 for human listeners and 0.200 for ECAPA. Read the columns, though. Of the 0.180 by which its penalty is smaller than ECAPA's, 0.140 comes from rejecting more same-actor impostors (0.620 to 0.760) and 0.040 from rejecting fewer pitch-matched different-actor impostors (0.820 to 0.780). Humans beat it on both. In the same table, on 320 verification items, humans have a balanced error of 0.226 on MONO where the anime model gets 0.107 and KyaraEmbed 0.143, but 0.318 on XLING where KyaraEmbed gets 0.214. Machines already out-hear these listeners within one language; across a dub the anime-domain model (0.232) and KyaraEmbed (0.214) do too, and ECAPA (0.375) does not.

• Do not use a speaker-verification score as a dub QC gate for character consistency. At 0.367 EER cross-dub, ECAPA would flag more than a third of correct recasts at its equal-error threshold. Those systems are measuring the actor, and a dub is supposed to change the actor.
• Separate the two identities in your metadata. KyaraBench keeps va_jp, va_en and char as separate fields; 12 of its 71 Japanese voice actors play more than one benchmark character. A voice dataset that stores only a speaker id cannot support character-consistent TTS, recasting or voice-match search, and cannot be relabelled cheaply later: the audit here rejected 42.2% of machine-labelled clips.
• Budget for the rights, not the model. The checkpoint is non-commercial because its training audio is. LICENSE-MODEL names Anim-400K and tri-jek; it does not mention that the base model was trained on a visual-novel voice dataset, or under what terms the 91-hour internal set (86 characters, each voiced in four languages by four actors) was recorded. Our judgement: a studio that wants this capability in production needs its own consented same-character, multi-actor, multi-language recordings, which is the data shape the authors had to build internally.
• Treat bundled scores as optimistic. Four-clip bundles with an aligned line in 79% of positives are easier than real recast checks, where the new actor rarely reads the same line in the same scene.
• Watch the loader. embedders/kyaraembed.py loads the base char model and only swaps in KyaraEmbed weights if KYARAEMBED_CKPT is set. Forget the variable and you silently score the domain baseline. The loader also truncates every clip to 10 seconds and does not apply the 95 two-speaker trim spans, while kyarabench_v1.0_source_clips.json says to "trim the restored wav ... before embedding".

For the voice-cloning side of the same tension, see our note on the identity trade-off in cross-lingual voice cloning, and for how much a dubbed minute costs when the actor does change, the cost of a dubbed minute in 2026.

Check it yourself

Release state and licence history, no account needed:

curl -s https://api.github.com/repos/sizigi/kyarabench | grep -E '"(created_at|pushed_at|size|stargazers_count)"'
curl -s "https://api.github.com/repos/sizigi/kyarabench/commits?per_page=100" | grep -E '"(date|message)"'
curl -s https://api.github.com/repos/sizigi/kyarabench/commits/6453727 | grep -o 'MIT licensed separately'
curl -s https://huggingface.co/api/models/spellbrush/kyaraembed/tree/main | grep -E 'path|"size"'
# 6 Oct 2026: 2 commits, kyaraembed_v1.0.pt 83125959 bytes, card license cc-by-nc-4.0
curl -s https://huggingface.co/api/datasets/davidchan/anim400k | grep -o '"gated":"[a-z]*"'

The manifest counts (aligned-pair share, trial totals, funnel, actor pairs):

curl -sLO https://raw.githubusercontent.com/sizigi/kyarabench/main/data/kyarabench_v1.0_manifest.json
python3 - <<'PY'
import json
m=json.load(open('kyarabench_v1.0_manifest.json'))
print(m['meta']['funnel']['coverage and quality gates'])   # {'identities':114,'utts':1772,'clips':3544}
pairs={(u['jp'],u['en']) for v in m['identities'].values() for u in v['utts'] if u['en']}
B=lambda x: x if isinstance(x,list) else [x]
for k in ['XLING','XLING_K','XLING_KP']:
    P=[t for t in m['trials'][k] if t['label']==1]
    n=sum(any((a,b) in pairs or (b,a) in pairs for a in B(t['a']) for b in B(t['b'])) for t in P)
    print(k,len(P),n,round(n/len(P),3))                # 646 0 / 636 503 0.791 / 648 513 0.792
pos=sum(t['label']==1 for k in ['MONO','MONO_EN'] for t in m['trials'][k]); print('Mono +',pos)  # 1778
print('same-VA pairs',len(m['same_va_pairs']),'impostors',len(m['trials']['MONO_VA']))         # 16, 309
PY

The paper's tables: curl -sL https://arxiv.org/pdf/2610.06013 -o p.pdf, then Table 1 and the audit tallies are on page 2, Tables 3 and 4 on page 3, Table 5 on page 4. Running score.py needs the audio: sign the Anim-400K gate, download anim400k_audio_clips/*, run prepare/restore.py (CUDA only, about an hour on one GPU per the repo), then KYARAEMBED_CKPT=kyaraembed_v1.0.pt python score.py --clips restored --embedder embedders.kyaraembed. The README warns restoration is not bit-deterministic, so expect small differences from Table 3.

What would prove this reading wrong

Our claims are that the cross-dub gap is real and large for speaker models, that KyaraEmbed's single-clip XLING result is the honest headline, and that the bundled numbers are flattered by aligned lines.

Prediction: by 30 June 2027, an independent run of the released score.py on restored Anim-400K audio will put KyaraEmbed's XLING EER within 0.02 of 0.215 and ECAPA's XLING / MONO ratio above 2. A third-party XLING EER above 0.235 for KyaraEmbed, or an ECAPA ratio below 2, would show that restoration noise, the untrimmed two-speaker clips or the 1,828 versus 1,772 gap changes the result. Separately, if a re-scoring of XLING-K restricted to the 133 positives without an aligned pair gives KyaraEmbed an EER no worse than its full-set 0.159, the aligned-line advantage we describe is negligible. We also expect that no commercially licensed character-verification encoder with a KyaraBench score will be public by that date.

Sources

1. Joonyong Park, Jerry Li (Spellbrush). Character Identity is not Speaker Identity: KyaraBench and KyaraEmbed for Character Verification. arXiv:2610.06013v1, submitted 5 October 2026. Tables 1 to 5, Sections 3 to 5, acknowledgments.
2. github.com/sizigi/kyarabench: README, LICENSE, LICENSE-DATA, LICENSE-MODEL, NOTICE, data/ manifests, kyarabench/audio.py, embedders/kyaraembed.py. Commits 4cf0090 and 6453727, 1 October 2026; checked 6 October 2026.
3. spellbrush/kyaraembed on Hugging Face: created 30 September 2026, last modified 1 October 2026, licence cc-by-nc-4.0; checked 6 October 2026.
4. Kevin Cai, Chonghua Liu, David M. Chan. Anim-400K: A Large-Scale Dataset for Automated End-to-End Dubbing of Video. arXiv:2401.05314, 10 January 2024; ICASSP 2024. Dataset and access terms: davidchan/anim400k (gated, TOU and NDA).
5. litagin. anime_speaker_embedding_ecapa_tdnn_groupnorm, model card, licence MIT, last modified 22 June 2025; base model of KyaraEmbed and the paper's clustering embedder.
6. Shinnosuke Takamichi. tri-jek corpus, trilingual single-speaker corpus used for the language-alignment term.
7. BLOMEGA computations, 6 October 2026: aligned-pair shares, trial and duplicate counts, actor-pair and split counts from the v1.0 manifest; XLING / MONO ratios, relative reductions and the Table 4 decomposition from the paper's tables; licence history from the GitHub commits API.

FAQ

Can I use KyaraEmbed to check character voice consistency in a commercial dub?

Not under its licence. The checkpoint on Hugging Face (spellbrush/kyaraembed) is CC BY-NC 4.0 because its training audio includes Anim-400K, distributed for non-commercial research only, and the non-commercial tri-jek corpus. The code (MIT) and the benchmark trial lists (CC BY 4.0) are usable commercially, but the audio they point to sits behind Anim-400K's terms of use and NDA.

How badly do speaker-verification models fail on dubbed characters?

On KyaraBench (arXiv:2610.06013, Table 3), ECAPA-TDNN trained on VoxCeleb goes from 0.138 equal error rate on same-language trials to 0.367 when a Japanese clip is compared with the English dub of the same character, 2.66 times worse. ReDimNet-b2 goes from 0.156 to 0.383 and WavLM-base+ SV from 0.201 to 0.314. KyaraEmbed scores 0.155 and 0.215.

Is the KyaraBench release complete and consistent with the paper?

Mostly. The trial counts in Table 2 and the paper's "approximately 79%" aligned-pair share reproduce exactly from the shipped manifest (503 of 636 and 513 of 648). Two gaps remain as of 6 October 2026: the manifest's funnel lists 1,772 utterances after the quality gates where the paper says 1,828, and the reference loader does not apply the 95 two-speaker trim spans that the clip list says to apply before embedding.