Lip-sync TTS scores 96% on voice activity, but a dub that never pauses scores 83%
96.21% frame-level voice-activity accuracy is the headline of AppTek's lip-sync TTS paper, posted on 22 September 2026 and accepted at Interspeech 2026, up from 72.69% without its voice-activity mask. We ran Silero VAD on the paper's demo audio: speech fills 83.4% of the source frames, so a dub that never pauses would score 83.36%, above the unconditioned model on the same clips (79.53%). Measured on silent frames alone, the mask lifts silence recall from 36.3% to 88.6%, a bigger effect than the paper's own metric shows, bought at a WER rise from 8.5% to 13.9%.
What did AppTek publish on 22 September 2026?
Alejandro Pérez-González-de-Martos, Florian Lux and four colleagues at AppTek GmbH posted arXiv:2609.26486 on 22 September 2026 (v1, 14:24 UTC), accepted at Interspeech 2026. The model is an F5-TTS-style flow-matching synthesizer that takes one extra input: a per-frame bit saying whether the source-language clip is voiced or silent. The bit comes from Silero VAD, so no video, face tracking or lip encoder is involved.
Evaluation uses 291 clips from Multilingual TEDx, balanced across Greek, French, Portuguese and Russian, each 7 to 15 seconds long with at least one pause over 500 ms. The English output is scored three ways: frame agreement with the source's voice activity (Table 1), a 40-rater Prolific listening test on 25 clips (Table 2), and ASR word error rate plus TTSDS scores (Table 3). A demo page carries 8 of the 25 listening-test clips and 8 constructed examples.
The paper frames the mask as optional: training randomly hides it, so an editor can enforce the source timing, relax it, or unmask only the frames around a pause.
What the paper reports, next to what we measured on its demo audio
| Measure | LibriTTS, no mask | LibriTTS, VAD mask | Multilingual, no mask | Multilingual, VAD mask | Source |
|---|---|---|---|---|---|
| Silero frame accuracy, 291 clips | 72.69% | 96.21% | 71.35% | 91.59% | Paper Table 1 |
| Pause placement MOS (1 to 5) | 3.80 ± 0.98 | 3.74 ± 1.04 | 3.82 ± 1.02 | 3.73 ± 1.05 | Paper Table 2 |
| Prosody MOS (1 to 5) | 3.71 ± 1.04 | 3.68 ± 1.14 | 3.81 ± 1.03 | 3.68 ± 1.09 | Paper Table 2 |
| WER, proprietary ASR | 8.5% | 13.9% | 6.3% | 8.3% | Paper Table 3 |
| TTSDS Intelligibility | 76.82 | 80.59 | 78.90 | 79.72 | Paper Table 3 |
| TTSDS Prosody | 85.00 | 86.98 | 82.96 | 86.32 | Paper Table 3 |
| Onset/offset error in seconds | not reported as a number | not reported as a number | not reported as a number | not reported as a number | Paper Figure 4 (density plot only) |
| Silero frame accuracy, 8 demo clips | 79.53% | 97.60% | not on the demo page | 94.07% | Our run, 24 Sep 2026 |
| Silence recall, 8 demo clips | 36.3% | 88.6% | not on the demo page | 80.0% | Our run, 24 Sep 2026 |
| Always-voiced output, 8 demo clips | 83.36% frame accuracy (speech share of the source frames) | Our run, 24 Sep 2026 | |||
Paper: arXiv:2609.26486 v1, Tables 1 to 3. Our rows: Silero VAD (pip package silero-vad, default thresholds, 100 ms minimum silence) on the 32 subjective-evaluation MP3s from the demo page, resampled to 16 kHz, compared frame by frame at 10 ms.
Two things in that table need a closer look. The no-mask rows are described in the paper as "overlap by chance", and the listening test is described as 40 ratings per cell. Both descriptions understate what the numbers can carry, in opposite directions.
How does a one-bit mask move pauses in a translated sentence?
The design choice that makes this work is the fixed-length canvas. Inpainting TTS generates a whole clip of a given length at once, so a mask computed on the French clip lines up frame for frame with the English one. We confirmed this on the demo page: all 24 generated clips match the duration of their source within 0.04 s (ffprobe, MP3 container).
The model decides where inside each voiced island the words go. Section 4 reports that it learned to tie pauses to commas and periods, speeds up or slows down to reach a punctuation mark near a silence, and past a limit drops, repeats or reorders words rather than move the pause. The demo's constructed examples show exactly that, which we check below.
Our finding: a dub that never pauses beats the unconditioned model on frame accuracy
Frame accuracy counts agreement on every 10 ms frame, voiced or silent. TEDx speech is mostly voiced. On the 8 listening-test clips on the demo page (90.13 s of source audio), Silero marks 83.4% of source frames as speech. So an output that is voiced on every frame, the sound of a narrator who never draws breath, scores 83.36% frame accuracy on those clips. The unconditioned LibriTTS model scores 79.53% on the same 8 clips, and the paper's own 291-clip figure is 72.69%.
The paper's own numbers point the same way. If two masks overlapped purely by chance and each were voiced a share p of the time, agreement would be p² + (1 − p)². Setting that equal to the reported 72.69% gives p = 83.7%; for 71.35% it gives p = 82.7% (our algebra). Those implied speech shares match the 83.4% we measured, so an always-voiced output would likely beat the "chance" figure across the full 291 clips too. That is our inference; the paper does not report the speech share of its test set.
This does not make the method weak. It makes frame accuracy the wrong headline. What a viewer notices is voice over a closed mouth, so the telling number is silence recall: of the source frames that are silent, how many stay silent in the dub.
Measured this way, the mask is worth far more than the headline says. Without it the LibriTTS model keeps 36.3% of the 14.96 s of source silence silent and talks over 9.53 s of it. With it, silence recall rises to 88.6% (1.71 s of talking over silence), and the multilingual model reaches 80.0% (2.99 s). On three of the eight clips (one Portuguese, two Russian) the unconditioned model scores 0.0% silence recall, and a fourth scores 1%. Frame accuracy compresses a 36-to-89 swing into 80-to-98.
Two cautions, both ours. The 8 clips were chosen by the authors for the demo page and their VAD-model accuracy (97.60%) runs above the paper's 96.21%, so they are probably easier than average. And the paper does not state its Silero thresholds; with a 300 ms minimum silence instead of 100 ms, our numbers move to 81.93% (no mask), 97.78% (mask) and 84.29% (always voiced), and the ordering does not change.
Is the listening test 40 ratings per cell, or 480?
Table 2's caption says "Each score is based on 40 ratings." Section 3.4 and Figure 5 say 40 participants each rated 48 clips (12 references in four configurations) on two scales, for 3,840 ratings "split uniformly across all systems and conditions". 3,840 over 4 systems and 2 scales is 480 ratings per cell, 40 raters times 12 clips each. The caption is reporting raters, not ratings.
That matters for the "not statistically significant" claim. With 480 ratings per cell and the reported standard deviations, the standard error of a difference is about 0.07 MOS, so the smallest drop the test could reliably detect (80% power, two-sided 5%) is about 0.19 MOS (our arithmetic: 2.8 times the standard error). The multilingual model's prosody drop, 3.81 to 3.68, gives z = 1.90 and p = 0.058 under a normal approximation that ignores rater clustering. That is a borderline result, not evidence of no effect. Had there really been 40 ratings per cell, detectable drops would have been over 0.6 MOS and every comparison would be uninformative.
Do the Table 1 totals add up?
| Source language | LibriTTS no mask | LibriTTS VAD | Multilingual no mask | Multilingual VAD | Source |
|---|---|---|---|---|---|
| Greek | 66.95% | 96.06% | 66.06% | 91.28% | Paper Table 1 |
| French | 68.65% | 96.09% | 68.80% | 92.21% | Paper Table 1 |
| Portuguese | 75.17% | 95.97% | 73.06% | 91.67% | Paper Table 1 |
| Russian | 81.17% | 96.74% | 78.50% | 91.14% | Paper Table 1 |
| Total | 72.69% | 96.21% | 71.35% | 91.59% | Paper Table 1 |
| Unweighted mean of the four rows | 72.99% | 96.22% | 71.61% | 91.58% | Our arithmetic |
| Reported Total minus that mean | −0.30 | −0.01 | −0.26 | +0.02 | Our arithmetic |
The two VAD columns equal the unweighted mean of the four languages to within 0.02. The two no-mask columns sit 0.30 and 0.26 below it. With 291 clips split 73/73/73/72 a clip-weighted mean would land at 72.96% and 71.58%, not the reported 72.69% and 71.35%. The most likely explanation is frame-level pooling with longer Greek and French clips, which would weight the two lowest no-mask languages more. The paper does not say. The gap is small, but it means the headline "roughly 73%" and the per-language rows were computed differently.
What the constructed demo examples show
The demo page's second table uses one reference clip with one pause and feeds it English texts with a comma after word 4, 5, 6 or 7 of a 10-word count. The authors mark the word-5 and word-6 versions as passes and word-4 and word-7 as failures. We ran Silero on all of them.
All six one-pause outputs are 5.24 s long, against a 5.20 s reference whose pause runs from 1.8 s to 3.7 s. Every output, including the two marked failures, holds a pause starting at 1.9 or 2.0 s and ending at 3.6 s. That matches Section 4: when the comma is too far from the silence, the timing survives and the words give way. For a QA pass this means a VAD check will not catch these failures. Only a transcript check (ASR against the script) will.
The two-pause examples behave differently. The reference is 3.95 s long, yet the failure output is 4.20 s and the "adapted text" success is 4.92 s, with its last voiced segment ending at 4.9 s. Either the reference on the page is trimmed, or those two outputs were not synthesized on a canvas of the reference's length. The page does not say which, and the adapted example, labelled a success, is 0.97 s (25%) longer than the clip it is supposed to match.
What does one point of lip-sync cost in intelligibility?
Table 3 lets you price the mask. Dividing the WER increase by the frame-accuracy gain gives the cost of each point of sync. Our arithmetic:
| Model | Frame accuracy gained | WER change | WER per point of accuracy | TTSDS Intelligibility change | Prosody MOS change | Source |
|---|---|---|---|---|---|---|
| LibriTTS | +23.52 pts (72.69 to 96.21) | +5.4 pts (8.5 to 13.9), +63.5% relative | 0.230 | +3.77 | −0.03 | Paper Tables 1 to 3, our arithmetic |
| Multilingual | +20.24 pts (71.35 to 91.59) | +2.0 pts (6.3 to 8.3), +31.7% relative | 0.099 | +0.82 | −0.13 | Paper Tables 1 to 3, our arithmetic |
The paper calls the LibriTTS WER change "a slight increase". A move from 8.5% to 13.9% is a 63.5% relative rise, from roughly 1 wrong word in 12 to 1 in 7. That is our judgement of "slight", not the paper's. The multilingual model pays less than half as much per point of sync (0.099 against 0.230), which is the stronger argument for the method.
The two robustness metrics also disagree with each other. WER rises with the mask, while TTSDS Intelligibility, which is meant to track the same property, also rises (76.82 to 80.59). TTSDS scores distributional similarity to real speech, so a clip can look more speech-like while saying the wrong words. For a dubbing buyer, the ASR number is the one tied to the script.
Decisions this supports, in our reading:
- Pick the metric before the vendor. Ask for silence recall, or voice-over-silence seconds, per clip. A frame-accuracy number near 80% can come from a model that ignores every pause.
- Budget script adaptation, not just synthesis. Section 4 ties the WER cost to unadapted translations, and Brannon et al. (TACL 2023), cited in the paper, find that source and dub pauses align only about 12% of the time in real films. The mask shifts the work to isochronous MT or a human adapter.
- Run ASR as the gate. The failures on the demo page keep perfect timing and drop or reorder words. Timing checks pass them.
- Use the optional mask. Unmasking frames near a pause, which the random-masking training allows, is the cheap fix when a comma sits one word off.
For how visual lip-sync is being approached from the other side, by altering the picture, see our note on Prime Video's visual dubbing. For what a synthetic dubbed minute costs, see the cost of a dubbed minute.
Check it yourself
The demo audio is public. This reproduces our demo-page numbers in about a minute on a laptop CPU.
B=https://anondemos.github.io/NotQuiteMyTempo/resources/audios
for id in el-el_1078_BbYgaeEuDXM el-el_40_RYIDcSD5PNI el-el_585_eFJy2q6F89Q fr-fr_1018_rgVjL8UBkWc \
pt-pt_1044_gMWud6IZQ3A pt-pt_344_MwRGd5YN9oo ru-ru_1044_g-p8QH574uk ru-ru_1843_MOwGqmuJ8lU; do
for s in sources libritts_no_vad libritts_vad multi_vad; do
mkdir -p $s; curl -sL $B/$s/$id.mp3 -o $s/$id.mp3; done; done
pip install silero-vad librosa
python3 - <<'PY'
import glob, os, numpy as np, librosa, torch
from silero_vad import load_silero_vad, get_speech_timestamps
m = load_silero_vad()
def mask(p):
y, _ = librosa.load(p, sr=16000)
ts = get_speech_timestamps(torch.from_numpy(y), m, sampling_rate=16000, return_seconds=True)
v = np.zeros(int(round(len(y) / 160)), bool)
for t in ts: v[int(t['start'] * 100):int(t['end'] * 100)] = True
return v
for sysn in ['libritts_no_vad', 'libritts_vad', 'multi_vad']:
agree = sil_kept = sil = frames = speech = 0
for src in sorted(glob.glob('sources/*.mp3')):
a = mask(src); b = mask(f'{sysn}/' + os.path.basename(src)); n = min(len(a), len(b)); a, b = a[:n], b[:n]
agree += (a == b).sum(); sil_kept += (~a & ~b).sum(); sil += (~a).sum(); frames += n; speech += a.sum()
print(sysn, 'acc %.4f silence recall %.3f always-voiced %.4f' % (agree / frames, sil_kept / sil, speech / frames))
PYExpected, to within a few tenths depending on library versions: frame accuracy 0.804 / 0.977 / 0.942; silence recall 0.363 / 0.886 / 0.800; always-voiced 0.834. The script pools frames across clips; the figures we chart (79.53%, 97.60%, 94.07%, 83.36%) are per-clip means, which differ slightly because the clips differ in length. The ordering is the same either way.
The durations: ffprobe -v error -show_entries format=duration -of csv=p=0 file on resources/qualitative/1234_567_8910.wav (3.95 s) and two_pauses_adapted.wav (4.92 s) under the same base URL. The MOS count needs only the paper: 40 raters × 48 clips × 2 scales = 3,840; divided by 4 systems and 2 scales = 480 per cell.
What would prove this wrong
Our central claim is that on the paper's 291-clip test set, speech fills more than 72.69% of source frames, so a constant always-voiced output would match or beat the reported no-mask accuracy. It is falsified if the authors, or anyone rerunning Silero on the same mTEDx subset, report a source speech share below 72.69%.
We predict that if silence recall (or any silence-only metric) on the full 291 clips is published by 30 June 2027, whether in the Interspeech camera-ready, a code release or a follow-up, it will be under 50% for the unconditioned LibriTTS model and over 80% for the conditioned one. A no-mask silence recall above 60% would mean the 8 demo clips are unrepresentative and our 36.3% overstates the problem.
On the listening test: if the camera-ready confirms 40 ratings per cell rather than 480, our power calculation is wrong and the MOS table supports even less than the paper claims.
Sources
- A. Pérez-González-de-Martos, F. Lux, A. Elizarova, M. Shkhanukova, A. Kellner, M. A. Di Gangi (AppTek GmbH). Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing. arXiv:2609.26486 v1, submitted 22 September 2026, accepted at Interspeech 2026. Sections 2 to 4, Tables 1 to 3, Figures 3 to 5.
- Demo page, anondemos.github.io/NotQuiteMyTempo: 8 listening-test clips in 3 of the 4 configurations (the multilingual no-mask system is absent) and 8 constructed examples. Downloaded 24 September 2026.
- snakers4/silero-vad, the VAD used for the mask and Table 1; pip package
silero-vad, default thresholds in our run. TEN-framework/ten-vad, one of the two extra VADs in the paper's Figure 3. - W. Brannon, Y. Virkar, B. Thompson. Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing. TACL vol. 11, 2023, pp. 419 to 435 (the 12% pause-alignment figure, as cited in the paper).
- BLOMEGA measurement, 24 September 2026: Silero VAD at 10 ms frames on all 42 audio files on the demo page; frame accuracy, silence recall, speech recall and always-voiced baseline; ffprobe durations; recomputation of Table 1 means, Table 2 cell counts and power, and Table 3 cost per point.
FAQ
What does AppTek's voice-activity TTS do?
It is a flow-matching text-to-speech model (arXiv:2609.26486, 22 September 2026) that takes a per-frame voiced or silent mask computed by Silero VAD from the source-language clip and places the translated speech only in the voiced regions, so pauses line up with the original actor's mouth. It needs no video input, and the mask can be switched off or partly masked at inference.
How accurate is the lip-sync timing?
The paper reports 96.21% frame-level agreement with the source's voice activity for its LibriTTS model and 91.59% for its multilingual model, against 72.69% and 71.35% without the mask, on 291 Multilingual TEDx clips. On the 8 demo clips we measured, the mask keeps 88.6% of source silence silent, against 36.3% without it.
What is wrong with frame accuracy as the metric?
Most frames in speech are voiced, so a dub that talks the whole time already agrees on most frames. On the demo clips speech fills 83.4% of frames, and an always-voiced output scores 83.36%, above the unconditioned model's 79.53%. Silence recall, the share of silent source frames that stay silent, separates the systems far more clearly.