スマホの指がエコーキャンセラを壊していた — 音でないもので発話が切れる

📝 Originally published (in Japanese) at forge.workstyle.tech.

The Avatar that Stopped Working on iPhone Safari

The voice dialogue avatar that worked fine on PC had an issue on iPhone Safari.

The avatar seems to be reacting to its own speech.
It also seems to be reacting when the speaker's volume changes while operating the smartphone.

What the Logs Showed

Looking at the judgment records, a pattern emerged.

13:39:52.843  Speech start detected (VAD)
13:39:52.848  Bot stopped speaking   ← The avatar was stopped 5 milliseconds later

All five instances had the same pattern. The decisive factor was that no transcription was output.

The VAD judged that "speech has started" and stopped the avatar's speech as an interrupt. However, the sound was not recognized as a word, and no turn was taken. The avatar was speaking and being stopped repeatedly at 2-second intervals.

Mechanism

The user's guess was " sudden decrease in volume". To clarify further, when you cover the speaker with your finger, the sound's loudness and resonance change. The browser's echo canceller learns the relationship between the sound sent to the speaker and the sound that comes back to the microphone, so when that relationship changes suddenly, it can't cancel it out, and the residual sound leaks out.

The VAD sees this as human speech. The reason there was no transcription output was that it wasn't a word.

The reason it didn't happen on PC was not because the countermeasures were effective, but because the browser's AEC was canceling out the echo during playback.

The Gate I Created Doesn't Work for This Symptom

I had already implemented a gate as a countermeasure against self-echo, which drops speech immediately after the sound finishes. However, this doesn't work. By design, it lets through speech during playback to prevent interrupting the speech.

Self-echo    … After the sound finishes (AEC tail) → Gate's responsibility
Misinterrupt … During playback                   → Outside gate's responsibility

Echo and misinterrupts occur at different times and require different countermeasures.

What the Industry is Doing

I investigated how web services that keep the microphone open for conversation handle this issue.

LiveKit's publicly available numbers were specific. They don't trigger an interrupt immediately, but look at the first few hundred milliseconds of speech (the rise, sustain, and rhythm of the waveform) before making a judgment.

Listen to 216ms of audio on average before interrupting
This rejects 51% of VAD-based interrupts
The judgment model's inference itself takes less than 30ms

In other words, delaying the judgment by 200-300ms to examine the sound's characteristics was the answer.

On the other hand, no one has a silver bullet for echo itself. Deepgram's official documentation only says "leave it to the browser's standard" and doesn't mention mobile or speakerphones. Depending on the client-side AEC is a common premise.

Implementation: Making VAD Less Sensitive During Playback

Since all the misfires observed were short sounds (no transcription output), I used the duration to filter them out.

During the avatar's speech, increase the continuous time required to recognize speech from 0.2 seconds to 0.5 seconds
Don't change the normal time ＝ The response time for turns remains unchanged. Only the interrupt becomes less sensitive

The framework I was using had a mechanism for interrupt strategy, but waiting for the transcription to be ready would delay the interrupt by about 1.6 seconds (1.2 seconds of silence waiting + speech recognition). This contradicts the requirement for responsiveness, so I didn't adopt it.

⚠️ The VAD parameter setting API couldn't be used. Because it initializes the internal state, calling it while the user is interrupting and speaking will cut off the speech. Moreover, the timing you want to call it (when the avatar's speech stops) is precisely that moment. I rewrote it to directly overwrite the threshold frame count.

⚠️ That attribute doesn't exist immediately after creating the object. It's born when the sampling rate is decided (when the pipeline starts). I was testing with a fake object, so I didn't notice, and it crashed the moment I added the test item with the real object. Until then, I was suppressing the exception and doing nothing.

Result

                    Before countermeasure    After countermeasure
Misinterrupts during playback    5 instances      0 instances
Self-echo disposal      0 instances      1 instance (speech 7ms after playback ended)
Real speech           ―        3 instances all passed (5.0 seconds / 10.5 seconds / 17.2 seconds later)

I also left evidence of the countermeasure being effective in the logs.

Misinterrupt countermeasure: Increase speech judgment time during playback from 0.2 seconds to 0.5 seconds

This line not appearing means the countermeasure is not working. Since I was once deceived by this, I made sure to present both the evidence of the countermeasure working and the fact that it didn't work.

Generalizable Points

"Sound input" and "speech" are different events, and VAD can only judge the former.

VAD cuts out speech intervals based on sound energy and characteristics. It can't determine if a person intentionally spoke. Therefore, coughs, noise, machine contact sounds, and echo residuals all become "speech".

There are two directions for countermeasures.

Cut by time (ignore sounds that don't continue for a certain time) — deterministic, light, and delayed by several hundred milliseconds
Cut by characteristics (look at the waveform or rhythm) — high accuracy, requires a model

The difficulty level also changes depending on the environment. Desktop PCs with headsets are easy, while handheld smartphones with speakers are difficult. The device moves, the way it's held changes, and the speaker is covered with a finger — the echo canceller's premise that "the sound path is constant" doesn't hold for mobile devices.

"Since it worked on PC, it's fine" doesn't apply to voice.