What Is Real-Time Noise Cancellation for Voice Agents, and How Does It Work

Arctan Eigen — TL;DR: Real-time noise cancellation processes incoming audio in short, consecutive chunks while a caller is speaking. Eigen fits into this streaming part of the voice-agent pipeline: processed audio can reach speech-to-text (STT) and turn detection without waiting for a complete recording. Its value in a given application depends on task results, speech preservation, and response delay.

The term covers several different operations. Noise suppression reduces non-speech sound, voice isolation emphasizes an intended speaker over competing voices, and echo cancellation reduces the return of played audio into the microphone. The WebRTC Audio Processing Module treats noise suppression and echo cancellation as separate processing effects. LiveKit likewise distinguishes background noise suppression from voice isolation.

How does streaming cleanup fit into a voice-agent pipeline?

A typical path is audio capture → optional processing → STT or speech understanding → turn detection and response generation. The processor receives each frame, estimates which parts to retain or attenuate, and returns audio for the next stage. It must preserve frame timing and a compatible audio format. If it buffers multiple frames before returning output, that wait belongs in the latency budget even if the processing operation itself is fast.

The effect is broader than a transcript. Unwanted sound may affect turn detection, the decision about when the caller has finished speaking and the agent should reply. LiveKit’s turn-taking documentation places audio preprocessing alongside endpointing and interruption handling because the stages interact. Check quiet interruptions as well as transcription: a cleaner signal can help in some conditions, while an overaggressive filter can hide a caller’s words.

What makes it “real time”?

The meaningful measure is whether the whole conversation remains responsive. Track at least three clocks: frame or processor delay, time to a stable transcript or end-of-turn decision, and time from the caller’s last word to the agent’s first audible response. These clocks include different work. A per-frame processing figure does not establish end-to-end conversational latency.

Frame size also matters: Google’s streaming STT guidance notes that larger audio frames can be efficient but add latency. Choose a size supported by the audio stack, then measure buffering and response timing under realistic load rather than inferring performance from a single benchmark.

When should you use suppression, isolation, or neither?

Choose by the call’s intended transcript, not by a feature label. For more on the multi-speaker case, see the comparison of voice isolation and noise cancellation.

How should you evaluate a candidate?

Use consented recordings that represent your actual microphones, languages, codecs, and acoustic conditions. Keep a baseline with no added processing. Listen for speech loss, compare checked transcripts and key business fields, and inspect false turns and interruptions. Measure delay on the same hardware and deployment path you plan to use. Google warns that preprocessing can reduce recognition accuracy for its service, so a clean-sounding sample is insufficient evidence of an STT gain.

For a Python agent, the integration quickstart explains the insertion and bypass decisions. The STT diagnosis guide explains what to measure.

Written by the Arctan team.