How to Remove Background Voices from Voice Bot Calls Without Losing the Caller

TL;DR: If a bot should hear one caller but its input also contains a television or nearby conversation, test primary-speaker isolation on the incoming track. Confirm with real calls that the caller stays audible, especially during overlap, quiet speech, and speaker changes. If the task requires two speakers, preserve both rather than filtering one out.

Another person's speech is a different challenge from traffic or a fan. A general background noise processor may reduce non-speech sounds while leaving competing words in the stream. LiveKit distinguishes background noise suppression from voice isolation for this reason. The difference matters when a bot extracts account details, order items, or a caller's request from a mixed recording.

When is background voice removal the right tool?

Start with the job the bot is meant to perform. For a single-caller support bot, a nearby coworker's conversation is unwanted input. A voice isolation step can emphasize the caller before audio reaches STT and turn detection. We designed Eigen's primary-speaker isolation for this case: keeping the primary speaker prominent when multiple voices are present.

For a family call, interpreter session, or shared-device workflow, a second voice may be part of the task. Suppressing it could remove essential information. LiveKit recommends background noise suppression rather than voice isolation for multi-speaker rooms. If speakers arrive on separate channels, preserve that separation; Google's STT guidance recommends sending separate speaker channels separately when available.

How do you keep the intended caller instead of the loudest voice?

“Primary speaker” is not always the person the bot should serve. A television may be louder than a distant caller, or someone else may speak first. Test the actual selection behavior rather than assuming the model knows the caller's identity. Include cases where:

• The caller speaks softly while someone nearby speaks loudly.

• Two people talk at once.

• The caller moves between handset, speakerphone, and headset.

• The intended speaker changes during a transfer or handoff.

• Background voices use the same language as the caller or a different one.

We describe Eigen's processing as language agnostic. That describes the operating approach, not a promise of equal isolation quality for every language, accent, microphone, or overlap pattern. Check those conditions in your own evaluation set.

Where should isolation run in a live call?

Place the candidate processor on the inbound audio path before the bot's STT and turn-handling stages. Keep the original stream available during evaluation and as a bypass if processing fails or removes caller speech. Do not apply the same single-speaker filter to an agent-and-caller mix when the application needs both sides of the conversation.

Measure any added delay on the full path. We specify Eigen processing latency below 10 ms, but network transport, buffering, transcription, and reply generation contribute to the delay a caller feels. The integration guide shows the audio insertion point; the real-time processing guide explains what to time. We also offer an on-premise deployment option. If audio must stay within a particular boundary, confirm the package, data flow, and infrastructure requirements with our team before deployment.

How can you tell whether background voices were removed safely?

Build a consented test set from calls with known speaker roles. For each clip, compare the original and isolated audio using the same STT model and settings. Check three outcomes: words from background speakers entering the bot's transcript, intended-caller words lost or altered, and false turns or interruptions caused by competing voices. Listen to every regression, particularly where people overlap.

Keep the processed path only for the call types where those results improve. You can use our 14-day trial for that comparison, but test the speakers and acoustic conditions your bot actually handles. For the underlying distinction between suppression and isolation, see voice isolation vs. noise cancellation.