Why Background Noise Breaks Speech-to-Text Accuracy (and How to Fix It)

Most speech-to-text accuracy complaints are not actually about the STT model. They are about what the model is being asked to transcribe. Feed a modern STT engine clean, close-mic audio, and accuracy is generally strong. Feed it audio with background chatter, road noise, or overlapping speakers, and word error rates climb quickly, no matter how good the underlying model is. If you're chasing transcription accuracy by swapping STT providers, you may be solving the wrong problem.

Where the accuracy loss actually happens

Speech-to-text models are trained to separate speech from silence and, to some degree, from noise. But real-world noisy calls (contact centers, drive-thru windows, mobile field agents, in-vehicle voice assistants) push far past what most STT training data anticipates. The result shows up in a few predictable ways:

• Words get dropped entirely when background noise masks quieter syllables

• Background speech gets transcribed as if it were the caller talking

• Confidence scores drop, which cascades into worse intent detection and slot-filling downstream

• Call summaries and structured extraction (order details, account numbers, dates) become unreliable even when the overall transcript looks “close enough”

That last point is often the most costly. A transcript that is 90 percent accurate can still produce a completely wrong order summary if the 10 percent that got garbled happened to be the part number or the callback time.

Fixing it before the audio reaches STT

Rather than trying to make an STT model more noise-tolerant after the fact, the more reliable fix is to remove the noise before the audio ever reaches the transcription step. This is the same principle behind noise-cancelling headsets, applied to the call audio pipeline instead of a human ear.

Real-time noise cancellation, applied inline in the audio pipeline, strips out non-speech background noise and isolates the primary speaker before the audio is handed to whichever STT engine you use. Because this happens upstream of transcription, it works regardless of which STT provider is in your stack. It is a pipeline fix, not a vendor swap.

Arctan's Eigen is built for exactly this position in the pipeline: real-time processing under 10 milliseconds of added latency, language-agnostic operation, and primary-speaker isolation for calls where multiple voices are present. Because the added delay is small enough not to be noticeable in a live call, it can sit directly in front of any STT engine without changing the perceived responsiveness of the voice agent.

Where in the pipeline to insert it

For most voice agent architectures, the right insertion point is immediately after audio capture and before it reaches the STT engine, whether that is a cloud STT API or a self-hosted model. Client-side integration (inside the calling application or SDK) keeps processing close to the audio source and avoids sending noisy audio over the network at all. Server-side integration works well when the calling client cannot be modified, at the cost of sending unprocessed audio one extra hop before cleanup.

Measuring the improvement

Before rolling a change like this into production, it is worth measuring the effect directly rather than assuming it. Take a representative sample of your noisiest calls, run them through your STT pipeline as-is, then run the same calls through noise cancellation first and re-transcribe. Comparing word error rate and, more importantly, accuracy on the specific structured fields your business depends on (names, numbers, dates) will usually make the case clearly.

If your team has ruled out prompt and model tuning and is still seeing inconsistent transcription accuracy specifically on noisy calls, the audio going into the model, not the model itself, is the most likely place to look next.