"Our voice agent's latency is X milliseconds" means nothing on its own. Latency is not a single event but the sum of several stacked steps, and the number a vendor gives you usually measures only one of them.
This post breaks down the wait the customer experiences. The goal is to let you ask the right question when evaluating a voice agent: which step does that number measure?
The six steps that make up the wait
After the customer finishes their sentence, here is what happens in order before they hear an answer:
1. Detecting that speech has ended. The system has to notice the customer stopped. This is harder than it sounds: people also stop mid-sentence. Decide too fast and you cut the customer off; decide too slowly and you create dead air. In the field this is frequently the largest share of the wait, and it is more a matter of tuning than architecture.
2. Audio reaching the server. Network latency. It rises noticeably on mobile data and international calls. This is the one line nobody fully controls.
3. Processing the input. In a chained architecture this step is transcription. In voice-to-voice it does not exist as a separate step; audio goes straight to the model.
4. Generating the answer. The model deciding what to say and producing it. Measured on our own phone line, this step takes 0.65 seconds — 659 and 624 milliseconds across two independent measurements.
5. Converting the answer to speech. A separate step in a chained architecture, and in most setups it cannot start before the sentence is complete. In voice-to-voice, generation is already audio; there is no separate conversion.
6. Audio reaching the customer and starting to play. Network again, plus audio buffering.
The total wait is the sum of those six. If a vendor says "our latency is 400 milliseconds," the question to ask is: which step? It is usually step four, and that is a small part of what the customer experiences.
Why the chain charges extra
In a chained architecture, steps three and five exist separately and each adds its own wait. But the real issue is not the number of steps — it is that they are sequential.
In a chain, step five cannot start before step four finishes; you need text before you can synthesize speech. Setups that start early on partial text improve this somewhat, but audio generation may have to be undone when the sentence structure changes.
In voice-to-voice, generation and voicing are the same operation. That is not skipping a step; it is collapsing two steps into one.
You cannot improve what you do not measure
You do not need an expensive tool to measure the wait in your own system. Two methods:
Measuring from recordings. Open a call recording in an audio editor. Measure the gap between the customer falling silent and the agent starting to speak. Take the median across a fifteen-turn call — the median, not the mean, because a single long turn distorts the mean.
Measuring from logs. If your system stamps time per step, the job is easy. If it does not, make it: the moment end-of-speech is detected, the moment input goes to the model, the moment the first audio packet leaves. Those three stamps show you which of the six steps is costing you time.
Follow three rules when measuring:
- State the channel. Phone, WhatsApp call and web conversation produce different numbers. A measurement without a channel cannot be compared.
- State the turn type. A short answer and an answer requiring a knowledge base lookup are not the same.
- State the sample size. A number derived from three turns does not show intraday variation.
Ways to cut the wait on the setup side
There is time to be won without changing architecture. The four items we run into most:
Shorten knowledge base answers. A long answer both lengthens generation and tires the customer. One paragraph per question works well on a voice channel.
Remove unnecessary verification steps. Chains like "may I take your first name, thank you, and your surname" produce a wait on every turn. Do not ask unless you need it.
Calibrate end-of-speech detection. Too aggressive and you cut the customer off; too loose and you get dead air. Calibrate to your sector: people pause longer on technical support calls and shorter on order calls.
Review the tool calls in the loop. If the agent runs a CRM query on every turn, that query's duration is added directly to the wait. Pulling frequently used data once at the start of the call is usually faster.
What threshold is good enough
Rather than a hard number, a practical criterion: the wait must not exceed the point where the person on the other end starts wondering whether the call dropped. On a phone call that threshold is quite short, and a wait filled with something feels shorter than an empty one.
That is why some systems use a bridging phrase like "let me check that for you" while waiting. It does not solve latency but it corrects perception — a legitimate technique when an operation will genuinely take time. Used on every turn, it makes the conversation feel artificial.
We cover how to score voice agent quality call by call in the 10 August post, and the demo tests that reveal architecture in the 6 August post. To hear the agent talk for yourself, you can have it call you from the demo page.