A good support agent pays as much attention to how a customer says something as to what they say. "Fine, I understand" is positive confirmation in text. The same sentence, said with a short pause and a falling tone, usually means something else: I did not understand, but I have given up asking.
A good agent stops there and explains again. They can do that because they hear the voice, not the sentence.
In voice AI agents that distinction depends on architecture. In a system that transcribes speech to text, those two sentences are identical — because as text they genuinely are.
Four signals erased in transcription
Intonation and emphasis. "At that price?" can be surprise, objection or confirmation depending on where the emphasis falls. Transcribed, all that remains is one question.
Pauses and hesitation. A one-second gap mid-sentence signals indecision. On a sales call that is the moment an objection forms — an experienced salesperson turns straight to it.
Speaking pace. A customer speaking quickly has an urgent problem or little patience. A customer speaking slowly usually wants more detail. Delivering the same answer at the same pace to both is wrong.
Interruption. A customer cutting in is a message in itself: "what you are explaining is not my problem." In a text stream that is just two sentences placed in sequence.
Those four signals are a significant part of the information a human agent uses to manage a call. An architecture that transcribes never shows them to the model — and information lost that way cannot be recovered later.
Interruption: the most visible difference
The voice agent behavior most complained about in the field is talking over the customer, or failing to stop when interrupted.
Human conversation does not proceed in turns. People cut in the moment they understand, confirm with "yes, yes," and correct when they realize they were misunderstood. An agent can get this wrong in two ways:
It does not stop. The customer has started talking but the agent is determined to finish its sentence. Two people speak at once and the customer gets annoyed. This is the behavior that gives away a robot fastest.
It stops in the wrong place. The customer paused mid-sentence to think, and the agent reads that as its turn and starts talking. The customer never finishes their sentence.
Both come from the same root problem: the system cannot tell what a silence means. In a voice-to-voice architecture incoming audio is processed continuously, so interruption does not require a separate mechanism.
Still, let us be honest: architecture alone does not solve this, tuning does too. Background noise from a customer calling from a crowded place can be mistaken for an interruption. A well-configured system distinguishes them; a badly configured one goes quiet at every noise.
Being patient with hesitation
The most neglected setting when deploying a voice agent is how long it waits before deciding speech has ended.
Wait too briefly and the agent seems fast but cuts customers off. Wait too long and the agent is patient but dead air forms and the customer starts saying "hello?"
The right setting varies by sector:
- Technical support. The customer talks while looking at their screen and pauses at length. The wait should be long.
- Orders and reservations. Speech is fast and sentences are short. A short wait fits.
- Health and consulting. Customers pause while describing a sensitive matter. Patience here is a quality signal.
- Services with many elderly customers. The pace is slower; a standard setting causes constant interruption.
Fixing this to a single value and using it across every sector is one of the most common mistakes in voice agent deployments.
How tone shows up in the output
Preserving signal matters on the output side too, not just the input. If the model produces its answer directly as audio, where emphasis falls and where it pauses are part of generation.
In practice that means this: "unfortunately that date is full" can be said with regret or as flat information. Both carry the same information but leave a different impression. In a system converting text to speech, that choice is not encoded in the text, so it usually lands at random.
What not to expect
It is important not to overclaim here. A voice-to-voice architecture does not mean the model "understands emotions" or performs psychological analysis. The signals carried in speech reach the model and the model can behave differently because of them — that is all.
Equally, this architecture does not make a badly built agent good. If the knowledge base is weak, handoff rules are missing, or the purpose of the conversation is unclear, catching tone correctly is worth nothing. Architecture fixes the flow of the conversation; what gets said is still yours to design.
If you would like to test these behaviors with your own ears, you can have the agent call you from the demo page and run the eight tests from the next post directly.