Most voice AI agents do the same job from the outside: they pick up, they talk, they answer. From the inside there are two different architectures, and the difference between them determines how the conversation feels.
This post goes through both in detail — not in marketing terms, but by following the path the signal takes.
The chained architecture: three models back to back
The common setup has three separate components:
- STT (Speech to Text). Incoming audio is converted to text.
- LLM (Language model). The text goes to a model, which produces a written answer.
- TTS (Text to Speech). The written answer is synthesized into audio and played to the customer.
There is logic to this structure: each component is mature in its own domain, they can be swapped independently, and it is relatively easy to assemble. For a long time this was the only way to build a voice agent.
But it has three problems that come from the nature of a chain.
First, every link adds its own delay. You wait for audio to become text, you wait for the model to produce an answer, you wait for speech to be synthesized. On top of that, in most setups TTS cannot start playing before the sentence is complete.
Second, the moment audio becomes text, part of the information is erased. Text does not carry everything speech carries. The word "fine" is a single word in text; in speech it can be agreement, hesitation, resentment or sarcasm. The text reaching the model contains none of that.
Third, the chain is one-directional. Once the model starts speaking, an interruption from the other side has to reach the model, generation has to be stopped and the new input processed — that requires a separate mechanism. Even when that mechanism works, a gap opens up.
Voice-to-voice: one model, audio directly
In a voice-to-voice model there is no intermediate conversion. The model takes audio input directly and produces audio output directly. Text stops being a mandatory stop on the conversation's route.
Three consequences follow:
Conversion delays disappear. Instead of three steps to wait on there is a single operation. Measured on our own phone line, the model composes its answer in 0.65 seconds — 659 and 624 milliseconds across two independent measurements. That figure is the model's composition time; the total wait the customer experiences also includes audio travelling in both directions.
The information carried in speech is preserved. Intonation, emphasis, pace, pauses and hesitation reach the model. Whether a customer saying "yes" is certain or reluctant is a distinction that disappears in transcription and survives in audio.
The output is produced as audio too. The model does not write its answer and then have it voiced; it speaks directly. Where the emphasis falls in a sentence and where it pauses are part of generation itself.
How the difference shows up in conversation
The technical difference has four visible consequences on the customer's side.
Being able to stop when interrupted. In human conversation, interruption is frequent and natural. When a customer cuts in with "yes but my question is," the agent needs to stop and listen. In a voice-to-voice architecture the incoming audio is already being processed continuously.
Meeting a natural pause with patience. People stop mid-sentence, think, and continue. Chained systems often read that pause as "sentence finished" and talk over the customer. That is the behavior that generates the most complaints in the field.
Carrying intonation. An answer sounding conversational rather than robotic is not merely pleasant — it determines how long the customer keeps listening.
Delivering short answers fast. Running a short answer like "yes, your parcel has shipped" through three conversion steps takes longer than the answer itself.
The part of voice-to-voice that is not easy
To be honest, this architecture has costs too, and you should know them when deciding.
You do not get intermediate text for free. In a chained system the STT output is already in your hands as a transcript. In voice-to-voice, the transcript has to be produced separately. For businesses that need recording, archiving and quality review that cannot be skipped; we produce transcripts in a separate flow.
You lose component-swapping flexibility. In a chain, if you dislike your TTS provider you replace it. In a single model there is no such separation.
Debugging is different. In a chain it is relatively easy to see which link failed. In a single model, finding the source of a wrong answer means working through the knowledge base and conversation rules.
These three are solvable problems, and what you get in return — a conversation that actually behaves like a conversation — more than covers that cost in most scenarios. But it would be wrong to call it superior in every respect; an architecture is a choice, and choices have costs.
How to tell which architecture you are using
You can work out what your current system is without asking the vendor. Three quick tests:
- Cut it off mid-sentence. Does it stop, or finish its sentence?
- Go silent for two seconds mid-sentence. Does it talk over you?
- Ask a short question. Something like "are you open?" Is the delay the same as on long questions?
We cover these and five more in detail in the 6 August post.
If you would like to hear how the voice agent talks, you can have it call you from the demo page.