Voice agent demos usually go well. Not because the product is good, but because the demo runs under controlled conditions: a quiet room, prepared questions, a known scenario.
Real calls are not like that. Customers call from a car, cut their sentences short, change topic, get angry. The eight tests below bring a demo closer to a real call and show you what the architecture is without asking the vendor.
Running all of them takes five minutes. Run the same eight tests in the same order on every system you evaluate; only then is the comparison meaningful.
Test 1: Cut it off
After the agent starts talking, interrupt mid-sentence. "Hold on, what I actually wanted to ask was this."
What to watch: Does it stop, how quickly, and does it take account of what you said after stopping?
What it means: If it does not stop, or stops and resumes its old answer, interruption is being handled by a separate mechanism. In a well-built system this is close to instant.
Test 2: Go silent mid-sentence
Start a sentence, pause for two seconds in the middle, then continue. "I placed an order last week... [two seconds] ...but it hasn't arrived."
What to watch: Does the agent start talking during the pause?
What it means: If it does, end-of-speech detection is tuned aggressively. That is not an architecture problem on its own, but it is the behavior that generates the most complaints in the field and it needs fixing.
Test 3: Ask a short question
Ask a three-word question like "are you open?" and note how long the answer takes. Then ask a long question and compare.
What to watch: Does the short question come back noticeably faster?
What it means: If the wait is the same for both, there is a fixed processing cost — usually the fixed step cost of a chained architecture.
Test 4: Change your tone
Say the same sentence once neutrally and once with irritation. "This is my third call."
What to watch: Does the agent's answer change? Does an apology or a handoff behavior kick in on the irritated version?
What it means: If the answer is word-for-word identical in both cases, speech may be going through transcription and losing tone.
Test 5: Step outside the knowledge base
Ask something you are confident is not in the system. Ideally realistic but specific: "what time does your downtown branch close on Saturday?"
What to watch: Does it invent an answer, say it does not know, or hand off?
What it means: This test measures setup quality more than architecture, but it is the most critical one. An agent that invents answers gives customers wrong information in production, and that costs more than the other seven tests combined.
Test 6: Change topic midway
While talking about one thing, suddenly ask about another. In the middle of an order question: "by the way, can I get my invoice by email?"
What to watch: Can it move to the new topic and then return to the old one?
What it means: Systems bound to a rigid flowchart get stuck here and push the customer back to the previous topic. That is chatbot logic carried onto a voice channel.
Test 7: Call from a noisy place
Call from an open office, the street or a car. If possible, have someone else talking in the background.
What to watch: Can it follow the conversation? Does it mistake background speech for an interruption and go quiet?
What it means: A significant share of real calls come from noisy environments. An agent that is perfect in a quiet room can fall apart here.
Test 8: Ask for the post-call output
After the call, ask the vendor to show you the recording, transcript and summary. Compare the summary against what you remember.
What to watch: Is there a transcript, is the summary accurate, is the outcome tag sensible, what was written to the CRM?
What it means: This test measures what the product contributes to operations. An agent that speaks beautifully but leaves no data behind is an investment you cannot measure.
Scoring
Almost no system passes all eight; the goal is not to find a flawless one but to decide knowing where it is weak.
Our suggested weighting: make test 5 (invention) and test 8 (post-call output) pass-or-fail — a system that fails either should not go live no matter how it performs elsewhere. The remaining six can be scored.
While running the tests, listen to your own ears rather than the vendor's explanation. "That behavior is configurable" may well be true; in that case, ask to see it configured.
If you would like to run these tests against our system, you can have it call you from the demo page. We covered the architecture side in the 3 August post and how to measure latency in the 4 August post.