Latency, Accuracy, Cost: The Trade-Offs Behind a Voice AI Pipeline

For the last few months I have been building voice AI. Agents that sit in a live call, collect audio, and send it to a model that has to return something structured and correct.

Worth saying up front what these are, because "voice AI" now covers two quite different things. Mine do not talk back. There is no reply, no text-to-speech, no interrupting the user mid-sentence. They listen, and they turn what was said into structured data: a score, a transcript, a set of fields. The audio itself arrives as a live WebRTC stream and is buffered in memory as it comes, so nothing waits for a file to finish recording and get uploaded from the device. When "upload" appears below, it means the hop between my own services, not the user's phone sending a file. From there the audio goes straight to a multimodal model, with no separate speech-to-text step in between. If you are building an assistant that holds a conversation, your hardest problem is turn-taking and barge-in, and that is a different article.

Over those months one pattern kept repeating. Every design argument, every bad result, every piece of feedback came back to the same three questions:

How long does the user wait? (latency)
How often is the answer right? (accuracy)
What does one call cost? (cost)

I did not get that list from a design document. It came out of reading the model's output again and again, checking it against what was really said in the call, benchmarking the same recording across several providers, and going through the feedback from the product side.

That is all there is. Which model, which audio format, where to run the VAD (Voice Activity Detection, the model that decides which parts of the audio are actually speech), how many DB connections to open. None of these are separate decisions. Each one just picks a point between those three.

The three questions are not the useful part. The useful part is knowing which one your product is judged on. You will give up one of them. Better that you choose which.

These three are the ones you trade against each other. They are not the only things that matter. Privacy, consent and retention on recorded audio are requirements, not dials: you do not get to spend a bit of someone's consent to buy latency. Keep them out of this trade entirely.

This is a follow-up to The Latency Challenge, which was only about the plumbing. This one is about the trade-off.

Decide this before you build anything

There is no system with zero latency, 100% accuracy and zero cost. That system does not exist. So settle this before the first line of code: write down what is good enough for your app, as numbers. Not "it should feel fast", but from the moment the user stops speaking to the moment the result is on screen, under 5 seconds. Then the same for the other two. How often does the answer have to be right before people trust it? What can one call cost before the feature stops making sense?

You write those numbers down not to know when to start optimising, but to know when to stop.

I had two agents in the same codebase, on the same audio stack, and their numbers were completely different.

The first one is live. The user speaks, and a structured result has to come back while they are still standing there. The budget was 5 seconds from the end of speech. I measured it at 3.2s, so I had under two seconds of room on speed and no room at all on being right. One wrong line and the user has to fix it by hand, which is the whole reason the feature exists.

Trade latency for accuracy, never the other way round.

So that agent runs no VAD, on purpose. Not because clipping is unsolvable: keeping a short pre-roll buffer of audio from before the trigger fires is the standard fix for a lost first syllable. It is because the window is already gated by the app. The user taps to start and taps to stop, so what gets captured is a few seconds of someone deliberately speaking, with very little dead air in it to remove. VAD would have saved a few hundred milliseconds I was not short of, and added one more thing between the user's words and the result. Not worth it here.

The second one records the full call and analyses it afterwards. Its budget is 10 seconds, but measured from the end of the call rather than the end of a sentence. That sounds generous until you look at what has to happen inside it: a call that ran for minutes has to be encoded, uploaded and read by the model before a score comes back. Here the audio size is the problem.

So that agent runs a full streaming VAD, and it runs during the call rather than at the end. Cutting the recording down to the parts with speech in them is what keeps the upload and the model read inside those 10 seconds. The same trimming I refused in the first agent is the thing that saves the second one.

Same stack, opposite answers, and both are correct. The only thing that changed was the numbers. Get those wrong at the start and you can spend months tuning an axis nobody is judging you on.

1. Latency is a chain, not a number

My first mistake was thinking about "how fast the model is". The model was never the problem. The wait was spread thin across the whole chain, and almost none of it was inference. These are the places the time actually went:

Reply first, then do the work. An upload endpoint should not hold the client while storage, the model and the database finish. Take the file, return 200, run the pipeline in the background.
Load models when the worker starts, not when the job arrives. A per-job load blocks the connection every single time. Loaded once at startup it costs 0.11s and never touches the user's clock again.
Check your cold start against the framework's startup limit. A heavy SDK import (~4.3s) can push a cold worker to 7 to 10s, just past a 10s limit. Then every worker times out, gets killed, restarts, adds load, and makes the next one slower. The service registers fine and serves zero jobs. This one is not really latency at all, which is exactly why it cost me a day: nothing in the symptoms points at startup time.
Give every external process a hard timeout. A wrapper around ffmpeg with no timeout means one stuck encoder hangs until something kills it, and the upload dies with it. Drive it over pipes with a timeout you set. You also save about 32MB of disk writes per recording.
Measure your codec settings instead of guessing. OGG compression_level 5 sounds safer than 0. Measured, it was 2x slower and produced exactly the same file.
Keep inference in the same region as your servers. Boring, but when you are counting milliseconds the speed of light is a real line item.

The one thing I did not do here is trim the audio for speed. On the post-call agent the VAD is already cutting most of the recording out, and on the live agent trimming is banned for accuracy reasons. Sending less audio is a real win, but it belongs to the cost section, not this one.

2. Accuracy: the failures that look like successes

Latency is easy to measure. You have a clock. Accuracy is where these systems lie to you, and they lie in one particular way: the wrong answer comes back in the right format. Every check below exists because something got past me first:

Check the audio before you spend the call. One recording came in completely silent. Peak level −84.3 dBFS, a dead mic. The model still returned a full, structured, passing result, with direct quotes as evidence for things nobody had said. Every field filled in, valid JSON, no error anywhere. That is worse than a crash, because a crash gets noticed. Now the audio is probed first, and if there is no speech in it the call is never made. No result, no transcript, nothing downstream.
Enforce in code whatever you asked for in the prompt. I ask for structured JSON with a response schema, and the prompt says "never invent an id". A model can ignore a prompt. So every returned item passes through a small function that drops any line whose id is not in the list I sent, and replaces an unreadable quantity with a sensible default instead of zero. A prompt is a request. Code is the guarantee.
Make silent quality loss fail a test. The prompt carries a reference list and tells the model to return nothing for an item it cannot match. Better a missing line than a wrong one. Then I cut that list down during testing so results were easy to check by eye, and never put it back. For every item outside the short list the user spoke, the agent ran, the logs showed success, and the line was simply not there. No error, no wrong value to spot, just missing. The fix was not deleting the shortcut. It was a test asserting the opposite, that the list in memory matches the file on disk, so the next person who tries it breaks the build.
Benchmark vendors on your own audio, with the judge held constant. I had to choose between sending audio straight to a multimodal model and putting a speech-to-text vendor in front of it. Price lists and public benchmarks are no help for mixed Indian languages recorded in a noisy room. So I built an endpoint that runs one real recording through several pipelines at once: audio straight to the model in one, STT → judge in the others, with the same judge in every arm, so any difference is the vendor and nothing else. Two rules kept it honest. It writes nothing, so a test run can never show up as real output or real spend. And it reports usage in units, not money, because rates belong in one place only. It took about a day to build and ended every "I think X is better at Indic languages" argument for good.

3. Cost: measure per call, price somewhere else

Cost is the one teams notice last, usually when the bill arrives. These are the things I wish had been in place from the first week:

Write a usage row for every model call. Input tokens, output tokens, thinking tokens, cached tokens, audio seconds, model name. Make it best-effort: it logs errors and never raises, because billing data must not be able to break the feature it is measuring. Without this you cannot answer which task is expensive, only that the total went up.
Store units, not money. Save the billable units and leave the cost column empty, then let one separate job price them from a rate table with dates. Vendor rates change, and two places doing the same maths will disagree sooner or later.
Pick a model per task, and keep it in an environment variable. Scoring, transcription and summarising are different jobs with different accuracy needs, so each gets its own setting. Transcription was the biggest cost in my pipeline, and moving that one task to a lite model was about 4x cheaper per call. Because it is config, undoing it if quality drops is a restart, not a release.
Send less before you optimise anything else. Trimming the silence out of a recording removes about two-thirds of the audio, which cuts cost and latency together. Same idea as a notetaker bot joining audio-only when nobody is ever going to watch the video. The three axes are not always in conflict, so look for the moves that win two before you accept a real trade-off.
Put a hard limit on anything unbounded. An audio buffer with no stop signal (app backgrounded, app crashed, packet dropped) grew at about 115 MB/hour, and would then be encoded and sent to the model in one piece. A two-minute cap turns an unlimited bill into a known one.
Count your connections against the actual grant. Not AI, same lesson. Pools sized at 8 per worker across 4 workers came to 32, over a 20-connection limit, so lookups failed and a later step quietly degraded. Open per query with a semaphore cap instead, and no idle connection holds a slot real work needs.

Final Thoughts

None of this is specific to voice. Swap audio for documents or images and the shape stays the same. Three questions, one of which you are actually judged on, and a system built around that answer.

What I would tell myself at the start of it:

Decide the three numbers first. How long the user can wait, how often the answer has to be right, what one call can cost.
Measure all three on every request, or you will be guessing about which one you are losing.
Expect wrong answers to arrive looking correct, and check for them in code.
Keep the knobs in config, so a bad decision costs a restart and not a release.

The model is the easy part. It is one API call, and everyone gets the same one. The budget, the checks, the tracking and the limits are what you actually build. That is what decides whether the model ever gets a fair chance.