I would judge a voice agent by how it handles a correction, not just how quickly it starts speaking. A smooth reply is not enough if the system mishears an order number or reports an action it never completed. Voxtral 3 makes the voice pipeline easier to connect, but those handoffs still deserve separate checks.
This guide explains Mistral’s Voxtral 3 speech models and Voice Agents SDK through a simple order-support example: how audio becomes text, when a turn is complete, where tools execute, and how migration rules and billing differ between integration paths.
Start with the four jobs inside a spoken conversation
Speech to text (STT) turns recorded sound into written words. A large language model (LLM) interprets the conversation and prepares an answer or a tool request. A tool is a declared function or connected service that can retrieve information or carry out an operation. Text to speech (TTS) turns the answer into audio.
An SDK, or software development kit, supplies code for connecting these pieces. Mistral’s Voice Agents SDK builds on LiveKit for realtime audio transport. In the documented hosted path, the models run at Mistral while the application captures and plays audio. Packaging them as one pipeline does not make transcription, reasoning and speech generation one interchangeable model.
Figure 1 separates the application’s microphone and speaker from the hosted stages. Read the arrows as data dependencies, not as measured waiting time.
Mistral introduced Voxtral 3 on October 20. The audio overview now distinguishes three transcription models and two speech-generation modes/models, including realtime TTS. Therefore, the family should not be described as an immutable list of four endpoints. Choose a component for the job: a recorded meeting, a live transcript and a two-way agent need different handling.
Hearing the right words is different from knowing the turn is over
Imagine a fictional customer saying, “Where is order Orchid four… seven two?” The pause could be a search for the remaining digits, not permission to answer about order four. Turn detection estimates when the speaker has finished a conversational turn. Diarization labels who spoke. These solve different problems: a speaker label does not authenticate a customer or determine whether a request is complete.
The raw turn-detection interface distinguishes speech start, a tentative end, resumed speech and a confirmed end. A tentative end can support preparation that can be discarded; it is not an appropriate trigger for an irreversible operation. The last transcript words can also arrive after the turn-end event, so raw-stream applications must finish collecting the turn’s text before acting. The SDK handles tentative-end events internally.
Figure 2 shows why the order number must remain open during the pause. Its waveform is conceptual; its horizontal position does not represent milliseconds or an accuracy measurement.
Context biasing supplies likely spellings such as “Orchid” to the transcriber. It makes a term more likely, not guaranteed, and does not correct an invalid order number. Mistral’s vocabulary guide separates this input-side spelling hint from output-side pronunciation hints. A voice can pronounce “Orchid” beautifully while still reading the wrong order status.
A useful test therefore checks exact identifiers, not just fluent transcripts. Include pauses between digits, similar product names and a caller correcting a number. If an identifier is uncertain, ask for confirmation before querying a sensitive record.
Keep a reusable agent separate from one live session
A saved agent is a versioned configuration: instructions, chosen models, voice, tool declarations and conversation policy. A session is one running conversation using that configuration. Mistral’s configuration guide supports pinning an agent version and overriding options for one session without changing the saved agent.
Figure 3 follows one saved configuration into separate conversations. The example version number illustrates reuse; it does not mean that callers share conversation state.
For an order-support pilot, save the tested configuration and keep new instructions in a separate version until the correction cases still pass. Record the version used for each test. Otherwise a changed prompt, voice or tool declaration can make two results look like a model comparison when the configuration changed too.
A stored tool declaration is only the interface description. Mistral’s tool guide says the handler remains in application code and must be attached when the session starts. A declaration named get_order_status does not itself provide database access or prove that a lookup ran. A missing handler can produce an unhandled_tool event.
A browser tool is not automatically a telephone tool
In a browser or SDK application, a client-side handler can call an authenticated backend. In a direct phone call, no equivalent application handler is running for the caller. SIP is the protocol used to connect telephone systems; WebRTC carries realtime media in browser-based experiences. Neither transport supplies business permission checks by itself.
Figure 4 contrasts the execution surfaces. The distinction matters before moving a successful browser demonstration onto a phone number.
The telephony guide currently describes enterprise, non-self-serve inbound calling with server-side tools. It also says call transfer to a person is not supported there. Do not promise outbound calls or a live transfer just because the launch description is broader. Design an honest callback or follow-up path in the surrounding business system instead.
For the fictional Orchid lookup, the backend should verify the caller’s identity and record access, query the matching order, then return its actual state. If it reports manual_review_required, the agent should explain that review is needed, not claim that a replacement shipped. A lifecycle webhook that reports a call ending is not automatically a supported synchronous tool callback; implement the actual tool path for the deployment surface.
If the caller interrupts with “I meant Orchid 479,” treat that as a correction to conversational state. Stopping speech does not prove that an in-flight backend operation was cancelled. For operations that change records, track completion separately and make repeated requests safe to retry without duplicate changes.
Measure useful response time, not just the first sound
Mistral’s announcement promotes sub-second voice turns. That is a vendor claim, not a promise for every network, tool and language combination. The latency guide separates first transcript, first audio and per-turn waiting time. A short “I’m checking” filler may make silence less awkward while a tool runs, but it does not shorten the lookup or make the answer available sooner.
For the support example, record when the caller finishes, when the identifier is complete, when the tool returns and when the first useful answer is heard. Include interruptions and slow backend responses. Report typical and slow-tail results with sample counts, and count wrong identifiers and premature cut-offs alongside speed.
Raw realtime transcription has a configurable streaming delay; shorter waits provide less following-audio context. Start from an explicitly recorded setting, then change one variable at a time. Do not transfer a transcription benchmark score or a listening-preference rank into a claim about correct bookings, authorization or complete-call reliability.
Migration rules belong to the endpoint you are using
The Voxtral 2 migration guide lists concrete breaking cases: requesting word timestamps now returns an error, and offline transcription uses segment timestamps. Preserve overlapping speaker segments rather than assuming they cannot overlap. For realtime transcription, diarization and turn detection cannot be enabled together in one session; select the behavior required for that stream.
Figure 5 marks the configuration boundary of a raw transcription stream. It is not a diagram of every Voice Agents option becoming permanently locked.
In a raw realtime transcription connection, configure the session before sending audio and inspect session.updated. The following message fragment follows the documented field structure; it assumes that the incoming audio really is mono, 16-bit PCM at 24 kHz. It configures transcription, not a complete voice agent:
{
"type": "session.update",
"session": {
"audio_format": {"encoding": "pcm_s16le", "sample_rate": 24000},
"target_streaming_delay_ms": 480,
"language": "en",
"context_bias": ["Orchid"],
"turn_detection": {}
}
}The empty turn-detection object selects server defaults; it is not an enabled/mode schema. After the first audio chunk, raw transcription settings cannot change without opening a new session. By contrast, the Voice Agents configuration guide describes updates to live-session instructions, tools and turn settings. Keep those two scopes separate.
Two launch-document differences also deserve a narrow interpretation. The Mini transcription card shows 32K context, while the audio overview says 65,536 tokens; this guide therefore does not size a recording from either unconfirmed ceiling. The TTS model card says nine languages, while the announcement advertises more than 35. Confirm the exact endpoint, locale and account availability before committing to those limits. These differences do not change the pipeline explanation above.
Choose the billing path before adding component prices
The October 21 standard API rate table lists Mini file transcription at $0.0035 per audio minute, Small file transcription at $0.004, realtime transcription at $0.008, and TTS at $13 per million characters. These units are different: a minute of incoming audio is not a minute of generated speech, and a character count is not an LLM token count.
Separately, the launch announcement lists the Voice Agent API at $0.04/min with Mistral Small 4 for reasoning and $0.08/min with another reasoning model. Do not automatically add that managed rate to a self-assembled STT-plus-TTS estimate. Confirm billable duration, included services, rounding and carrier charges for the chosen account and route.
Figure 6 uses assumed usage to show the arithmetic, not to claim that the two paths have identical coverage or measured cost.
For individually metered endpoints, five realtime transcription minutes cost $0.04; 1,000 TTS characters cost $0.013. Their $0.053 subtotal excludes the language model, transport, tools, hosting and review. Five assumed billable managed-agent minutes give $0.20 or $0.40 at the announced rates, before resolving any additional applicable charges. Neither calculation is an observed invoice or a universal price for a five-minute call.
The operational measure is total relevant spending divided by correctly completed calls. A cheaper conversation that needs a second call, manual correction or duplicate-action recovery can cost more overall. Define what “completed” means before using that denominator.
A small release plan that tests the actual failure boundaries
- Start read-only. Test status lookups and factual questions before enabling cancellations, payments or account edits.
- Test hearing and speaking separately. Check identifiers in transcripts, then listen for the correct numbers, dates and names in the answer. Use only voices and recordings you are authorized to use.
- Exercise correction and failure. Include a long pause, a changed order number, a missing handler, an unauthorized lookup, a slow service and a failed tool result. Verify that spoken claims match the execution record.
- Repeat on the target surface. Browser success is not phone-path validation. Confirm the available server-side tools and the documented follow-up limits.
- Pin and observe. Preserve the tested configuration, record timing and errors without unnecessary personal data, check retention settings for each service, and keep a rollback path.
Practical questions
Is Voxtral 3 one speech-to-speech model?
The documented SDK connects transcription, a general language model, tools and speech generation. Evaluate each handoff instead of treating the product name as one model identifier.
Does speaking over the agent undo its tool call?
Not as an application guarantee. Conversation interruption and backend cancellation are separate concerns; check the operation’s actual status before retrying or announcing success.
When does a custom pipeline make sense?
When you need control over transport, model choices, where audio is processed or turn handling that the managed path does not provide. That flexibility also makes you responsible for the connections, measurement and recovery behavior.
Official technical sources
- Voxtral 3 announcement and audio overview — product positioning and model families.
- Voice Agents overview, configuration, tools and telephony — execution and deployment boundaries.
- Turn detection, realtime transcription, custom vocabulary and latency concepts — timing, transcript and hint behavior.
- Migration guide, STT modes, TTS modes, Mini transcription card, TTS card and API pricing — compatibility, documented discrepancies and rates checked October 21, 2026.