A backup model is less useful if every new message first waits for the same broken primary. I would use sticky fallback to remember a working route within a conversation, while keeping that routing choice separate from service health and permission to execute tools.
This guide explains Pydantic AI’s ModelFallback(..., sticky=True) through a small support-chat example. We will follow the history that carries the choice, test what happens when the backup fails, identify the streaming boundary, and budget nested retries without assuming a cache hit or an immediate failover.
Why an ordinary fallback can repeat the same delay
Pydantic AI is a Python framework for applications that ask models for answers and may let them request tools. A primary model is the preferred starting choice. A fallback is another candidate tried after an eligible failure. A retry instead repeats an attempt at the same layer, such as another request to the same provider.
A request step is one point in an agent run that asks a model for a response. A single user turn can contain several request steps, for example when a model requests a tool and then reads its result.
Imagine a support assistant that drafts a reply, shortens it on request and then translates it. Model A is unavailable and model B can answer. Without a remembered route, each new request step can start on A again, wait for failure and then reach B. The fallback works, but the conversation repeatedly pays for the failed start.
Read Figure 1: Compare the second turn, not just the first successful answer. Sticky behavior matters when another request uses the preserved conversation history.
Here sticky means that an eligible previous fallback success influences the next default starting model. It does not mean that every user shares the same remembered outage or that B has become the permanent primary. The unit to reason about is the conversation and its recorded responses.
What changed, and which release date to record
The project’s merged sticky-fallback proposal was merged October 30, 2026. The PyPI 2.64.0 release record lists the package’s upload and release date as October 31. These are different milestones. Pin the installed version when reproducing behavior; the implementation checks for this article used pydantic-ai-slim==2.64.0, the core distribution without the full set of optional provider packages.
The feature is off by default. The ModelFallback documentation explains that sticky=True uses previous model responses to choose a working candidate for later default requests. That can avoid repeatedly starting with a failing model. It changes where the chain begins; it does not make B’s responses equivalent in quality, tools or price to A’s.
The history is evidence, not just a transcript
A model response includes more than visible text. The response-field reference describes model_id, which records the model requested for that response, and failed_attempts, which records abandoned attempts before the answer. The identifier is routing evidence, not proof of an immutable provider checkpoint or a correct answer.
The sticky feature reads the relevant response history; it does not need a global “last working model” variable and does not stamp a separate sticky flag into arbitrary message metadata. Keep the framework’s typed messages when storing history. A typed message has defined fields that can be validated and restored, rather than only a role and a text string.
Read Figure 2: Follow the response fields through storage into a new worker. Recreating an agent is compatible with this design; losing the relevant message fields is the dangerous boundary.
The message-history guide provides ModelMessagesTypeAdapter for serialization and validation. Serialization converts messages into a storable form such as JSON, a structured text format; restoration reconstructs the typed records. Save the full history needed by the documented routing logic and pass it as message_history on the next run.
This creates an application responsibility. Isolate histories by conversation and user. If two turns read the same saved version and both overwrite it, one response can disappear along with its routing evidence. Serialize turns within a conversation, or use a version check before committing a new history. Test any summary, history compactor or migration that changes these records; a plain-text summary cannot be assumed to preserve routing state.
Stay on B, then wrap around when B fails
In an ordinary A → B → C configuration, suppose A fails and B answers. Later defaults can start on B. If B later fails, the chain tries C, then wraps to A and any remaining earlier candidates. Candidates already attempted in that request step are skipped. If A answers again, subsequent defaults can start on A.
Read Figure 3: A successful B response keeps the route on B. The route back to A is triggered by the chain’s results, not by an elapsed timer.
The merged proposal explicitly has no expiry timer. A healthy primary may therefore remain unused while B continues to work. If the product needs to test A every five minutes, that is a separate recovery policy. Define who performs the probe, how much traffic it receives and how to avoid sending the same user action twice.
Explicit model choices still take precedence over the default: a per-run model=..., Agent.override or SelectModel can select the first attempt. An explicit answer alone does not initiate stickiness. Test overrides in the installed version, especially when the explicit choice equals the default or falls outside the fallback chain. Suspended-response continuations also have special rules; do not treat them as fresh ordinary requests.
Use a local test before attaching real providers
The following original example uses FunctionModel, which runs a supplied Python function in place of a remote model. It simulates one unavailable primary and one working backup, then serializes history and recreates the agent for the next turn. The testing guide documents this testing mechanism. No key for a remote application programming interface (API) or paid inference call is needed.
from pydantic_ai import Agent, ModelMessagesTypeAdapter, ModelResponse, TextPart, models
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.exceptions import ModelAPIError
from pydantic_ai.models.function import AgentInfo, FunctionModel
models.ALLOW_MODEL_REQUESTS = False # FunctionModel runs locally.
calls = []
health = {"primary": False, "backup": True}
def response(name):
def answer(messages, info: AgentInfo):
calls.append(name)
if not health[name]:
raise ModelAPIError(model_name=name, message="Simulated outage")
return ModelResponse(parts=[TextPart(f"Local reply from {name}")])
return FunctionModel(answer, model_name=name)
def new_agent(sticky=True):
return Agent(response("primary"), capabilities=[
ModelFallback(response("backup"), sticky=sticky)
])
first = new_agent().run_sync("Draft a short reply.")
assert calls == ["primary", "backup"]
encoded = ModelMessagesTypeAdapter.dump_json(first.all_messages())
restored = ModelMessagesTypeAdapter.validate_json(encoded)
calls.clear()
second = new_agent().run_sync("Make it shorter.", message_history=restored)
assert calls == ["backup"]
assert second.response.model_id is not None
print("Restored history:", calls, second.response.model_id)
With version 2.64.0, the local check produced ['primary', 'backup'] on the first turn and ['backup'] after restoration. Additional local checks made B fail while A recovered, observed ['backup', 'primary'], then observed ['primary'] on the following turn. A fresh conversation and a configuration with stickiness disabled both started on A again.
These are deterministic routing checks using local functions, not provider reliability or latency measurements. They establish that the example preserves and reuses routing history. Real-provider tests must additionally verify the output schema, tool behavior, model identifiers, privacy requirements and supported context size for every candidate.
The first stream event is the failover boundary
Streaming delivers a response through a sequence of events instead of waiting for one complete answer. ModelFallback waits for a candidate’s first event before handing its stream to the caller. An eligible failure before that handover can move to another candidate. A later stream failure is raised to the caller rather than silently joining two models’ answers.
Read Figure 4: Use the first event as the boundary. It may occur before the first readable word, so a user interface with no displayed text is not enough evidence that transparent failover remains possible.
For a partial support reply, the application can mark the response interrupted and offer a restart. If the run also created a ticket, restarting the whole job needs duplicate-action protection. An idempotency key identifies one intended action so repeated delivery can be recognized. Retrying a language-model response does not automatically make a database write idempotent.
Keep provider failure and answer rejection distinct. The default fallback condition covers ModelAPIError, an API failure reported by a model. A custom response handler can reject a successful response under an application rule. If you supply only a response handler as fallback_on, it replaces the default API-error condition; include both when both are intended. A rejected generated answer may still have consumed billable tokens.
Budget all attempts before calling failover “fast”
An API is the interface used to request a model response. A provider’s software development kit (SDK) may retry below the agent. The transport that sends Hypertext Transfer Protocol (HTTP) requests can retry below that SDK. Meanwhile, output correction can trigger another logical model request, and a durable workflow engine can restart an outer activity. These are different scopes, as the retry guide explains.
For a worked upper bound, assume two logical request steps, three model candidates per step including A, two SDK attempts for each model call and two transport attempts per SDK attempt. If every eligible budget is exercised, the bound is 2 × 3 × 2 × 2 = 24 wire attempts. Each two-attempt budget includes its initial attempt and one retry.
Read Figure 5: Multiply the explicitly bounded layers. Do not read “two retries” as “two attempts,” and include outer workflow re-entry separately.
This bound is a planning example, not a forecast. Success can stop the chain early; some errors are ineligible; a deadline can interrupt it. If the number of logical steps is unbounded, the four-factor calculation supplies no overall bound. Record actual candidate attempts, elapsed waits, corrections and any outer restarts.
The retry guide recommends choosing a retry owner. Passing model_settings={'max_retries': 0} disables provider-SDK retries where that setting is supported; support varies by provider. It does not remove the first request, network timeout, transport retries, validation corrections or durable retries. Nor does it make failover instantaneous. Confirm the settings on each candidate rather than copying a universal zero-retry prescription.
Prompt-cache continuity is an opportunity, not a promise
A prompt cache reuses qualifying previously processed input at the provider, as the caching guide explains. Remaining on one provider/model can help preserve a reusable prefix and reduce unnecessary switching. But changing models does not necessarily delete a cache, and returning to B does not guarantee a hit. Prefix contents, cache lifetime, model identity and provider policy still matter.
Measure cached input and billed usage in the actual provider records before claiming savings. Log failed-attempt usage when available, but do not turn missing usage into zero cost. The same trace should distinguish an API failure with no usable response from a generated response rejected by policy; their observable work and charges can differ.
Decide whether you need conversation or fleet recovery
A circuit breaker is a shared control that limits requests to a service judged unhealthy and can allow controlled recovery probes. Sticky conversation routing has a smaller scope. Conversation X can stay on B while a new conversation Y still starts on A. A global outage view is not implied by X’s saved history.
Read Figure 6: Compare the two conversations before looking at the fleet policy. Sharing a deployment does not make their routing histories shared state.
| Requirement | Useful control | Evidence to retain |
|---|---|---|
| Continue a chat on its working backup | Sticky fallback plus preserved typed history | Serving model ID, prior failures and restored-history test. |
| Reduce requests to an unhealthy provider across users | Shared service-health routing or circuit breaker | Health signal, scope, probe policy and traffic-return rule. |
| Bound retries and waiting | Attempt budgets at each layer and an application deadline | Actual attempt counts, waits and terminal result. |
| Accept a backup’s output or tool request | Schema validation, task checks and separate authorization | Validated result, reviewer/policy decision and action record. |
For rollout, choose a bounded draft-only task first. Exercise an unavailable A, a working B, a failed B with recovered A, an explicit override, a newly created conversation, a history round-trip and a stream failure after the first event. Verify the application-visible outcome as well as which model was called.
My adoption rule is to enable stickiness when continued use of an acceptable backup is preferable to repeated primary failures within that conversation. Keep the backup subject to the same task and action constraints. A working route is valuable evidence about execution; it is not a grant of authority or proof that the generated answer is right.
Sources and scope: Official Pydantic documentation, merged PR and package record checked November 2, 2026. Routing checks used local FunctionModel responses with pydantic-ai-slim 2.64.0 and Python 3.14.3. Diagrams, the support-chat scenario, retry arithmetic and rollout criteria are original explanatory material; no real provider outage, prompt-cache benchmark or paid model call was used.