What your AI feature does when the model provider isn't
A client runs an inventory reconciliation tool. They paste two ledger exports and the app surfaces likely-matched rows using an embedding model. When the model provider returns 429 for the fifth time in a minute, the app shows a spinner, then a blank results panel, then a support email. The user reconciles by hand.
The embedding call is the product. When it fails, the product fails. We've stopped letting that be the end of the story.
The first thing we define for any model-backed feature is its degraded state — what the feature returns when the model is unavailable, rate-limited, or slow enough that the user has left the room. This is a design decision, not an incident response. We write it down before the feature ships, in the same doc where we define the happy path.
Most model-backed features have a non-model fallback that is worse but tolerable. The reconciliation tool can fall back to exact key matching on invoice number and amount — deterministic, no embeddings, catches the sixty percent of rows that match cleanly. A summarization feature can fall back to showing the first three sentences. A classification feature can default to the most common class and flag the result as low-confidence. None of these are as good as the model, but they keep the user working.
The cost question is what you're willing to spend on retries before you fall back. A retry with exponential backoff costs a request slot and maybe a user's patience. We set a hard ceiling: three retries with jittered backoff, total wait under four seconds, then fall back. If the provider is genuinely down, three retries just burn time. If it's a transient blip, the third usually catches. The ceiling matters because without it the retry loop becomes a queue that backs up across users.
For the inventory tool the fallback path lives in the same endpoint that handles the model path. The caller doesn't know which ran.
async def reconcile(paste_a: str, paste_b: str) -> Result:
try:
return await embed_and_match(paste_a, paste_b)
except (RateLimitError, APIConnectionError) as e:
log.warning("model unavailable", error=type(e).__name__)
return exact_match_fallback(paste_a, paste_b)The fallback result carries a flag — `method: "exact"` instead of `method: "embedding"`. The UI shows it as a badge: "Exact match only — some rows may be missed." That sentence does two things. It sets the user's expectation and it tells them when to come back and retry.
Explainability and fallback are the same problem from opposite directions. When the model works, the user wants to know why it matched two rows. When the model is down, the user wants to know what they're looking at instead. Both require you to label the output with its provenance. A result that says "matched on invoice number and amount, 100% confidence" is explainable and is also the fallback. A result that says "matched on semantic similarity, 87%" is the model path. The label is the contract.
Where we won't add a fallback is a feature whose entire value is the model's judgment. If the user is asking for a draft of a legal clause, we don't fall back to a template — we return a clear message that generation is temporarily unavailable and cache the last successful response for that input if it exists. A bad draft is worse than no draft.
Monitoring catches the transition between paths. We log every fallback invocation with the upstream error type and the time since the last successful model call. A sustained spike — say, fallback rate above twenty percent for ten minutes — pages us. That threshold is low because we'd rather investigate a transient provider issue than discover at the standup that the model has been down since 2 AM and nobody noticed.
The inventory tool's degraded mode runs about forty times a month. The provider has a mid-afternoon rate window where the client's usage pattern consistently trips the limit. That's not an outage; it's capacity planning we haven't done yet. The fallback absorbs it until we decide whether to raise the rate limit or shift the workload.