Model Health Is a Signal, Not a Switch: Building Better Provider Recovery

RFC 9110 HTTP Semantics provides the foundation for interpreting methods, statuses, and retry-related semantics. Application contracts add another layer. Consult each provider's documented error model before deciding that a particular response proves an outage or authorizes a retry. A routing controller should not invent that contract from a status number alone.

Combine passive observations with representative probes

Passive observations describe real workload behavior without generating additional requests. They are valuable during normal operation, but become scarce after traffic has been diverted. They are also selective: if only easy requests remain on a route, their success says little about the harder requests moved elsewhere.

Active probes fill that gap. Use small, non-sensitive fixtures that exercise the affected capability and validate the expected response contract. For image editing, a text-only prompt is not representative. A fixture should include a permitted reference image and check that a usable image artifact is returned, not merely that an acknowledgment arrives.

Probes consume money and provider quota. Give them independent concurrency limits, deadlines, and a spending allowance, and keep them out of customer billing. Do not send private customer content through a probe simply because it reproduces an incident. Curated fixtures improve repeatability, although they still cannot represent every production prompt or asset.

Associate delayed completions with the attempt and configuration that produced them. Otherwise a request started before isolation can finish afterward and appear to prove a successful recovery trial. Accurate timestamps and attempt identities prevent this mistake without requiring perfectly synchronized clocks for every decision.

Make recovery a state machine

A useful controller distinguishes eligible, suspect, isolated, and trial states. These names are illustrative. Each state needs allowed actions, entry evidence, exit criteria, and a fallback when observations are missing. Persist transitions so a process restart does not accidentally restore unrestricted traffic to an isolated route.

In suspect state, reduce exposure while collecting evidence. Isolation stops ordinary dispatch but may permit bounded probes. Trial state admits a controlled amount of compatible production work. Eligibility returns only when the trial satisfies the agreed observation window and sample requirements. A successful probe opens a trial; it does not erase the incident.

Use hysteresis: the evidence needed to restore traffic should differ from the evidence that reduced it. Require a minimum observation count as well as a time window, and place a maximum concurrency limit on trials. Fixed waiting periods alone cannot distinguish sustained recovery from a route that simply received no requests.

Low-volume routes may never accumulate strong statistical evidence. Keep their status explicitly uncertain and choose a conservative exposure policy or manual approval. Do not report high confidence merely because a timer expired. Thresholds should follow business impact and actual workload data, rather than copying unexplained constants from another service.

Recover capacity gradually

Suppose reference-image editing fails while text drafting remains stable. The controller isolates only the affected editing route and records its configuration revision. A representative probe later succeeds. The controller then admits a small, bounded editing trial, measuring completion, contract validity, latency, and cost before expanding eligibility.

Concurrency matters more than a percentage when total traffic changes suddenly. A small percentage of a large burst can overwhelm a recently recovered provider. Combine traffic allocation with an absolute in-flight ceiling and paced admission. Do not release the entire accumulated backlog as soon as the route becomes eligible again.

Multiple application instances need coordinated limits or carefully partitioned allowances. Otherwise each replica may independently launch what it considers a small trial. Coordination adds operational complexity; partitioning is simpler but may waste capacity when replicas are unevenly loaded. Document the choice and test behavior during scaling and controller restarts.

Keep fallback inside the product contract

An available fallback is not necessarily compatible. Check supported operations, reference-asset handling, output format, data residency, safety requirements, and authorized cost before dispatch. A route that produces a different kind of output may be technically successful but still fail the customer's task. Compatibility must be established before an incident.

In the creative application, the backup model might generate new images but not preserve the subject of a reference edit. Silently substituting it would change the product contract. Better choices include queueing within a deadline, returning a specific temporary limitation, or requesting approval for a different operation. Availability does not outrank correctness by default.

Recovery traffic also competes with retry traffic. The Amazon Builders' Library guidance on timeouts, retries, and backoff with jitter explains why uncoordinated retries can amplify an overloaded dependency. Keep probe, retry, and normal dispatch allowances visible together. Separate controllers must not independently spend the same capacity budget.

Preserve operator judgment without losing history

Automation needs an override for credential incidents, suspected data-handling problems, and failures that probes cannot detect. An override should record its scope, reason, owner, creation time, and review deadline. Its precedence over automated decisions must be explicit. An operator quarantine should not disappear because a background probe succeeds.

Choose expiry behavior by risk. A temporary traffic cap might expire automatically, while a security quarantine may require affirmative release. Alert before an override becomes stale. Record what evidence was available when a decision occurred so reviewers can evaluate the controller's reasoning without reconstructing it from scattered screenshots.

Diagnostic records should exclude tokens, raw credentials, and unnecessary customer prompts. The OWASP Logging Cheat Sheet offers guidance on sensitive-data handling and useful security events. Recovery needs enough context to explain a decision, not an unrestricted archive of everything sent upstream.

Recovery review checklist

• Does each health record identify the operation, configuration, and observation age?
• Are validation, authorization, policy, transport, and output failures distinguished?
• Do probes exercise the capability that actually failed?
• Are probes isolated from customer billing and private production content?
• Can a successful probe start only a bounded trial rather than full restoration?
• Do all replicas respect the intended trial and provider concurrency limits?
• Are fallback compatibility, residency, and cost checked before routing?
• Can operators explain, override, and review every recovery transition?

Better recovery is not a faster red-to-green switch. It is a sequence of bounded decisions whose evidence matches the workload, whose uncertainty remains visible, and whose mistakes cannot immediately restore the full incident blast radius.