2026-10-02

Design Circuit Breakers for AI APIs

Design Circuit Breakers for AI APIs: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits.

The operating rule

Design Circuit Breakers for AI APIs should be implemented as a reliability control contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are closed, open, half_open, failure_window.

Place circuit breakers around a concrete route and failure class, not around the entire AI feature. Count only attributable, policy-defined failures; open before the dependency consumes the caller’s deadline; probe recovery with bounded half-open traffic; and keep capacity protection separate from model-quality fallback.

Draw the failure boundary first

Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once.

Reliability controls must be defined on attempts and terminal states, not on a vague notion of a request “working.” Preserve the caller deadline, distinguish overload from permanent errors, and make rejection, cancellation, retry, fallback, and settlement observable as separate outcomes.

The owner record for this page is ai-api-circuit-breaker; its frozen controls are closed, open, half_open, failure_window. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label.

State and control contract: Design Circuit Breakers for AI APIs

Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response.

Control point	Fixed decision	Evidence to retain
`breaker_key`	`provider_route_plus_protocol_plus_failure_class`	`route_id + endpoint + normalized_error_class`
`closed_state`	`rolling_window_with_minimum_sample`	`eligible_attempts + failures + window_bounds`
`open_state`	`fail_fast_without_upstream_attempt`	`decision_timestamp + reopen_at + returned_error`
`half_open_state`	`bounded_probe_concurrency`	`probe_count + outcomes + state_transition`
`fallback`	`same_contract_eligible_route_only`	`eligibility_reason + route_choice + terminal_state`
`operator_override`	`time_bounded_and_audited`	`actor + reason + expiry + restored_policy`

Exercise the critical transition

The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result.

state = CLOSED
if eligible_failures(window) >= threshold and samples >= minimum:
  state = OPEN
  reopen_at = now + cool_down
if state == OPEN and now < reopen_at:
  return fail_fast
if state == OPEN and now >= reopen_at:
  state = HALF_OPEN
if bounded_probe_succeeds():
  state = CLOSED
else:
  state = OPEN

Verify behavior under failure

Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request.

Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for breaker_key: enforce provider_route_plus_protocol_plus_failure_class and retain route_id + endpoint + normalized_error_class.
Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for closed_state: enforce rolling_window_with_minimum_sample and retain eligible_attempts + failures + window_bounds.
Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for open_state: enforce fail_fast_without_upstream_attempt and retain decision_timestamp + reopen_at + returned_error.
Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for half_open_state: enforce bounded_probe_concurrency and retain probe_count + outcomes + state_transition.
Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for fallback: enforce same_contract_eligible_route_only and retain eligibility_reason + route_choice + terminal_state.
Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for operator_override: enforce time_bounded_and_audited and retain actor + reason + expiry + restored_policy.

Failure amplification to prevent

Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions.

global_breaker — One unhealthy provider route can unnecessarily disable unrelated protocols, models, tenants, or regions.
mixed_denominator — Counting client cancellations, invalid requests, and provider failures together produces meaningless transitions.
half_open_stampede — Allowing normal concurrency during recovery can immediately overload a dependency that has only partially recovered.
fallback_cascade — Moving all traffic to the remaining route can overload it and spread a local failure.

Trip signals, denominators, and stop conditions

Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins.

Signal	Decision threshold	Action
`eligible_failure_ratio`	`reviewed_window_and_minimum_sample`	`open_route_breaker`
`open_fail_fast_latency`	`<_caller_remaining_deadline`	`repair_local_decision_path`
`half_open_probe_concurrency`	`<=_configured_probe_limit`	`reject_extra_probes`
`fallback_headroom`	`>=_required_reserved_capacity`	`shed_load_instead_of_failover`

Modelflare boundary and limitations

Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth.

Circuit breakers do not create spare capacity, repair incompatible routes, or make non-idempotent retries safe. They are one stateful guard in a larger timeout, retry-budget, load-shedding, and observability design. Thresholds must come from the workload and failure cost, not from a copied percentage.

Continue through the topic cluster

Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field.

Frequently asked questions

What should key a breaker?

Use the smallest stable failure domain that can be isolated and recovered independently, usually route plus protocol and normalized failure class.

Does open always mean fallback?

No. Failing fast or shedding load can be safer when no compatible route has reserved headroom.

How is half-open tested?

Permit a bounded number of probes, record every transition, and return to open on the defined failure condition.

Sources and verification date

Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state.