Reliable AI API Routing & Diagnostics
Design reliable AI API routing with explicit fallback order, group RPM limits, and per-request timing evidence for diagnosing failures.
Reliable AI API routing is not one retry loop. It is a sequence of explicit decisions: which API key policy applies, which model group can serve the request, whether that group has remaining RPM capacity, which healthy channel is selected, and what evidence is retained when the request slows down or fails.
A good routing design makes each decision observable and prevents a fallback from silently changing the requested model or billing policy.
Start with an explicit key policy
Modelflare supports two routing approaches:
- A regular API key uses one explicit primary group and an ordered list of fallback groups.
- A Smart API Key evaluates the groups currently available to the account and continues through eligible candidates according to its strategy.
Fallback order is not traffic splitting. It is a priority sequence. A later group is considered when the earlier candidate cannot serve the request under the current access, model, health, or rate-limit conditions.
Create separate keys for separate applications or environments. This makes group access, quota, expiration, and usage records easier to reason about than one shared key used everywhere.
A fallback must preserve the request contract
Before adding a group to a fallback list, verify that it:
- exposes the requested model;
- supports the same endpoint and streaming behavior;
- permits any required Service Tier or provider-specific field;
- has an acceptable usage multiplier and RPM limit;
- is actually unlocked for the account.
A fallback group is not a license to substitute an arbitrary model. The requested model and protocol still define the wire contract.
Understand group RPM behavior
Routing-group RPM is enforced against the concrete group selected for the request. The current limiter uses a 60-second sliding window shared by all API keys and channels used by the same user in that group.
Two details matter operationally:
- retries inside the same request and group do not consume another RPM reservation;
- when the selected group is full, a Smart API Key or an ordered fallback list can continue to the next eligible group before billing or an upstream request.
If no next candidate exists, the request receives 429. Client retries should use bounded backoff rather than immediately creating a burst of new requests.
Separate routing time from upstream time
When a request is slow, total duration alone cannot identify the bottleneck. Review these timing fields in order:
| Field | What it helps isolate |
|---|---|
auth_ms | API-key and user authentication |
distribution_ms | Group and channel selection policy |
upstream_headers_ms | Network path, upstream queueing, or saturation before headers |
first_response_ms | Time until the first effective text, reasoning, or tool delta |
first_text_delta_ms | Time until visible text for requests that should produce text |
visible_output_tps | Generation speed after visible output starts |
channel_inflight_count | Concurrent load on the selected channel |
High upstream_headers_ms with normal local fields points toward the upstream path. Normal headers with high first_response_ms points more often to model scheduling, queueing, or prefill work. Low visible_output_tps means generation itself is slow after output begins.
Tool-call-only responses may not have a visible text delta, so first_response_ms is the safer time-to-first-effective-output signal for Responses traffic.
Use evidence that is safe to retain
Modelflare timing metadata does not include prompts, response text, raw request bodies, API keys, emails, or plaintext IP addresses. This keeps the operational record useful without turning latency diagnostics into a second content store.
For an incident, capture:
- request ID and timestamp;
- requested model and selected group;
- endpoint and streaming mode;
- downstream status;
- retry and selected-channel evidence;
- upstream header and first-response timing;
- whether the client cancelled before upstream completion.
That evidence can distinguish local rejection, a real upstream attempt, fallback, slow generation, and downstream cancellation.
Production reliability checklist
- Give every application its own API key.
- Keep the primary and fallback order intentional.
- Verify model and protocol support in every candidate group.
- Set a client timeout long enough for the real workload.
- Use bounded retries with jitter for retryable failures.
- Do not retry authentication, quota, or model-access errors as if they were transient.
- Test both non-streaming and streaming requests.
- Monitor
429, upstream header latency, first response, output throughput, and cancellation. - Review group pricing before treating a fallback as equivalent.
Reliability comes from preserving the request contract while making failure boundaries visible. Ordered fallbacks reduce dependence on one route, but the timing and usage record is what makes the system explainable when a request still fails.