# Modelflare: Product Reference
## Summary
Modelflare is an AI model gateway operated by Havenbyte LLC. It provides stable, lower-cost access to GPT, Claude, Grok, and other leading models through compatible interfaces for applications, tools, and coding agents.
Every request includes a complete record of token usage and cost. Individuals and teams can review consumption in real time, understand spend by request, and manage usage centrally.
Canonical website: https://modelflare.dev/
## Core capabilities
### Leading model access
Modelflare connects models from multiple model vendors behind one gateway. The currently available model catalog and supported API formats are listed on the models and pricing page:
https://modelflare.dev/pricing
### Model groups
Users choose both a model and a model group. The live catalog shows the groups currently available for each model. Regular API keys can use an explicit primary group and ordered fallback groups; Smart API Keys evaluate eligible available groups according to the key's routing strategy.
### Usage and cost records
Usage logs show the model, model group, request status, token usage, latency, and cost for each call. This gives individuals and teams a clear per-request record for real-time consumption review and centralized usage management.
### Compatible clients and coding agents
Modelflare provides setup guides for compatible applications and coding tools:
- Codex: https://docs.modelflare.dev/guides/clients/codex/
- Codex interactive quick setup: https://docs.modelflare.dev/guides/clients/codex/start/
- Claude Code: https://docs.modelflare.dev/guides/clients/claude-code/
- Claude Code interactive quick setup: https://docs.modelflare.dev/guides/clients/claude-code/start/
- Grok Build: https://docs.modelflare.dev/guides/clients/grok-build/
- OpenCode: https://docs.modelflare.dev/guides/clients/opencode/
- CC-Switch: https://docs.modelflare.dev/guides/clients/cc-switch/
The documentation site contains API-key, endpoint, model-name, and connection guidance:
https://docs.modelflare.dev/
### Technical article library
The technical article library contains long-form implementation and operations guidance for AI API clients:
- OpenAI-compatible API migration: https://modelflare.dev/articles/openai-compatible-api-guide
- AI API gateway concepts and evaluation: https://modelflare.dev/articles/what-is-ai-api-gateway
- Responses API versus Chat Completions: https://modelflare.dev/articles/responses-api-vs-chat-completions
- SSE streaming, first output, and timeouts: https://modelflare.dev/articles/ai-api-streaming-guide
- Reliable routing and request diagnostics: https://modelflare.dev/articles/reliable-ai-api-routing
- Authentication, access, rate-limit, cancellation, and 5xx diagnosis: https://modelflare.dev/articles/ai-api-error-troubleshooting
- Token usage and cost tracking: https://modelflare.dev/articles/ai-api-cost-tracking
- API key security and workload cost controls: https://modelflare.dev/articles/api-key-security-cost-controls
Browse all topics:
https://modelflare.dev/articles
### Global availability
Modelflare commercial API service is currently available in 176 supported countries, regions, and territories. The canonical list is maintained here:
https://modelflare.dev/supported-regions
### Trust and security
The Trust Center explains the standard API data flow, ordinary request-body handling, diagnostic retention, service-provider categories, safeguards, reliability boundaries, enterprise review materials, and responsible security reporting:
https://modelflare.dev/trust
## Canonical pages
- Homepage: https://modelflare.dev/
- About: https://modelflare.dev/about
- Trust and security: https://modelflare.dev/trust
- Models and pricing: https://modelflare.dev/pricing
- Documentation: https://docs.modelflare.dev/
- Technical articles: https://modelflare.dev/articles
- Supported countries and regions: https://modelflare.dev/supported-regions
- Terms of Service: https://modelflare.dev/terms
- Usage Policy: https://modelflare.dev/usage-policy
- Privacy Policy: https://modelflare.dev/privacy-policy
- User Agreement: https://modelflare.dev/user-agreement
## Naming
Preferred product name: Modelflare
Recognized variants: ModelFlare, Model Flare
## Full technical articles
### OpenAI-Compatible API Guide: Change the Base URL
Canonical URL: https://modelflare.dev/articles/openai-compatible-api-guide
An OpenAI-compatible API lets an existing client keep a familiar authentication header, JSON request shape, and streaming pattern while sending traffic to a different gateway. The change can be as small as replacing the base URL and API key, but compatibility is still a protocol contract—not a promise that every model supports every endpoint or provider-specific field.
This guide explains the safe migration path for applications, scripts, and AI tools that already speak an OpenAI-style API.
#### What OpenAI compatibility actually covers
The most reusable parts of the contract are:
- Bearer-token authentication through the Authorization header.
- JSON requests and responses on versioned /v1 endpoints.
- Common endpoints such as /v1/models, /v1/chat/completions, and /v1/responses.
- Server-sent events for supported streaming requests.
- Familiar request fields such as model, messages, input, stream, and tool definitions when the selected protocol supports them.
Compatibility does **not** mean that one model can be moved freely between Chat Completions and Responses. A model may be exposed only through one verified protocol. Provider-specific reasoning, search, or multimodal fields may also require raw pass-through rather than gateway-side translation.
Treat the live [Models & Pricing](/pricing) catalog as the source of truth for the model, group, and API format you intend to use.
#### Prepare the migration
Before changing application code:
1. Create a dedicated key in **API Keys** instead of reusing a personal or unrelated integration key.
2. Select a primary model group that can access the intended model.
3. Add ordered fallback groups only when they also support that model and fit the same cost and reliability policy.
4. Record the current endpoint, model ID, streaming setting, and tool usage so the before-and-after behavior can be compared.
The canonical Modelflare OpenAI-compatible base URL is:
~~~text
https://modelflare.dev/v1
~~~
Most SDKs expect the base URL to end at /v1 and append /chat/completions or /responses themselves. Check the client's documentation before adding a second endpoint suffix.
#### Verify authentication and model access first
Store the key in an environment variable rather than source code:
~~~bash
export MODELFLARE_API_KEY='YOUR_MODELFLARE_API_KEY'
~~~
Then confirm that the key can list its available models:
~~~bash
curl -sS https://modelflare.dev/v1/models \
-H "Authorization: Bearer $MODELFLARE_API_KEY"
~~~
A successful response proves that the hostname, TLS path, and key are valid. It does not yet prove that a particular request format is valid for every returned model, so the next check must use the intended endpoint.
#### Send one request with the intended protocol
For a Chat Completions model listed as compatible with that format:
~~~bash
curl -sS https://modelflare.dev/v1/chat/completions \
-H "Authorization: Bearer $MODELFLARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "YOUR_CHAT_MODEL",
"messages": [
{"role": "user", "content": "Reply with the active model name."}
],
"stream": false
}'
~~~
For a Responses-compatible coding model:
~~~bash
curl -sS https://modelflare.dev/v1/responses \
-H "Authorization: Bearer $MODELFLARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-5.6-sol",
"input": "Reply with the active model name.",
"stream": false
}'
~~~
Use the exact model ID shown in the live catalog. A model_not_found result usually means the selected key or group cannot access that model, not that changing letter case will fix the request.
#### Validate streaming separately
Non-streaming success is not enough for an application that depends on streaming. Repeat the request with "stream": true, confirm that events arrive incrementally, and make sure the client does not buffer the complete response before rendering it.
When diagnosing a slow stream, separate:
- time spent authenticating and selecting a route;
- time until service provider response headers;
- time until the first effective content or tool delta;
- generation throughput after visible output begins.
Modelflare usage logs retain request-level timing metadata for this purpose without storing prompts, response text, raw request bodies, API keys, emails, or plaintext IP addresses.
#### Production migration checklist
- Keep the API key in a secret store or environment variable.
- Pin the base URL to https://modelflare.dev/v1.
- Use a model that explicitly supports the chosen endpoint.
- Test non-streaming and streaming modes independently.
- Test tools, structured output, reasoning controls, and multimodal input if the application uses them.
- Preserve explicit zero or false values when they carry meaning in the client request.
- Set client timeouts for the real workload instead of a short health-check prompt.
- Review usage logs after cutover for status, latency, token usage, selected group, and cost.
Once the protocol boundary is verified, the client can usually keep its existing request lifecycle while Modelflare provides model access, routing policy, and per-request visibility behind the compatible endpoint.
For the operational layer behind that endpoint, read [What Is an AI API Gateway?](/articles/what-is-ai-api-gateway). If the client can use more than one OpenAI format, compare [Responses API vs Chat Completions](/articles/responses-api-vs-chat-completions) before changing paths.
### Responses API vs Chat Completions
Canonical URL: https://modelflare.dev/articles/responses-api-vs-chat-completions
Responses API and Chat Completions both send prompts to language models, but they organize input, output, tools, and streaming differently. Choosing between them should start with the client and model contract—not with the assumption that a newer endpoint automatically works for every provider.
The shortest decision rule is:
- Use **Responses API** for a coding agent or application that already expects Responses items, tool events, and the Responses streaming lifecycle.
- Use **Chat Completions** for broadly compatible chat clients and for provider families that expose their OpenAI-style models through raw Chat Completions pass-through.
Always verify the selected model's supported API format in [Models & Pricing](/pricing).
#### Protocol comparison
| Question | Responses API | Chat Completions |
| --- | --- | --- |
| Primary input shape | input and typed input items | A messages array |
| Output model | Typed output items and events | Assistant message choices and deltas |
| Streaming | Responses event stream | Chat completion chunk stream |
| Tool activity | Typed tool-call and tool-output items | Tool calls attached to assistant messages |
| Best fit | Agents, coding tools, and Responses-native applications | Chat clients and widely supported OpenAI-compatible providers |
| Model portability | Only models verified for Responses | Only models verified for Chat Completions |
The table describes the wire contract. It does not imply that Modelflare converts every provider feature between the two formats.
#### When Responses API is the better fit
Choose /v1/responses when the client already treats a model run as a sequence of typed items rather than one assistant message. This is common in coding agents that need to distinguish visible text, reasoning summaries, function arguments, custom tool input, and other events.
A minimal request looks like:
~~~bash
curl -sS https://modelflare.dev/v1/responses \
-H "Authorization: Bearer $MODELFLARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-5.6-sol",
"input": "List the three checks to make before an API migration.",
"stream": true
}'
~~~
Responses streaming should be validated end to end. A client that understands Chat Completions chunks but not Responses events can connect successfully and still fail to render useful output.
#### When Chat Completions is the safer choice
Choose /v1/chat/completions when the application is built around system, user, assistant, and tool messages, or when the selected provider family documents an OpenAI-compatible Chat Completions endpoint.
~~~bash
curl -sS https://modelflare.dev/v1/chat/completions \
-H "Authorization: Bearer $MODELFLARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "YOUR_CHAT_MODEL",
"messages": [
{"role": "system", "content": "Answer concisely."},
{"role": "user", "content": "What should an API health check verify?"}
],
"stream": true
}'
~~~
For non-OpenAI provider families, Modelflare can expose raw Chat Completions pass-through so private request fields such as provider-specific reasoning or search controls reach the service provider service unchanged. That behavior is intentionally narrower than claiming full Responses compatibility.
#### Do not choose by model name alone
Three separate checks are required:
1. **The API key can access the model.** Access depends on the key and available model groups.
2. **The model supports the endpoint.** A model in /v1/models is not automatically valid for both request formats.
3. **The client understands the stream.** Responses events and Chat Completions chunks are different client contracts.
If any of these checks fail, switching the endpoint path alone may turn a clear compatibility error into an empty or partially rendered response.
#### Tool and structured-output migration
Before moving an existing integration:
- compare the tool definition schema used by the client;
- verify how tool-call IDs and tool outputs are returned;
- preserve explicit optional values instead of dropping 0 or false;
- check whether the provider requires private fields that must pass through unchanged;
- test tool-only responses that may not contain a visible text delta;
- confirm how the client detects completion and usage.
The same prompt producing similar text is not a sufficient protocol test. A real test should include the features the application depends on.
#### A practical selection flow
1. Open [Models & Pricing](/pricing) and choose the model and group.
2. Confirm the supported API format.
3. Follow a client-specific guide in [Modelflare Docs](/docs) when one exists.
4. Send one non-streaming request.
5. Send one streaming request.
6. Exercise tools or structured output.
7. Review the request in usage logs for status, timing, token usage, and cost.
Responses API is not a universal replacement for Chat Completions, and Chat Completions is not obsolete. The correct format is the one jointly supported by the client, the selected model, and the service provider provider contract.
After choosing the format, use the [AI API Streaming Guide](/articles/ai-api-streaming-guide) to implement its event parser, timing metrics, cancellation, and timeout policy.
### Reliable AI API Routing & Diagnostics
Canonical URL: https://modelflare.dev/articles/reliable-ai-api-routing
Reliable AI API routing is not one retry loop. It means giving each API key a clear primary group, fallback groups, model capabilities, and request limits, while retaining enough user-facing evidence when a request slows down or fails.
A good routing design makes the result explainable and prevents a group switch from silently changing the requested model or billing policy.
#### Start with an explicit key policy
Modelflare supports two routing approaches:
- A regular API key uses one explicit primary group and an ordered list of fallback groups.
- A Smart API Key evaluates the groups currently available to the account and continues through eligible candidates according to its strategy.
Fallback order is not traffic splitting. It is a priority sequence. When the primary group is unavailable, requests continue through the later groups in order.
Create separate keys for separate applications or environments. This makes group access, quota, expiration, and usage records easier to reason about than one shared key used everywhere.
#### A fallback must preserve the request contract
Before adding a group to a fallback list, verify that it:
1. exposes the requested model;
2. supports the same endpoint and streaming behavior;
3. permits any required Service Tier or provider-specific field;
4. has an acceptable usage multiplier and request-rate limit;
5. is actually unlocked for the account.
A fallback group is not a license to substitute an arbitrary model. The requested model and protocol still define the wire contract.
#### Understand group request limits
Request-rate limits are applied according to the account and the group that handles the request. Separate API keys make usage easier to analyze, but they do not automatically bypass account or group limits.
When the current group reaches its limit, an API key with fallback groups can continue to the next available group. If no fallback is available, the request receives 429. Client retries should use bounded backoff rather than immediately creating a burst of new requests.
#### Use request-level performance metrics
Usage logs expose the product-level metrics needed to understand a request:
| Metric | What it helps explain |
| --- | --- |
| Total response time | Time from submitting the request until completion |
| First response | Time until the first effective text, reasoning, or tool event |
| First visible text | Time until text appears for requests expected to produce text |
| Visible output speed | Generation speed after visible output begins |
| Output tokens | The amount of output produced by the request |
A slow first response usually means the request waited longer before generation began. A normal first response followed by low output speed points more toward the generation phase. Combine these metrics with status, model, group, and time range to distinguish a one-off delay from a sustained issue.
Tool-call-only responses may not contain visible text, so “first response” is the safer first-effective-output signal for Responses traffic.
#### Use evidence that is safe to retain
Modelflare timing metadata does not include prompts, response text, raw request bodies, API keys, emails, or plaintext IP addresses. This keeps the operational record useful without turning latency diagnostics into a second content store.
For an incident, capture:
- request ID and timestamp;
- requested model and selected group;
- endpoint and streaming mode;
- downstream status;
- status, total response time, and first-response timing;
- whether the client cancelled before the request completed.
That evidence lets support verify group switching, slow model generation, and client cancellation without exposing internal service topology in ordinary logs.
#### Production reliability checklist
- Give every application its own API key.
- Keep the primary and fallback order intentional.
- Verify model and protocol support in every candidate group.
- Set a client timeout long enough for the real workload.
- Use bounded retries with jitter for retryable failures.
- Do not retry authentication, quota, or model-access errors as if they were transient.
- Test both non-streaming and streaming requests.
- Monitor 429, first response, output throughput, and cancellation.
- Review group pricing before treating a fallback as equivalent.
Reliability comes from preserving the request contract while retaining enough evidence when a request fails. Ordered fallback groups reduce dependence on one group, while performance and usage records make remaining failures explainable.
Use the [AI API Error Guide](/articles/ai-api-error-troubleshooting) to interpret authentication, access, rate-limit, cancellation, and 5xx failures. Apply the controls in [AI API Key Security](/articles/api-key-security-cost-controls) before expanding fallback access.
### AI API Cost Tracking: Tokens and Model Groups
Canonical URL: https://modelflare.dev/articles/ai-api-cost-tracking
AI API cost tracking is most useful when every charge can be connected to one request, one model, one model group, and one measured unit of work. A monthly total can show that spend changed; a request-level record can explain why.
Modelflare combines model pricing, measured usage, group policy, and request logs so users can review cost without estimating from visible text alone.
#### The four layers of a request cost
##### 1. Model pricing
Each model has its own pricing basis. Many text models use input and output tokens, while image, audio, reranking, or other APIs may use different metered units. Some models also use versioned, advanced pricing rules for features that cannot be represented by one flat token rate.
Use [Models & Pricing](/pricing) for the current model and API-format contract instead of copying a price into application code.
##### 2. Measured request usage
The gateway records the usage returned or derived for the completed request. For token-priced text models, input and output are kept separate because their rates may differ.
Streaming text visible in a terminal is not a reliable cost oracle. Tool items, reasoning, cached input, normalized tokens, or provider-specific usage fields may affect the bill even when they are not displayed as ordinary assistant text.
##### 3. Model-group multiplier
The selected routing group can apply a sales multiplier to base usage billing. For example, a group ratio of 0.9 applies 90% of the base model charge for that request.
The multiplier belongs to usage billing. It does not change how many credits a top-up adds.
##### 4. The final request record
The usage log connects the calculated amount to the request's model, group, status, token usage, timing, and other safe operational metadata. Failed or cancelled requests can follow different settlement paths, so the persisted result is more authoritative than a client-side estimate.
#### What to compare in usage logs
When spend changes, compare requests along these dimensions:
- **Model:** Did traffic move to a model with a different input, output, or feature price?
- **Group:** Did the selected group or fallback group use a different multiplier?
- **Protocol:** Did the request move between Chat Completions and Responses with a different usage shape?
- **Input size:** Did prompts, retrieved context, files, or tool results grow?
- **Output size:** Did completion limits or agent loops produce more output?
- **Status and retries:** Did failures cause additional completed attempts?
- **Timing:** Did longer generation correlate with more output rather than just queueing?
Use a stable request ID or application-specific API key to isolate a workload instead of comparing unrelated account traffic.
#### Set practical cost boundaries
##### Separate API keys by workload
Create different keys for production, development, automation, and personal tools. Each key can have its own name, expiration, group policy, and finite or unlimited quota.
##### Choose groups intentionally
Do not select a group only by its label. Check live model availability, the group multiplier, access requirement, RPM, and fallback policy. A cheaper primary group with an acceptable fallback can be more predictable than an implicit routing choice that is difficult to explain.
##### Limit input before limiting output
Large system prompts, repeated conversation history, retrieved documents, and tool results often dominate input usage. Trim context that no longer affects the answer, and avoid resending unchanged data when the client can safely reference or cache it.
##### Review agent loops
One user action can create several model requests. Track each tool round and retry rather than treating the final visible answer as one API call.
#### Why client-side estimates drift
A local tokenizer or character count can be useful for planning, but it may differ from billed usage because:
- providers can normalize or count content differently;
- cached, reasoning, image, audio, or tool units may use separate rates;
- gateway settlement uses the actual selected model and group;
- failed requests and refunds depend on the real request lifecycle;
- pricing can change while old requests retain their recorded result.
For financial review, reconcile the persisted usage and credit-ledger records rather than deriving remaining credits from one display field.
#### A repeatable review process
1. Filter usage logs to one API key and time range.
2. Group requests by model and selected group.
3. Compare input, output, status, and cost per request.
4. Inspect outliers for retries, long context, tool loops, or fallback.
5. Confirm the live price and group policy before changing routing.
6. Set a finite key quota when the workload needs a hard spending boundary.
7. Recheck the same metrics after the change.
Clear cost control starts with attribution. When model, group, usage, and request outcome remain connected, teams can optimize the actual source of spend instead of guessing from aggregate totals.
Turn that review into an enforceable workload boundary with the quota, expiration, model, IP, rotation, and routing controls in [AI API Key Security](/articles/api-key-security-cost-controls).
### What Is an AI API Gateway? Routing & Usage
Canonical URL: https://modelflare.dev/articles/what-is-ai-api-gateway
An AI API gateway is a service boundary between an application and one or more model backends. The application sends a request to the gateway; the gateway applies authentication and access policy, selects an eligible route for the requested model, forwards the request, and records the result.
That definition is more useful than saying “one API for every model.” A gateway can centralize the parts that should be consistent, but it does not make every model or provider support the same protocol, tools, input types, or latency profile.
#### Where an AI API gateway sits
A typical request path has six stages:
| Stage | Responsibility |
| --- | --- |
| Client | Chooses a model, protocol, input, and streaming mode |
| Compatible endpoint | Accepts the selected wire format, such as Chat Completions or Responses |
| API key policy | Checks account access, quota, expiration, model limits, IP rules, and routing configuration |
| Model route | Selects an eligible model group and channel without changing the requested contract |
| Model backend | Executes the request and returns protocol-specific output |
| Usage record | Connects status, model, group, tokens, timing, and cost to the request |
In Modelflare, the public base URL is https://modelflare.dev/v1. Clients can use compatible endpoints while keeping model availability, protocol support, and current pricing discoverable through [Models & Pricing](/pricing).
#### What a gateway can centralize
##### Authentication and key policy
Applications use one Modelflare API key instead of putting several service provider credentials into every deployment. Separate keys can be created for production, development, agents, and automation so each workload has its own quota, expiration, model access, IP rules, and routing policy.
##### Model discovery
The authenticated model list shows the model IDs available to the current key:
~~~bash
curl -sS https://modelflare.dev/v1/models \
-H "Authorization: Bearer $MODELFLARE_API_KEY"
~~~
This is an access check, not a universal compatibility promise. A returned model still needs to support the endpoint and features used by the client.
##### Routing and fallback
A regular API key can use one primary model group and ordered fallback groups. A Smart API Key evaluates the concrete groups available to the account according to its routing strategy. In both cases, routing chooses where the requested model is served; it should not silently replace that model with a different one.
For production design, see [Reliable AI API Routing](/articles/reliable-ai-api-routing).
##### Usage and cost visibility
A gateway can connect the completed request to its selected model and group. Modelflare usage logs expose request status, token usage, response timing, and cost so an operator can investigate one request instead of estimating from a monthly total. The accounting layers are described in [AI API Cost Tracking](/articles/ai-api-cost-tracking).
#### What remains model- and protocol-specific
An AI API gateway does not erase the service provider contract. Verify these boundaries separately:
- **Endpoint support:** a model may support Chat Completions, Responses, Anthropic Messages, Gemini-style requests, or only a subset.
- **Streaming events:** Chat Completions chunks and Responses events are different client contracts.
- **Tools and structured output:** schemas and supported tool types can vary by model and provider.
- **Multimodal input:** image, audio, files, and other inputs require explicit model and endpoint support.
- **Provider-specific fields:** some compatible endpoints intentionally pass private controls through rather than translating them.
- **Latency and limits:** route availability does not make every backend share the same first-output time or request limit.
The safest migration starts with the [OpenAI-Compatible API Guide](/articles/openai-compatible-api-guide) and tests the exact features used by the application.
#### A practical gateway evaluation
Evaluate a gateway with the real workload, not one short prompt:
1. Create a dedicated API key with the intended quota and routing policy.
2. Retrieve /v1/models and select a model visible to that key.
3. Confirm the supported API format on the live catalog.
4. Send one non-streaming request and preserve the returned status.
5. Test streaming, tools, structured output, and multimodal input separately when required.
6. Review the resulting usage record for model, group, tokens, timing, and cost.
7. Exercise the fallback path without changing the requested model or protocol.
8. Repeat with representative context size and client timeouts before production cutover.
#### When a gateway is a good fit
An AI API gateway is useful when a team needs consistent key management, access to several model families, explicit routing policy, centralized usage records, or a stable integration boundary for agents and applications.
A direct provider integration can still be appropriate when one application depends on a provider-only feature that has no compatible gateway contract, or when the team intentionally wants to own provider credentials, routing, settlement, and diagnostics itself.
The decision is not “gateway or provider” in the abstract. It is whether the gateway preserves the exact protocol and features the workload needs while reducing operational complexity around access, routing, and evidence.
#### Frequently asked questions
##### Does one gateway mean every model uses the same request format?
No. A gateway can expose several compatible formats, but the client, endpoint, model, and service provider provider must still agree on one contract. Check the catalog before switching an existing request between Chat Completions and Responses.
##### Does fallback automatically choose a different model?
Modelflare fallback groups are routes for the requested model. They are not an instruction to substitute an arbitrary model. Every candidate group must expose the same model and protocol capabilities required by the request.
##### What should be measured before production?
Measure authentication success, model access, non-streaming output, first effective streaming output, total response time, usage, cost, and failure behavior. A successful health-check prompt alone does not prove production compatibility.
### AI API Streaming: SSE, First Output & Timeouts
Canonical URL: https://modelflare.dev/articles/ai-api-streaming-guide
AI API streaming sends output incrementally while a model is generating instead of waiting for the complete response body. It can improve perceived responsiveness, but it does not automatically reduce model latency, and it adds a protocol contract that the client must parse correctly.
The first rule is to match the parser to the endpoint. Chat Completions uses streamed completion chunks, while Responses uses typed response events. A client can receive HTTP 200 and still render nothing if it expects the wrong event shape.
#### Start with an unbuffered request
For a Chat Completions model, curl -N disables curl output buffering:
~~~bash
curl -N -sS https://modelflare.dev/v1/chat/completions \
-H "Authorization: Bearer $MODELFLARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "YOUR_CHAT_MODEL",
"messages": [{"role": "user", "content": "Explain SSE in three points."}],
"stream": true
}'
~~~
For a Responses-compatible model:
~~~bash
curl -N -sS https://modelflare.dev/v1/responses \
-H "Authorization: Bearer $MODELFLARE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "YOUR_RESPONSES_MODEL",
"input": "Explain SSE in three points.",
"stream": true
}'
~~~
Use an exact model ID from [Models & Pricing](/pricing). Test the same request once with streaming disabled first; this separates request validation from stream parsing.
#### Treat the event stream as a protocol
Server-Sent Events are framed records, not arbitrary JSON fragments. A production client needs to:
- keep the response body unbuffered through its HTTP library, proxy, and UI layer;
- accumulate partial network reads until a complete event is available;
- parse the event type used by the selected endpoint;
- handle text deltas, reasoning or tool events, completion events, and error events;
- avoid assuming that every useful response contains visible text;
- preserve cancellation and final usage information;
- close the stream after a terminal event instead of waiting forever.
When possible, use an SDK that natively supports the chosen protocol. A home-grown parser should be tested with fragmented frames, multiple events in one read, tool-only output, and a clean terminal event.
#### Measure more than time to first token
“Time to first token” is often too vague for agent traffic. Separate these measurements:
| Measurement | What it means |
| --- | --- |
| Connection and authentication | Time to reach the gateway and validate the key |
| Service provider response headers | Time until the selected route starts a response |
| First effective output | First useful text, reasoning, or tool event |
| First visible text | First text a user can actually see |
| Output throughput | Visible generation speed after output begins |
| Total response time | Time until completion, failure, or cancellation |
A tool call may produce effective output before any visible text. For that reason, first effective output is the safer operational metric, while first visible text is useful for user experience.
Modelflare usage logs retain request-level timing evidence so these phases can be compared with model, group, status, tokens, and cost.
#### Design timeouts by phase
One short global timeout creates ambiguous failures. A streaming client should reason about several budgets:
1. **Connection timeout:** failure to establish the HTTP connection.
2. **Header or first-output timeout:** no response or effective event after the request was accepted.
3. **Idle timeout:** the stream started but no new event arrived for an unexpectedly long period.
4. **Overall deadline:** the complete operation exceeded the workload's business limit.
Reasoning-heavy or tool-driven requests can legitimately wait longer before visible text than a short chat completion. Choose budgets from observed workload behavior, and cancel intentionally when the user no longer needs the result.
#### Understand cancellation and 499 records
If the client, reverse proxy, browser tab, or caller deadline closes the downstream connection, the gateway can record the request as client-cancelled. A 499 label in a request record is evidence that the downstream connection ended before completion; it is not proof by itself that the selected model or channel failed.
When investigating cancellation, compare:
- the client deadline and abort signal;
- proxy buffering and idle timeouts;
- time until first effective output;
- whether any event reached the client;
- the selected model and group;
- the request ID and cancellation timestamp.
#### Diagnose a stream that returns no visible text
Use this order:
1. Repeat the request with "stream": false.
2. Confirm the model supports the selected endpoint.
3. Capture raw events before the UI transforms them.
4. Check whether the response contains a tool call or reasoning event instead of text.
5. Verify that an intermediary is not buffering the body.
6. Confirm that the parser handles the endpoint's terminal event.
7. Review usage logs for status, first output, total time, and cancellation.
If non-streaming works and raw streaming events arrive, the failure is usually in client parsing or rendering rather than authentication or model access. If no raw event arrives, continue with the [AI API Error Guide](/articles/ai-api-error-troubleshooting).
#### Production streaming checklist
- Match the parser to Chat Completions or Responses explicitly.
- Disable buffering in the HTTP client and every controlled proxy layer.
- Test text, reasoning, tool-only, and error events used by the application.
- Distinguish first effective output from first visible text.
- Use phase-appropriate timeouts and an intentional abort signal.
- Treat retries as new requests unless the operation has a safe idempotency design.
- Preserve request ID, status, selected model, group, and timing for diagnosis.
- Compare the endpoint decision in [Responses API vs Chat Completions](/articles/responses-api-vs-chat-completions).
#### Frequently asked questions
##### Does streaming make the model generate faster?
Not necessarily. Streaming exposes output earlier; generation speed and time before the first event still depend on the model, route, context, reasoning, and tools.
##### Why does curl work while the application shows nothing?
The application or an intermediary may buffer the body, or the parser may expect Chat Completions chunks while receiving Responses events. Capture raw frames at the application boundary before changing the model or route.
##### Should a client retry after a stream disconnects?
Only when repeating the operation is safe. A disconnected stream may have already caused service provider work or tool activity. Use bounded retries and application-level idempotency where duplicate side effects matter.
### AI API Error Guide: 401, 403, 429 & 5xx
Canonical URL: https://modelflare.dev/articles/ai-api-error-troubleshooting
AI API errors are easiest to diagnose when the status code is treated as the beginning of an investigation, not the complete cause. Preserve the request ID, timestamp, endpoint, model, API key name, selected group, response body, and timing before retrying.
The goal is to identify which layer rejected or ended the request: client, authentication, access policy, protocol validation, routing, model backend, or downstream connection.
#### What common status codes usually indicate
| Status | First interpretation | First action |
| --- | --- | --- |
| 400 | Invalid request shape, field, or protocol contract | Compare the payload with the selected endpoint; do not retry unchanged |
| 401 | Missing, malformed, disabled, expired, or invalid credential | Verify the Authorization header and the current key |
| 403 | Authenticated but blocked by account, model, group, IP, or policy | Check key restrictions and model/group access before testing channels |
| 404 | Wrong endpoint path or unavailable model identifier | Verify the base URL, path, and authenticated model list |
| 429 | Request-rate, quota, or eligible-route limit | Inspect quota and group limits; use bounded backoff or configured fallback |
| 499 | Downstream client disconnected or cancelled before completion | Inspect client deadlines, aborts, proxies, and first-output timing |
| 500 | Internal processing or adapter failure | Preserve the request ID; retry only if the operation is safe |
| 502/503 | Bad service provider response, unavailable route, or temporary capacity issue | Check route evidence and use bounded fallback or retry |
| 504 | A gateway or service provider time budget expired | Compare first-output and total timing with client deadlines |
These interpretations are diagnostic starting points. The error body and request record determine the actual reason for a specific call.
#### Diagnose 401 before changing the model
A 401 normally occurs before useful model routing. Check:
- the header uses Authorization: Bearer ...;
- the application did not include quotes, whitespace, or a placeholder value;
- the key is enabled and has not expired;
- the request is sent to the intended Modelflare host;
- a secret manager or deployment did not retain an older rotated key.
Do not paste the full key into logs or support messages. Reproduce with a dedicated test key if exposure is possible, then revoke the compromised credential.
#### Diagnose 403 as an access-policy failure first
A 403 does not automatically mean an service provider provider rejected the request. Authentication can succeed while access fails before channel selection because:
- the key restricts the requested model;
- the key's IP allowlist does not include the observed client IP;
- the account cannot use the selected model group;
- the user or key is disabled by policy;
- the requested feature is not allowed for that route.
Retrieve /v1/models with the same key, inspect its group and model limits, and compare the exact error code. If no channel was selected, changing service provider channels cannot fix the earlier policy decision.
#### Diagnose 429 without creating a retry storm
A 429 is a capacity or policy signal, not an instruction to retry immediately in a tight loop. Determine whether the limit belongs to the key quota, account, model group, or selected route.
For retryable rate limits:
1. honor Retry-After when present;
2. use exponential backoff with jitter;
3. cap attempts and total elapsed time;
4. keep application concurrency within a known bound;
5. use ordered fallback groups only when they preserve the requested model and protocol.
Creating more API keys does not necessarily bypass an account- or group-level request limit.
#### Diagnose 499 from the caller outward
Modelflare can record 499 when the downstream client connection ends before the response completes. Start with the caller:
- Was an abort signal triggered?
- Did the client, load balancer, CDN, or reverse proxy reach an idle or total timeout?
- Did the user close the page or stop the agent?
- Did the stream produce a tool event without visible text?
- How long elapsed before first effective output?
A 499 record alone does not establish a channel outage. Compare other requests in the same time window and the same model/group before changing routing policy.
#### Separate 5xx failures by layer
For 5xx, preserve the original status and error details instead of converting every failure into a generic 500 in the client.
- A validation or adapter 500 can be deterministic and may fail again unchanged.
- A 502 can indicate an invalid or incomplete service provider response.
- A 503 can indicate no eligible route or temporary unavailability.
- A 504 indicates that a time budget expired, but the responsible layer still needs to be identified.
Use request timing, selected group, channel evidence available to support, and nearby requests to distinguish a single bad response from a sustained route issue.
#### Decide whether to retry
| Failure class | Default retry decision |
| --- | --- |
| Invalid payload or unsupported protocol | Fix the request; do not retry unchanged |
| Invalid key or denied access | Fix credentials or policy; do not retry unchanged |
| Quota exhausted | Increase or reallocate quota intentionally; do not loop |
| Rate limit | Retry with bounded backoff when permitted |
| Temporary 502/503/504 | Retry idempotent work with a strict attempt budget |
| Client cancellation | Retry only if the user still wants the work and duplicates are safe |
| Tool or write side effect | Require application-level idempotency before repeating |
Retries create new work and can create new cost. Record every attempt and avoid hiding the original failure behind a later success.
#### Evidence to capture safely
For support or incident review, capture:
- request ID and UTC timestamp;
- endpoint and streaming mode;
- requested model and selected group;
- HTTP status and structured error code;
- total time and first-effective-output time;
- whether the caller cancelled;
- a sanitized description of the request feature, such as tools or image input.
Do not include the full API key, prompt, response content, raw request body, email, or plaintext IP address unless an approved secure process specifically requires it.
After locating the failing layer, use [Reliable AI API Routing](/articles/reliable-ai-api-routing) for fallback design and the [Streaming Guide](/articles/ai-api-streaming-guide) for event and timeout failures.
#### Frequently asked questions
##### Does a 403 prove that a provider is down?
No. A 403 can be produced by account, key, model, group, IP, or feature policy before an service provider route is selected. Check the request record and error code first.
##### Is 499 a standard service provider model error?
No. In Modelflare request records it represents a downstream cancellation observed before completion. Investigate caller and proxy deadlines before treating it as model failure.
##### Should every 5xx response be retried?
No. Retry only operations that are safe to repeat, use a bounded attempt and time budget, and preserve the first error. Deterministic adapter or request problems will not improve with repetition.
### AI API Key Security & Cost Controls
Canonical URL: https://modelflare.dev/articles/api-key-security-cost-controls
An AI API key should identify one workload boundary, not an entire organization forever. Separate keys make access, quota, routing, rotation, and usage records easier to control without exposing one credential across unrelated applications.
Modelflare API keys can combine an expiration date, finite or unlimited quota, model limits, an IP allowlist, and routing policy. These controls are most effective when they are designed together rather than added only after an incident.
#### Start with one key per workload
Create separate keys for boundaries that need independent access or accounting, such as:
- production API and background workers;
- development and CI;
- a coding agent or local tool;
- a customer-facing application;
- a temporary contractor or migration;
- a high-cost batch workload.
Use names that describe environment and purpose, such as prod-support-agent or dev-evaluation. A clear name makes request records and rotation decisions understandable without revealing the secret.
#### Keep the secret out of client-visible code
Store the full key in a server-side secret manager or protected environment variable. Do not put it in:
- browser JavaScript or mobile packages that users can inspect;
- source control, container images, screenshots, or tickets;
- analytics events, error trackers, shell history, or application logs;
- URLs, query strings, or copied command examples with real values.
For a browser application, send requests through a backend that authenticates the user and holds the Modelflare key. If a key may have leaked, create a replacement, move the workload, verify traffic on the new key, and revoke the old one.
#### Use quota and expiration as independent boundaries
A finite key quota places a hard usage boundary on that key. An expiration date limits how long the credential remains valid. They solve different risks:
| Control | Primary purpose |
| --- | --- |
| Finite quota | Limits the amount of usage attributable to the key |
| Unlimited quota | Avoids a key-level cap; account credits and other policy still apply |
| Expiration | Ends access after a defined time |
| Disable or revoke | Stops new access immediately |
Development, contractor, and evaluation keys should usually have both a finite quota and an expiration. Production keys may need a larger or unlimited key quota, but they still benefit from separate workload attribution and a rotation plan.
Quota is not a substitute for concurrency limits or client retry control. A burst can consume the remaining amount quickly, and repeated failures can create additional requests.
#### Restrict models when the workload is narrow
Model limits prevent a key from requesting models outside its intended set. This reduces accidental use of an incompatible or unexpectedly expensive model and gives a clearer 403 boundary when the client asks for something it should not access.
Keep the allowed set aligned with the endpoint. A key that can list a model still needs a client and protocol that support that model's request format. Review the live contract in [Models & Pricing](/pricing) instead of copying model IDs into a long-lived policy document.
#### Add IP rules only when the network path is stable
An IP allowlist checks the client IP observed by Modelflare against individual addresses or CIDR ranges configured on the key. It is useful for servers with known egress, but it can break traffic when:
- a cloud workload changes outbound NAT addresses;
- requests move through a new proxy or gateway;
- IPv4 and IPv6 paths differ;
- a developer switches networks;
- the configured CIDR is broader or narrower than intended.
Verify the real egress path before enabling the restriction, keep an emergency rotation path, and do not treat IP allowlisting as a replacement for secret protection.
#### Choose regular or Smart routing intentionally
| Key type | Routing behavior | Best fit |
| --- | --- | --- |
| Regular API Key | One explicit primary group plus optional ordered fallback groups | Workloads that require a known route order and group policy |
| Smart API Key | Evaluates account-eligible groups using balanced, stability-first, or low-price-first strategy | Workloads that want broader automatic route selection |
Both key types still send the requested model and protocol. Routing policy chooses an eligible group; it does not guarantee that every group supports every model or feature. A regular fallback list should be tested in order. A Smart API Key should be evaluated with real latency, price, and compatibility requirements before production.
For detailed behavior, read [Reliable AI API Routing](/articles/reliable-ai-api-routing).
#### Rotate without an avoidable outage
Use an overlapping rotation:
1. Create a new key with the same intended restrictions.
2. Store it in the application's secret system.
3. Deploy the new credential without deleting the old one.
4. Verify model access and one real request.
5. Confirm new usage records appear under the new key.
6. Remove the old key from every deployment and automation job.
7. Disable or delete the old key.
8. Monitor for unexpected requests using the retired credential.
If compromise is active, revoke first and accept the controlled interruption. The overlap procedure is for planned rotation, not continued exposure.
#### Connect key policy to cost review
Key-level isolation makes cost review actionable. Filter usage logs by key, then compare model, group, input and output usage, status, retries, and cost. A finite quota can stop further use, but the request records explain what consumed it.
See [AI API Cost Tracking](/articles/ai-api-cost-tracking) for a repeatable review process.
#### API key checklist
- One key per application, environment, or accountable workload.
- No secrets in browser code, repositories, logs, URLs, or screenshots.
- Finite quota and expiration for temporary or experimental use.
- Model restrictions for narrow workloads.
- IP rules only for a verified, stable egress path.
- Explicit regular fallback order or an intentionally selected Smart strategy.
- Rotation tested before the old key is removed.
- Usage and cost reviewed by key after every policy change.
#### Frequently asked questions
##### Does an unlimited key remove all spending limits?
No. It removes the key-level quota cap. Account credits, model pricing, group policy, request limits, and other service controls still apply.
##### Can multiple keys bypass a rate limit?
Not necessarily. A limit can apply at account, model group, or route level. Create keys for access and attribution, not as an unverified rate-limit workaround.
##### Is an IP allowlist enough if a key leaks?
No. Network rules reduce where a key can be used, but network paths change and configuration can be wrong. Rotate exposed credentials and keep the secret protected at rest and in deployment.