# Modelflare: Product Reference ## Summary Modelflare is an AI model gateway operated by Havenbyte LLC. It provides stable, lower-cost access to GPT, Claude, Grok, and other leading models through compatible interfaces for applications, tools, and coding agents. Every request includes a complete record of token usage and cost. Individuals and teams can review consumption in real time, understand spend by request, and manage usage centrally. Canonical website: https://modelflare.dev/ ## Core capabilities ### Leading model access Modelflare connects models from multiple model vendors behind one gateway. The currently available model catalog and supported API formats are listed on the models and pricing page: https://modelflare.dev/pricing ### Model groups Users choose both a model and a model group. The live catalog shows the groups currently available for each model. Regular API keys can use an explicit primary group and ordered fallback groups; Smart API Keys evaluate eligible available groups according to the key's routing strategy. ### Usage and cost records Usage logs show the model, model group, request status, token usage, latency, and cost for each call. This gives individuals and teams a clear per-request record for real-time consumption review and centralized usage management. ### Compatible clients and coding agents Modelflare provides setup guides for compatible applications and coding tools: - Codex: https://docs.modelflare.dev/guides/clients/codex/ - Codex interactive quick setup: https://docs.modelflare.dev/guides/clients/codex/start/ - Claude Code: https://docs.modelflare.dev/guides/clients/claude-code/ - Claude Code interactive quick setup: https://docs.modelflare.dev/guides/clients/claude-code/start/ - Grok Build: https://docs.modelflare.dev/guides/clients/grok-build/ - OpenCode: https://docs.modelflare.dev/guides/clients/opencode/ - CC-Switch: https://docs.modelflare.dev/guides/clients/cc-switch/ The documentation site contains API-key, endpoint, model-name, and connection guidance: https://docs.modelflare.dev/ ### Technical article library The technical article library contains long-form implementation and operations guidance for AI API clients: - OpenAI-compatible API migration: https://modelflare.dev/articles/openai-compatible-api-guide - AI API gateway concepts and evaluation: https://modelflare.dev/articles/what-is-ai-api-gateway - Responses API versus Chat Completions: https://modelflare.dev/articles/responses-api-vs-chat-completions - SSE streaming, first output, and timeouts: https://modelflare.dev/articles/ai-api-streaming-guide - Reliable routing and request diagnostics: https://modelflare.dev/articles/reliable-ai-api-routing - Authentication, access, rate-limit, cancellation, and 5xx diagnosis: https://modelflare.dev/articles/ai-api-error-troubleshooting - Token usage and cost tracking: https://modelflare.dev/articles/ai-api-cost-tracking - API key security and workload cost controls: https://modelflare.dev/articles/api-key-security-cost-controls Browse all topics: https://modelflare.dev/articles ### Global availability Modelflare commercial API service is currently available in 171 supported countries, regions, and territories. The canonical list is maintained here: https://modelflare.dev/supported-regions ### Trust and security The Trust Center explains the standard API data flow, ordinary request-body handling, diagnostic retention, service-provider categories, safeguards, reliability boundaries, enterprise review materials, and responsible security reporting: https://modelflare.dev/trust ## Canonical pages - Homepage: https://modelflare.dev/ - About: https://modelflare.dev/about - Trust and security: https://modelflare.dev/trust - Models and pricing: https://modelflare.dev/pricing - Documentation: https://docs.modelflare.dev/ - Technical articles: https://modelflare.dev/articles - Supported countries and regions: https://modelflare.dev/supported-regions - Terms of Service: https://modelflare.dev/terms - Usage Policy: https://modelflare.dev/usage-policy - Privacy Policy: https://modelflare.dev/privacy-policy - User Agreement: https://modelflare.dev/user-agreement ## Naming Preferred product name: Modelflare Recognized variants: ModelFlare, Model Flare ## Programs and integrations - [Plugins and integrations](https://modelflare.dev/integrations): Installation paths for supported developer tools and applications. - [Project Plaza](https://modelflare.dev/projects): Published projects and partnerships. - [Affiliate program](https://modelflare.dev/affiliates): Applications, referral links, and eligible commissions. - [Project sponsorship](https://modelflare.dev/sponsorship): Eligibility, review, quota plans, and release conditions. - [Partner Program Terms](https://modelflare.dev/partner-program-terms) - [Top-up and Refund Policy](https://modelflare.dev/topup-refund-policy) The website supports multiple interface languages. Available translated versions are declared by each page's hreflang links; some policies have fewer translations. ## Published technical articles ### Kimi K3.1 Pro: Parameters, Capabilities, Prices, and Model Comparisons Canonical URL: https://modelflare.dev/articles/kimi-k3-1-pro-review-pricing Kimi K3.1 Pro is Moonshot AI's new flagship for coding, visual and 3D work, and multi-step research. Released on 23 October 2026, it combines a 3-trillion-class parameter scale, a 1,048,576-token context window, and an always-on reasoning mode. Its international API list price is $2 per million input tokens and $10 per million output tokens; the Highspeed version costs $4 / $20. This places the standard version alongside Claude Sonnet 5.5 on uncached input and output price, and at half Claude Opus 5.5's corresponding rates. Sources: official model list (https://platform.kimi.com/docs/models), Kimi pricing (https://platform.kimi.ai/docs/pricing/chat), and Claude pricing (https://platform.claude.com/docs/en/about-claude/pricing). The interesting question is what those prices buy for a real workflow. This guide separates model specifications, API controls, published evaluations, and calculated token budgets. The calculations are illustrative; Modelflare has not run a comparative model benchmark for this article. Provider results are attributed to their publisher. Facts and prices were checked on 23 October 2026, at approximately 18:03 UTC. The public Modelflare catalog at that time contained kimi-k3, but neither K3.1 model ID. This article introduces the official release; it does not announce K3.1 availability on Modelflare. 1. Which K3.1 model are we discussing? Item | Kimi K3.1 Pro | Kimi K3.1 Pro Highspeed | Official API ID | kimi-k3.1-pro | kimi-k3.1-pro-highspeed | Kimi Code ID | k3.1-pro | k3.1-pro-highspeed | Product positioning | Flagship reasoning and agent work | Same stated intelligence, faster output | Context window | 1,048,576 tokens | 1,048,576 tokens | Reasoning | Always enabled | Always enabled | Effort settings | low, high, max; default max | Same | Relative output speed | Reference | Approximately 2–3×, according to Kimi | Exact sustained tokens/second | Not established by this article | Not established by this article | API parameter restrictions | Shared within this pair | Same as standard Pro | Sources: model list (https://platform.kimi.com/docs/models), parameter reference (https://platform.kimi.com/docs/api/models-overview), and Kimi Code model configuration (https://www.kimi.com/code/docs/kimi-code/models.html). The coding subscription and the pay-as-you-go API use different IDs and billing systems. A display name such as “K3.1 Pro” is not an API identifier. Highspeed changes the latency and price choice. It does not establish a new quality tier. A tool-heavy job may spend most of its time compiling, retrieving documents, or rendering a scene; accelerating model output cannot accelerate those external operations. Measure time to a useful result, not only token generation speed. 2. Parameters and architecture: what is actually disclosed? The release blog (https://www.kimi.com/en/blog/kimi-k3-1-pro) identifies K3.1 Pro as a post-trained K3 model with a larger reinforcement-learning investment. It says full weights will be released by 4 November. A promised weight release is different from a downloadable checkpoint available on the article's check date. “3T-class” is a scale description, not an exact statement of 3,000,000,000,000 parameters. For architectural context, the previous K3 checkpoint has a detailed official model card (https://huggingface.co/moonshotai/Kimi-K3): Architecture field | Published K3 baseline; not a separately verified K3.1 checkpoint specification | Architecture | Mixture of Experts, or MoE | Total parameters | 2.8T | Activated parameters | 104B | Transformer layers | 93 | Dense layers | 1 | Attention composition | 69 KDA layers + 24 Gated MLA layers | Attention hidden dimension | 7,168 | Attention heads | 96 | Latent MoE dimension | 3,584 | Expert hidden dimension | 3,072 | Experts | 896 | Selected experts per token | 16 | Shared experts | 2 | Vocabulary | 160K | MoE makes the distinction between total and activated parameters important: the model can store far more capacity than it uses for each token. This does not mean a 104B activated model has the storage requirements of a 104B total model. Weight memory, activation memory, cache, parallelism, and serving overhead are separate deployment costs. For K3.1 specifically, this article does not assert an exact activated-parameter count, new expert count, layer revision, training-token total, knowledge cutoff, quantization format, final weight license, or minimum GPU configuration. Those require the corresponding release artifacts. The predecessor's architecture explains the family; it should not be silently relabeled as a new model card. 3. What its capabilities mean in practice Coding and long-running agents Kimi positions the model for repository fixes and larger engineering tasks, including web and game work. Its quickstart (https://platform.kimi.com/docs/guide/kimi-k3-1-pro-quickstart) describes use of Unity, Unreal Engine, Godot, and Three.js. These are tool-assisted workflows: the environment supplies files, engines, dependencies, and execution privileges. Our evaluation recommendation is to give the model an existing project with an actual defect, a bounded change request, and an independently defined acceptance condition. Check whether it locates the right component, preserves unrelated behavior, handles failed commands, and produces a reviewable result. Attractive generated code alone does not answer those questions. 3D creation and spatial reasoning The same quickstart highlights script-driven modeling with Blender and CAD tools, including interpretation of sketches and reference images. This is different from a general-purpose API that directly returns an image or a video file. The useful deliverable may be a script, parametric geometry, an editable scene, or an asset produced by external software. For a mechanical-part trial, specify units, dimensions, tolerances, and export format. For a visual asset, specify the camera, materials, polygon budget, and target renderer. A convincing preview and a manufacturable object have different acceptance conditions. These are suggested evaluation tasks, not results measured by Modelflare. Financial research and spreadsheets Kimi also emphasizes financial research and modeling. A useful evaluation should inspect provenance and spreadsheet behavior: do figures reconcile to the source period, do formulas remain linked, and does changing an assumption update dependent sheets correctly? Require a source trail and distinguish estimates from reported figures. The model's ability to assemble a workbook does not establish the correctness of an investment conclusion. For this article, the relevant capability is document analysis, numerical reasoning, and tool-assisted production of an auditable artifact. Visual context and structured workflows The K3.1 quickstart (https://platform.kimi.com/docs/guide/kimi-k3-1-pro-quickstart) documents image and video input, tool calls, and structured output. Its model-specific image restriction matters: use base64 data or an uploaded ms:// file reference; do not assume an arbitrary public image URL works because an older general vision example contains one. For application design, keep the roles separate: the model interprets evidence and proposes a tool action; the application validates arguments, executes the permitted operation, and returns its result. A JSON document can be syntactically valid while containing an incorrect claim. Schema validation and evidence checking solve different problems. 4. How to read the published evaluation results The following numbers are Moonshot's reported results, not an independent Modelflare reproduction. The release blog (https://www.kimi.com/en/blog/kimi-k3-1-pro) supplies these comparisons: Evaluation | K3.1 Pro | Reference | Scope | Terminal-Bench 4.0 | 65.7% | Opus 5.5: 66.4%; GPT-6 Astra: 58.2% | Terminal tasks under the publisher's evaluation setup | Kimi Code Bench 3.0 Long-horizon | 61.2% | K3: 55.95% | Internal engineering benchmark | BenchCAD, with tools | 92.7 | Opus 5.5: 96.2; GPT-6 Astra: 95.9 | 1,000-file subset; Voxel IoU, not task-success rate | The comparison supports a focused reading: K3.1 is a candidate for demanding agent and spatial workflows. It does not establish that it wins every coding task, is a drop-in replacement for another model, or delivers those results at your application's token budget. Tool access, reasoning effort, retries, and evaluator design all affect the result. There is also a launch-day source discrepancy. The English release blog's hard factuality evaluation reports 23.4% → 8.3% at claim level; the Chinese API quickstart (https://platform.kimi.com/docs/guide/kimi-k3-1-pro-quickstart) reports 23.35% → 6.85%. We do not merge these into one headline reduction or choose the more favorable number. Both describe deliberately difficult evaluation samples, not ordinary traffic error rates. 5. Official prices: international USD and domestic CNY All token prices below are per 1,000,000 tokens. A 1,048,576-token context window and a one-million-token billing unit are different quantities. International and domestic price lists are quoted separately; the CNY figures are not a currency conversion performed by Modelflare. International API list prices Model | Uncached input | Output | Cache read | Cache write, 5m | Cache write, 1h | Kimi K3.1 Pro | $2 | $10 | $0.20 | $2 | $4 | Kimi K3.1 Pro Highspeed | $4 | $20 | $0.20 | $4 | $8 | Kimi K3 | $3 | $15 | $0.30 | $3 | $6 | Source: international API pricing (https://platform.kimi.ai/docs/pricing/chat), including the price table embedded in the live page. These are the provider's published rates before applicable taxes. K3.1's standard input and output rates are one-third below K3. Highspeed doubles those two rates relative to standard Pro, while the cache-read rate remains $0.20. Domestic API list prices Model | Uncached input | Output | Cache read | Cache write, 5m | Cache write, 1h | Kimi K3.1 Pro | ¥13 | ¥68 | ¥1.30 | ¥13 | ¥26 | Kimi K3.1 Pro Highspeed | ¥26 | ¥136 | ¥1.30 | ¥26 | ¥52 | Kimi K3 | ¥20 | ¥100 | ¥2 | ¥20 | ¥40 | Source: domestic API pricing (https://platform.kimi.com/docs/pricing/chat). Against the domestic K3 list, standard Pro is 35% lower on input and 32% lower on output. Those percentages differ from the international comparison because the two price lists differ. Cache writes must not be counted twice Kimi's cache documentation (https://platform.kimi.com/docs/guide/context-caching) splits input into mutually exclusive uncached, cache-read, and cache-write portions. For Chat Completions, prompt_tokens is the total; the detailed counters identify cached and written tokens. A token counted as a cache write is not also charged as an ordinary uncached token. For an illustrative request with separate 5-minute and 1-hour write buckets: cost = (uncached_tokens * input_price + cache_read_tokens * read_price + cache_write_5m_tokens * write_5m_price + cache_write_1h_tokens * write_1h_price + billed_output_tokens * output_price) / 1_000_000 Use disjoint usage buckets when applying this formula. The default cache TTL is 5 minutes; the documented longer option is 1 hour. Keep stable instructions and tools at the beginning of a request and changing task content toward the end. A repeated prefix can be economical; merely making a prompt long does not create a saving. 6. Price comparison with other models This is an official list-price comparison in USD, not a comparison of Modelflare group discounts, coding subscriptions, or measured quality. Standard online inference is used; taxes, paid tools, regional premiums, and batch discounts are excluded. Model | Input / 1M | Output / 1M | Cache read / 1M | Price qualification | Kimi K3.1 Pro | $2 | $10 | $0.20 | Standard | Kimi K3.1 Pro Highspeed | $4 | $20 | $0.20 | Faster output tier | Kimi K3 | $3 | $15 | $0.30 | Previous flagship | Claude Sonnet 5.5 | $2 | $10 | $0.10 | Standard global routing | Claude Opus 5.5 | $4 | $20 | $0.20 | Standard global routing | GLM-5.3 | $1.40 | $4.40 | $0.26 | Z.AI international API | DeepSeek-V4.1-Flash-1017 | $0.30 | $1.20 | $0.006 | Peak hours | DeepSeek-V4.1-Flash-1017 | $0.15 | $0.60 | $0.003 | Off-peak hours | Sources: Kimi (https://platform.kimi.ai/docs/pricing/chat), Anthropic (https://platform.claude.com/docs/en/about-claude/pricing), Z.AI (https://docs.z.ai/guides/overview/pricing), and DeepSeek (https://api-docs.deepseek.com/quick_start/pricing/). The current official DeepSeek ID is deepseek-flash; the version behind it is shown in the table. Its peak hours are 01:00–04:00 and 06:00–10:00 UTC on weekdays excluding Chinese public holidays; other hours use off-peak rates. K3.1 Pro and Sonnet 5.5 have equal standard input/output rates, but Sonnet's listed cache reads cost half as much. Highspeed and Opus 5.5 match on the three columns shown; their cache-write prices differ. GLM-5.3 and DeepSeek Flash offer lower ordinary input/output prices. These observations identify budget candidates, not a universal quality ranking. Anthropic lists 5-minute cache writes at $2.50 for Sonnet and $5 for Opus, versus Kimi's $2 and $4 for Pro and Highspeed. Its 1-hour writes are $4 and $8 respectively. Z.AI lists cached-input storage as temporarily free. Match the caching policy as well as the headline token price when comparing repeated workflows. 7. A reproducible token-budget example Assume 100,000 input tokens and 10,000 billed output tokens. Output includes any billable reasoning; it is not a promise of 10,000 visible answer tokens. The cold case has no cache reads and uses standard input pricing, or an equal-priced Kimi 5-minute write. The warm case contains 90,000 cache-read tokens and 10,000 ordinary input tokens, with no new writes. Both exclude external tool costs and retries. Model / tier | Cold request | Warm request | Kimi K3.1 Pro | $0.300 | $0.138 | Kimi K3.1 Pro Highspeed | $0.600 | $0.258 | Kimi K3 | $0.450 | $0.207 | Claude Sonnet 5.5 | $0.300 | $0.129 | Claude Opus 5.5 | $0.600 | $0.258 | GLM-5.3 | $0.184 | $0.0814 | DeepSeek Flash, peak | $0.042 | $0.01554 | DeepSeek Flash, off-peak | $0.021 | $0.00777 | Modelflare calculation, not a benchmark. The chart uses the cold column above; it does not include latency or completion quality. For standard Pro, the cold calculation is 0.1 × 2 + 0.01 × 10 = $0.30. The warm calculation is 0.09 × 0.20 + 0.01 × 2 + 0.01 × 10 = $0.138. At the domestic prices, the same two requests are ¥1.98 and ¥0.927. Multiplying these fixed workloads by 1,000 gives $300/$138 or ¥1,980/¥927, provided the assumptions remain true for every request. The warm cost omits the earlier request that created the cache. For a whole-session budget, add that initial write, any TTL changes after expiry, and every subsequent request. Keep cost per accepted task alongside token cost: a cheap request that requires several retries and manual repair may not be a cheap completed workflow. 8. API controls and integration details Context and generation limits Parameter or behavior | K3.1 Pro / Highspeed | Context window | 1,048,576 tokens | max_completion_tokens default | 131,072 | Maximum configurable max_completion_tokens | 1,048,576, constrained by remaining context | When input plus requested output exceeds context | API reduces the generation allowance to remaining space | Input alone | Must be smaller than the context window | Truncation signal | finish_reason: "length" | temperature | Fixed at 1.0 | top_p | Fixed at 0.95 | n | Fixed at 1 | presence_penalty, frequency_penalty | Fixed at 0 | reasoning_effort | low, high, max; default max | tool_choice | auto, none, required | thinking | Do not carry the K2-specific switch into K3.1 requests | Sources: Chat Completions reference (https://platform.kimi.com/docs/api/chat), parameter reference (https://platform.kimi.com/docs/api/models-overview), and reasoning guide (https://platform.kimi.com/docs/guide/use-reasoning-effort). Fixed parameters should normally be omitted. Sending a different value can fail; changing a generic client's default temperature is an integration concern, not a model-quality problem. With 900,000 input tokens, only 148,576 tokens remain inside the advertised window before accounting for any further context consumption. Setting the output parameter to one million does not make those capacities additive. Applications should handle a length-limited answer explicitly instead of accepting a truncated JSON object or unfinished patch. A documented request body For the official domestic Chat Completions endpoint, POST https://api.moonshot.cn/v1/chat/completions, a bounded streaming request can use the following JSON. Authentication belongs in the client's secret configuration. This example was syntax-checked locally and was not sent to a model. { "model": "kimi-k3.1-pro", "messages": [ { "role": "user", "content": "Explain how to check a spreadsheet for inconsistent formulas." } ], "reasoning_effort": "high", "max_completion_tokens": 16384, "stream": true, "stream_options": { "include_usage": true } } The effort field is intentionally explicit, rather than relying on the model's default. This JSON is for Chat Completions; it is not a Responses or Messages request body. Kimi also documents Responses (https://platform.kimi.com/docs/api/responses) and Messages (https://platform.kimi.com/docs/api/messages), each with its own input and event contract. Reasoning, tools, and structured answers Preserve the full assistant message across turns, including reasoning_content and tool_calls; return each tool result using its corresponding tool_call_id. The thinking guide (https://platform.kimi.com/docs/guide/use-thinking-models) explains the preserved-reasoning contract. Reducing a previous turn to visible content can discard state the next step needs. Separate reasoning deltas from answer deltas in a streaming UI. Parse the final answer field for application data, and use the final usage record for accounting. A reasoning token stream does not itself establish that a tool ran or that a job completed. For extraction, Kimi documents response_format with json_schema and strict: true. Validate required fields and business rules after parsing. For tool calls, constrain permissions and handle failures in the application. For a length-truncated response, a retry may need a revised output budget or a smaller task; blind retries can repeat the same failure. 9. Kimi Code membership is a separate product The Kimi Code configuration page (https://www.kimi.com/code/docs/kimi-code/models.html) says standard Pro is available from Plus, with 1M context from Pro; Highspeed requires Pro. Legacy plan names are listed separately there. Use k3.1-pro or k3.1-pro-highspeed in that product, not the longer pay-as-you-go API IDs. Start a fresh session when changing from an older model, and keep the model and effort stable when possible. The provider warns that switching can disturb cached context and behavior. Membership capacity is not a fixed number of API dollars, and a monthly subscription should not be placed in the same column as a per-million-token rate. 10. Choosing a candidate for your workload Editorial workflow diagram. It illustrates evaluation criteria and is not a screenshot or a Kimi-generated result. Your priority | Candidate approach | What to measure | Large repository or multi-stage coding work | Evaluate standard Pro first | Accepted change, recovery from failures, total tokens | Interactive work where output wait dominates | Compare Pro with Highspeed | End-to-end latency and extra cost per accepted result | Repeated long prompts | Compare cache policy across Kimi and Claude | Cold start, read/write buckets, hit rate, TTL expiry | Large-volume economical inference | Include GLM-5.3 and DeepSeek Flash | Quality on your data, failure rate, peak/off-peak economics | CAD, scenes, or game assets | Evaluate K3.1 with the actual target tools | Geometry, editability, export compatibility, render result | Research and spreadsheets | Use source-linked tasks and independent checks | Evidence accuracy, formula linkage, numerical reconciliation | These are evaluation recommendations derived from the contracts and prices above. They are not hands-on product endorsements. Fix the input set, tool access, effort settings, stopping rules, and acceptance criteria before measuring models; otherwise a lower bill may reflect a different task rather than greater efficiency. 11. Modelflare availability and next steps The read-only Modelflare pricing response (https://modelflare.dev/api/pricing) checked at 18:03 UTC on 23 October listed kimi-k3 in kimi-award and kimi-stable. It did not list kimi-k3.1-pro or kimi-k3.1-pro-highspeed. No K3.1 routing, group price, successful request, or deployment capability is claimed here. Renaming the existing K3 ID in a client is not proof that the new model is available. For current access, consult models and pricing (/pricing). For integration background, see the OpenAI-compatible API guide (/articles/openai-compatible-api-guide), Responses versus Chat Completions (/articles/responses-api-vs-chat-completions), and API cost tracking (/articles/ai-api-cost-tracking). 12. Sources, method, and update record Primary references are linked beside the claims and tables they support. The factual sources are Moonshot's release and API documentation, its K3 model card, the other vendors' price lists, and Modelflare's public catalog. Prices were checked against both Kimi regional pages; the dynamic price-table data was read from the actual page rather than inferred from a search snippet. The original contribution is the disjoint-token cost worksheet, two worked scenarios, and workflow acceptance matrix. Artwork follows Modelflare's engineering cover system and contains no fabricated product UI or benchmark result. No live model call was made. Exact K3.1 checkpoint internals, independent speed measurements, and the conflicting factuality figures remain explicitly unresolved. 23 October 2026: First edition, covering the official K3.1 Pro and Highspeed launch, current regional prices, source-qualified parameters, comparison budgets, and Modelflare availability at the recorded check time. ### Apply Backpressure to AI API Workloads Canonical URL: https://modelflare.dev/articles/ai-api-backpressure Apply Backpressure to AI API Workloads: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits. The operating rule Apply Backpressure to AI API Workloads should be implemented as a reliability control contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are max_inflight, bounded_queue, caller_deadline, load_shedding. Apply backpressure before the AI dependency is saturated. Bound in-flight work per failure domain, keep queues small and deadline-aware, reject work that cannot finish within the caller’s remaining time, and reserve capacity for critical traffic. A long queue converts overload into slow and expensive failure rather than creating capacity. Draw the failure boundary first Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once. Reliability controls must be defined on attempts and terminal states, not on a vague notion of a request “working.” Preserve the caller deadline, distinguish overload from permanent errors, and make rejection, cancellation, retry, fallback, and settlement observable as separate outcomes. The owner record for this page is ai-api-backpressure; its frozen controls are max_inflight, bounded_queue, caller_deadline, load_shedding. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label. State and control contract: Apply Backpressure to AI API Workloads Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response. Control point | Fixed decision | Evidence to retain | admission_key | tenant_plus_workload_plus_route_class | identity + priority + selected_capacity_pool | inflight_limit | fixed_from_exercised_capacity | limit_version + active_count + saturation_test | queue_limit | bounded_items_and_wait_time | queue_depth + oldest_age + caller_deadline | deadline | propagated_and_checked_before_each_stage | received_deadline + remaining_budget + rejection_stage | shedding | cheap_explicit_overload_response | normalized_status + retry_guidance + no_upstream_attempt | fairness | per_tenant_or_priority_reservations | pool_allocation + rejected_identity + headroom | Exercise the critical transition The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result. remaining_ms = caller_deadline_ms - elapsed_ms queue_budget_ms = remaining_ms - minimum_service_budget_ms if inflight >= max_inflight: if queue_depth >= max_queue or queue_budget_ms <= 0: reject_overload_without_upstream_attempt() else: enqueue_with_deadline(queue_budget_ms) drop_expired_work_before_dequeue() Verify behavior under failure Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request. Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for admission_key: enforce tenant_plus_workload_plus_route_class and retain identity + priority + selected_capacity_pool. Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for inflight_limit: enforce fixed_from_exercised_capacity and retain limit_version + active_count + saturation_test. Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for queue_limit: enforce bounded_items_and_wait_time and retain queue_depth + oldest_age + caller_deadline. Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for deadline: enforce propagated_and_checked_before_each_stage and retain received_deadline + remaining_budget + rejection_stage. Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for shedding: enforce cheap_explicit_overload_response and retain normalized_status + retry_guidance + no_upstream_attempt. Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for fairness: enforce per_tenant_or_priority_reservations and retain pool_allocation + rejected_identity + headroom. Failure amplification to prevent Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions. unbounded_queue — Queue growth consumes memory, hides saturation, and completes work after the caller no longer needs it. deadline_ignorance — Starting an expensive request with insufficient remaining time wastes provider capacity and may trigger retries. global_pool — One noisy tenant or batch workload can occupy all slots and raise latency for interactive traffic. retry_after_without_capacity_model — Telling every rejected caller to retry at the same moment can create the next overload wave. Trip signals, denominators, and stop conditions Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins. Signal | Decision threshold | Action | inflight_utilization | reviewed_saturation_band | shed_low_priority_or_scale | oldest_queue_age | <_queue_wait_budget | reject_expired_work | deadline_expired_before_start | 0 | shrink_queue_or_raise_rejection | tenant_capacity_share | <=_configured_fairness_limit | isolate_pool | Modelflare boundary and limitations Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth. A queue is useful only when bursts are shorter than the available service headroom and callers can still wait. For steady overload, early rejection and load shedding are safer. Capacity limits must be derived from a realistic load test that includes streaming duration, tool loops, provider latency, cancellations, and retry behavior. Continue through the topic cluster Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field. Parent decision guide Next implementation guide (/articles/ai-api-rate-limit-handling) Current setup documentation (/docs) Frequently asked questions Is backpressure the same as rate limiting? No. Rate limits bound arrivals over time; backpressure also controls in-flight work, queues, deadlines, and downstream saturation. How large should the queue be? Use measured burst shape and deadline budget. A copied queue length has no reliable meaning for another workload. What should be rejected first? Use explicit product priority and fairness policy; never infer business value from model price alone. Sources and verification date Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state. Google SRE: Addressing Cascading Failures (https://sre.google/sre-book/addressing-cascading-failures/) Microsoft: Queue-Based Load Leveling pattern (https://learn.microsoft.com/en-us/azure/architecture/patterns/queue-based-load-leveling) Microsoft: Design for self-healing (https://learn.microsoft.com/en-us/azure/architecture/guide/design-principles/self-healing) ### Trace an AI API Request Across Every Attempt Canonical URL: https://modelflare.dev/articles/ai-api-request-tracing Production guide: Trace an AI API Request Across Every Attempt. It includes a deterministic artifact, failure boundaries, rollout checks, and source-qualified limitations. Decision first Trace an AI API Request Across Every Attempt is an explicit production contract, not an isolated code change. Define the success signal, terminal failure state, and rollback condition before moving traffic. The artifact below separates evidence from assumption and keeps every attempt attributable. Start with request, prove attempt under a deterministic case, and make privacy a release gate rather than a follow-up task. The reusable artifact Use this table as the review record. A row passes only when its evidence comes from the same request, test window, or configuration snapshot. Checkpoint | Evidence to capture | Pass condition | request | request_id,tenant_pseudonym,operation,accepted_at | The record joins to one logical request and one concrete attempt. | attempt | attempt_number,route_version,model,channel_pseudonym | The record joins to one logical request and one concrete attempt. | timing | connect,headers,first_event,first_output,idle,total | The value is exact, preserved, and compared at the wire boundary. | terminal | HTTP,provider_class,cancellation_source,retry_decision | The owner, source, checked date, and known limitation are recorded. | usage | normalized_input/output/cache_categories_and_provenance | Reserved, observed, and final values reconcile through one durable ledger. | privacy | allowlisted_attributes_only;no_prompt,secret,or_raw_arguments | The limit is explicit and the request fails closed when it is exceeded. | Worked example The example is synthetic and deterministic. Replace identifiers and thresholds with reviewed values from your own workload; never paste production secrets or customer prompts into a test fixture. trace_id: 7f3a...synthetic request_id: req_demo_001 operation: responses attempts: - attempt: 1 route_version: rv_42 model: model_alias timings_ms: { headers: 812, first_output: 1240, total: 3100 } terminal: upstream_503 - attempt: 2 timings_ms: { headers: 620, first_output: 980, total: 2400 } terminal: success usage: { input: 1800, output: 220, source: provider_normalized } Implementation procedure Freeze the current request, response, configuration, and observable baseline before changing behavior. Run one deterministic positive case and retain the complete client-visible result. Run the paired negative or limit case so failure behavior is proved rather than inferred. Join every attempt to one logical request ID, then record timing, terminal state, and usage without sensitive content. Roll out through a bounded cohort with explicit stop conditions; do not widen traffic merely because the happy path passed. Re-read the durable state and public behavior after the change, and execute the prepared rollback if any invariant fails. Failure modes The following failures invalidate the result even when the outer HTTP request appears successful: Prompts, credentials, raw tool arguments, or provider payloads are copied into high-cardinality telemetry. Parallel calls or streamed fragments are correlated by array position instead of a stable call identifier. The integration reads one convenience field and silently drops typed output items, tool calls, refusals, or partial results. Percentiles are published without the workload distribution, concurrency, failures, region, and sample window. Modelflare boundary Modelflare can centralize OpenAI-compatible routing, keys, groups, usage records, and failure handling, but a configured route is not proof that every upstream supports every optional field. Verify the selected model and channel with the native protocol path. Keep provider prices in the current pricing source, preserve explicit zero values, and treat the final durable settlement—not an estimate—as billing truth. Use the parent guide (/articles/ai-api-latency-metrics) for the broader decision boundary and the setup documents (/docs) for current client configuration. Publication checklist The main question is answered before background material. Every request field, status, metric, and formula has one named owner. Examples use synthetic identifiers and contain no credential or customer data. All supported locales preserve the same headings, tables, code, limits, and caveats. Volatile API contracts, model support, and prices are rechecked at T-1 day; an invalidated claim moves the publication date. Before the scheduled time, the public API, localized routes, and sitemap still exclude the article. Sources and verification date Sources were checked on 2026-08-07. They establish the external contract or operating principle; they do not prove support on an untested provider route. W3C Trace Context (https://www.w3.org/TR/trace-context/) OpenTelemetry HTTP semantic conventions (https://opentelemetry.io/docs/specs/semconv/http/) ### Make AI API Operations Idempotent Canonical URL: https://modelflare.dev/articles/ai-api-idempotency Make AI API Operations Idempotent: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits. The operating rule Make AI API Operations Idempotent should be implemented as a reliability control contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are logical_request_id, idempotency_key, request_digest, durable_result. Make side-effecting AI operations idempotent at the application boundary. Bind a caller-supplied idempotency key to one authenticated principal, operation, and canonical request digest; reserve the operation atomically; persist terminal success or failure; and return the same durable result on replay. Model call IDs are correlation data, not a substitute for this ledger. Draw the failure boundary first Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once. Reliability controls must be defined on attempts and terminal states, not on a vague notion of a request “working.” Preserve the caller deadline, distinguish overload from permanent errors, and make rejection, cancellation, retry, fallback, and settlement observable as separate outcomes. The owner record for this page is ai-api-idempotency; its frozen controls are logical_request_id, idempotency_key, request_digest, durable_result. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label. State and control contract: Make AI API Operations Idempotent Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response. Control point | Fixed decision | Evidence to retain | key_scope | principal_plus_operation_plus_idempotency_key | principal_id + operation + key | request_binding | canonical_digest_must_match | request_digest + schema_version | reservation | atomic_pending_record_before_side_effect | unique_constraint + created_at + owner | replay | same_key_same_digest_returns_durable_result | original_result_id + replay_count | conflict | same_key_different_digest_rejected | conflict_status + both_digests_without_payload | recovery | stale_pending_requires_owned_reconciliation | lease + side_effect_receipt + terminal_write | Exercise the critical transition The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result. BEGIN; INSERT INTO operation_ledger (principal_id, operation, idempotency_key, request_digest, status) VALUES (:principal, :operation, :key, :digest, 'pending') ON CONFLICT DO NOTHING; -- only the transaction that owns the new row may perform the side effect -- persist terminal result before returning it to the caller COMMIT; Verify behavior under failure Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request. Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for key_scope: enforce principal_plus_operation_plus_idempotency_key and retain principal_id + operation + key. Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for request_binding: enforce canonical_digest_must_match and retain request_digest + schema_version. Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for reservation: enforce atomic_pending_record_before_side_effect and retain unique_constraint + created_at + owner. Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for replay: enforce same_key_same_digest_returns_durable_result and retain original_result_id + replay_count. Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for conflict: enforce same_key_different_digest_rejected and retain conflict_status + both_digests_without_payload. Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for recovery: enforce stale_pending_requires_owned_reconciliation and retain lease + side_effect_receipt + terminal_write. Failure amplification to prevent Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions. memory_only_deduplication — Process memory cannot protect retries after restart, failover, or concurrent execution on another node. key_without_digest — Reusing a key for a different payload can return the wrong result or hide a caller bug. check_then_act_race — Reading for an existing record and writing later allows two workers to perform the same side effect. model_call_id_as_key — A model-generated call ID may change across retries and does not define the authenticated business operation. Trip signals, denominators, and stop conditions Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins. Signal | Decision threshold | Action | duplicate_side_effect_count | 0 | stop_operation_and_reconcile | idempotency_digest_conflicts | review_each_event | reject_and_alert_owner | stale_pending_age | <_owned_recovery_deadline | reconcile_with_external_receipt | replay_result_mismatch | 0 | disable_endpoint | Modelflare boundary and limitations Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth. Idempotency does not mean every model generation becomes identical. It means the protected business operation has one durable outcome for the same authenticated request. Read-only generation may be retried under a budget, while payments, messages, database mutations, and tool actions need operation-specific keys and reconciliation. Continue through the topic cluster Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field. Parent decision guide (/articles/ai-api-error-troubleshooting) Next implementation guide Current setup documentation (/docs) Frequently asked questions Is an HTTP POST automatically non-idempotent? HTTP method semantics and application idempotency are related but different; the ledger defines replay behavior for the business operation. Can the gateway invent the key? The layer that understands the business operation should define it. A gateway request ID alone may not identify the user-visible side effect. How are stale pending rows handled? Use an owned recovery lease and external receipt lookup; never blindly repeat the side effect. Sources and verification date Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state. RFC 9110: HTTP Semantics (https://www.rfc-editor.org/rfc/rfc9110) OpenAI API reference: request IDs (https://developers.openai.com/api/reference/overview) OWASP: Transaction Authorization Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Transaction_Authorization_Cheat_Sheet.html) ### Handle 429s Without Retry Storms Canonical URL: https://modelflare.dev/articles/ai-api-rate-limit-handling Production guide: Handle 429s Without Retry Storms. It includes a deterministic artifact, failure boundaries, rollout checks, and source-qualified limitations. Decision first Handle 429s Without Retry Storms is an explicit production contract, not an isolated code change. Define the success signal, terminal failure state, and rollback condition before moving traffic. The artifact below separates evidence from assumption and keeps every attempt attributable. Start with classification, prove retry_after under a deterministic case, and make terminal a release gate rather than a follow-up task. The reusable artifact Use this table as the review record. A row passes only when its evidence comes from the same request, test window, or configuration snapshot. Checkpoint | Evidence to capture | Pass condition | classification | 429_source:_application,gateway,provider,or_shared_pool | The record joins to one logical request and one concrete attempt. | retry_after | delta-seconds_or_HTTP-date_parsed_against_a_monotonic_deadline | The value is exact, preserved, and compared at the wire boundary. | jitter | bounded_random_delay_after_server_guidance | The limit is explicit and the request fails closed when it is exceeded. | attempts | one_end-to-end_attempt_budget_shared_by_every_layer | The limit is explicit and the request fails closed when it is exceeded. | concurrency | admission_reduced_before_retry_volume_grows | The limit is explicit and the request fails closed when it is exceeded. | terminal | final_429_returned_with_request_ID_and_no_hidden_retries | The owner, source, checked date, and known limitation are recorded. | Worked example The example is synthetic and deterministic. Replace identifiers and thresholds with reviewed values from your own workload; never paste production secrets or customer prompts into a test fixture. attempt_budget = 3 for attempt in 1..attempt_budget: result = call_once() if result.status != 429: return result if request.deadline_expired(): break delay = parse_retry_after(result.headers) ?? bounded_backoff(attempt) reduce_admission_concurrency() wait(min(delay + full_jitter(), request.remaining_time)) return terminal_429_without_another_retry() Implementation procedure Freeze the current request, response, configuration, and observable baseline before changing behavior. Run one deterministic positive case and retain the complete client-visible result. Run the paired negative or limit case so failure behavior is proved rather than inferred. Join every attempt to one logical request ID, then record timing, terminal state, and usage without sensitive content. Roll out through a bounded cohort with explicit stop conditions; do not widen traffic merely because the happy path passed. Re-read the durable state and public behavior after the change, and execute the prepared rollback if any invariant fails. Failure modes The following failures invalidate the result even when the outer HTTP request appears successful: Independent retry layers multiply attempts and turn a small 429 episode into sustained overload. Client cancellation does not reach queued work or the active upstream request, consuming capacity after the result is useless. A single text response is treated as proof of protocol compatibility across streaming, tools, schema, usage, and errors. Prompts, credentials, raw tool arguments, or provider payloads are copied into high-cardinality telemetry. Modelflare boundary Modelflare can centralize OpenAI-compatible routing, keys, groups, usage records, and failure handling, but a configured route is not proof that every upstream supports every optional field. Verify the selected model and channel with the native protocol path. Keep provider prices in the current pricing source, preserve explicit zero values, and treat the final durable settlement—not an estimate—as billing truth. Use the parent guide (/articles/ai-api-error-troubleshooting) for the broader decision boundary and the setup documents (/docs) for current client configuration. Publication checklist The main question is answered before background material. Every request field, status, metric, and formula has one named owner. Examples use synthetic identifiers and contain no credential or customer data. All supported locales preserve the same headings, tables, code, limits, and caveats. Volatile API contracts, model support, and prices are rechecked at T-1 day; an invalidated claim moves the publication date. Before the scheduled time, the public API, localized routes, and sitemap still exclude the article. Sources and verification date Sources were checked on 2026-08-07. They establish the external contract or operating principle; they do not prove support on an untested provider route. RFC 6585: Additional HTTP Status Codes (https://www.rfc-editor.org/rfc/rfc6585.html) RFC 9110: Retry-After (https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after) ### Haiku 5.5: Anthropic Is Turning Model Capability into a Division of Labor Canonical URL: https://modelflare.dev/articles/claude-haiku-5-5-division-of-labor The value of Claude Haiku 5.5 is not only that it answers faster and costs less, but that it begins to turn model calls into infrastructure that can be deployed at scale. Anthropic released Claude Haiku 5.5 on 7 October 2026. One name is easy to mix up: in Anthropic's public materials, the model name is Claude Haiku 5.5 and the model ID is claude-haiku-5-5. There is no official model named "Claude Hiya 5.5". This article uses Haiku 5.5 throughout. Anthropic's positioning is explicit. This is a small model for high-concurrency, low-latency, cost-sensitive work: classification, information extraction, summarization, context compression, database queries, browser actions, and agent subtasks. Anthropic says the average cost of running Haiku 5.5 is about 75% lower than Haiku 4.5. It is also the first Haiku model with an adjustable effort setting. That changes the question from "is this another stronger small model?" to a more practical one: when a model is cheap enough to be called in large numbers, what happens to the division of labor among models in an AI product? The Claude 5.5 family is forming a clear division of labor Looking only at the names, it is easy to treat Opus, Sonnet, and Haiku as three rungs on one capability ladder. A more accurate reading is that they are forming a division of labor across different workloads. Model | Better suited as | Typical tasks | Claude Opus 5.5 | Expert on complex problems and long-horizon planner | Complex agents, long coding sessions, hard reasoning, critical decisions | Claude Sonnet 5.5 | Default model for everyday work | Code changes, document generation, analysis, multi-step knowledge work | Claude Haiku 5.5 | High-frequency execution layer | Classification, extraction, summarization, compression, routing, browser actions, and subagent tasks | Anthropic's model overview uses a similar placement. Opus 5.5 is for long-running agent coding and knowledge work. Sonnet 5.5 emphasizes the balance of speed and intelligence. Haiku 5.5 is for high-concurrency, low-latency classification, extraction, and routing. This means Haiku 5.5's advantage does not have to show up as "stronger than a mid-size model at everything." It is more likely to show up somewhere else: whether it can turn many local tasks that were not worth automating, because of cost, latency, or throughput, into system steps that can keep running. A lower price changes how calls are made Haiku 5.5's official price has to be read as two ranges. For requests whose input prompt does not exceed 100K tokens, input is $0.10 per million tokens and output is $0.50 per million tokens. Above 100K tokens, those prices are $0.50 and $2.50. Request size | Input | Output | Cache read | Up to 100K tokens | $0.10 / MTok | $0.50 / MTok | $0.01 / MTok | Over 100K tokens | $0.50 / MTok | $2.50 / MTok | $0.05 / MTok | Two details are easy to miss. First, a lower unit price and a lower real cost are not the same thing. Anthropic's "about 75% lower average running cost" already includes the change in token use from the new tokenizer. Dividing the old and new price per million tokens does not tell you that a product bill will fall by the same ratio. Second, whether the model actually finishes the task changes the real cost. One call can be cheap, and the task can still be expensive if it needs retries, a fallback to Sonnet, or extra human review. Product teams should watch this measure instead: Cost per successful task = total model and tool spend to finish the tasks ÷ number of tasks finished successfully A comparison across architectures should also put retries, fallback, tool calls, and human takeover into the total bill. One of the most important changes in Haiku 5.5 is adjustable effort Haiku 5.5 is the first Haiku model with adjustable effort. That setting means model choice is no longer the only cost switch. A system can also change how much reasoning it spends inside the same model. Think of it as a car's driving modes: For simple classification and format conversion, use a lower effort and favor speed and low cost. For structured extraction and long-document compression, use a medium effort and balance quality against spend. For tasks that need multi-step judgment, raise the effort. If the task still fails, move up to Sonnet or Opus. That produces a new decision formula: Final result = model × effort × context × tools × retry policy Comparing models is then no longer only "which is stronger, Haiku 5.5 or Sonnet 5.5?" The further questions are: On the same task, which effort level is the better deal? Does the quality gain from a higher effort cover the extra cost? On a simple task, does a higher effort only add waiting time? Is it cheaper to let Haiku try once more, or to call Sonnet once? This is why Haiku 5.5 is better judged by task-level cost than by how one output feels. Agents may move from one large model to layered collaboration Many agents have been shaped like this: after the user makes a request, one large model plans, retrieves, calls tools, organizes the results, and writes the final answer. User request → large model plans → large model searches → large model summarizes → large model answers The shape is simple, and it also spends the same expensive model on every local action. For an agent that searches dozens of documents, processes hundreds of records, or keeps calling a browser, cost and latency add up quickly. Haiku 5.5 fits a layered shape like this: User request ↓ Sonnet or Opus splits the task and makes the plan ↓ Haiku 5.5 searches, classifies, extracts, summarizes, and makes single tool calls ↓ Sonnet or Opus reviews the results and makes the final decision In this shape, Haiku 5.5's value is not finishing the hardest task on its own. It becomes the agent's worker layer. It handles the local work that is most numerous, relatively structured, and possible to check. Work that fits Haiku 5.5 Route a user request into different workflows. Extract fixed fields from one document or a few documents. Classify tickets and set their priority. Compress a long conversation into the context the next agent turn needs. Deduplicate search results and write a first-pass summary. Perform one explicit browser action. Check code formatting, generate a simple test, or explain a local piece of code. Finish a short piece of work as a subagent of Sonnet or Opus. Work that should not go to Haiku 5.5 by default Complex projects that need long-horizon planning. Code changes that touch many files, many dependencies, and several rounds of feedback. Critical decisions where one mistake is expensive. Work that must keep a complex state across a long process. Tasks with no automatic check, where a person has to judge the result. The point is not to label a model "can" or "cannot." It is to judge the cost of failure. When a task can be checked automatically and retried after a failure, Haiku 5.5's price-to-result ratio is more attractive. When failure is expensive, Sonnet or Opus is still the steadier choice. How an ordinary developer can choose among the three Start with a simple route based on task complexity and the cost of failure: Task | Default | Move up when | Fixed output format that can be checked automatically | Haiku 5.5 | Repeated format errors, or a required field is missing | High call volume and sensitivity to latency | Haiku 5.5 | p95 latency or the failure rate crosses the product threshold | Summarization, compression, classification, or extraction | Haiku 5.5 | Cross-document reasoning or a context conflict appears | Ordinary knowledge work and code changes | Sonnet 5.5 | The task spans a long horizon or needs several rounds of planning | Complex agents and long-horizon coding | Sonnet 5.5 or Opus 5.5 | Tolerance for mistakes is very low, or the task needs deep reasoning | High-stakes judgment and final review | Sonnet 5.5 or Opus 5.5 | Decide from the cost of an error and from how well the result can be checked | This table is not a permanent answer. Before production, measure your own task set and find where each model separates on success rate, latency, and cost per successful task. 100K tokens is a price boundary to watch Haiku 5.5's advertised price is low, and the price rises sharply after 100K tokens. Teams building on long documents, code repositories, or long conversations cannot look only at the published starting price. A long-context workflow can be split into steps: Build a cache the first time the document is read. Use Haiku 5.5 to compress the context. Send Sonnet only the passages that matter to the current question. Save intermediate results as structured state. Do not resend the full history on every turn. This does two things. It lowers the chance of crossing the 100K price range, and it lets different models carry different work. The long-context value of Haiku 5.5 is therefore not measured only by "how many tokens it can read." The practical questions are: How much raw material has to enter the model? Which material should be compressed first? What is the cache hit rate? Is the price above 100K still acceptable? Can the information lost in compression make a later task fail? The Haiku 5.5 release also changes what evaluation should measure The official page already lists results for GDPval-AA, OSWorld, Humanity's Last Exam, Terminal-Bench, and others. Those results help a reader see the rough range of the model. A product team still needs tests on its own tasks. In a real system, the number worth watching is not one score on one benchmark. It is this set: Task success rate. Output quality. p50 and p95 latency. Input and output token counts. Retry rate. Tool-call error rate. Fallback rate. Human takeover rate. Cost per successful task. Pay particular attention to stability across repeats. One attractive answer to one prompt only shows that this run succeeded. It does not show that the model will reliably finish the same kind of task in production. A more reliable test looks like this: Prepare an independent sample set for each kind of task. Run them with the same system prompt, context, tool definitions, and region. Repeat each sample several times. Judge the results with rules, held-out tests, or a blind human review. Report the success rate and the confidence interval. List the worst cases and the failure types separately. That method moves evaluation from "which output was the best one?" to "can this system keep finishing the work?" What Haiku 5.5 means for an individual user An individual user may not feel the API unit price directly. Haiku 5.5's place can still be read from three angles. First, it fits fast, repeated work with a clear boundary: organizing text, extracting information, generating a structured draft, and handling a large number of small questions. Second, it does not have to replace every stronger model. Complex writing, long-horizon planning, difficult code, and work that must keep state across a long stretch still depend more on Sonnet or Opus. Third, the difference between models looks more and more like a difference in how the work is done, not a simple high-versus-low ranking. Choose a model by looking first at how often the task runs, how much latency it can take, what an error costs, and how the result can be checked. Look at single-call ability after that. What a product team has to recalculate is the economics of one unit of work Haiku 5.5 makes it easier to try steps that were not worth automating: One more layer of input classification. One more pass that compresses retrieval results. One more subagent dedicated to documents. One more low-cost check of the output. One more structured check before the final answer. Those extra steps increase the number of calls. If they lower the final failure rate, the cost of the product as a whole can still fall. That is why a single model price is not enough. A product team needs to compare: Cost of one call → total cost of one task → total cost of one successful task → total cost of one deliverable result When Haiku 5.5 is cheap enough, a system can trade several small tasks for a more reliable final result. That change affects how agents are designed, how a product's margin is structured, and which work a team decides to hand to a model. Closing: Haiku 5.5 is an architecture change The release of Haiku 5.5 can be read as a small-model upgrade. It can also be read as Anthropic pushing the shape of the model product forward. It puts three questions on the same decision sheet: How much intelligence does this task need? How much latency can this task bear? How much money is this task worth? Once model choice sits with task routing, effort, cache, retries, and human takeover, "which model is the strongest?" is no longer the only question. The more important question becomes: Which model should handle which call, and how do you get a reliable enough result at the lowest end-to-end cost? The point of Haiku 5.5 may be that this question is cheap enough, for the first time, to be worth asking at scale. Sources Anthropic: Introducing Claude Haiku 5.5 (https://www.anthropic.com/claude-haiku-5-5) Claude Platform Docs: Models overview (https://platform.claude.com/docs/en/models/overview) ### Recover Safely from LLM Stream Disconnects Canonical URL: https://modelflare.dev/articles/llm-stream-disconnect-recovery Recover Safely from LLM Stream Disconnects: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits. The operating rule Recover Safely from LLM Stream Disconnects should be implemented as a reliability control contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are first_event, partial_output, terminal_event, replay_policy. Treat a stream disconnect as a state transition, not an automatic retry signal. Before the first effective event, a bounded retry may be equivalent for a read-only request. After partial text, reasoning, or tool arguments, mark the result incomplete and restart only under an explicit replay policy; never splice two generations as if they were one authoritative response. Draw the failure boundary first Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once. Reliability controls must be defined on attempts and terminal states, not on a vague notion of a request “working.” Preserve the caller deadline, distinguish overload from permanent errors, and make rejection, cancellation, retry, fallback, and settlement observable as separate outcomes. The owner record for this page is llm-stream-disconnect-recovery; its frozen controls are first_event, partial_output, terminal_event, replay_policy. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label. State and control contract: Recover Safely from LLM Stream Disconnects Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response. Control point | Fixed decision | Evidence to retain | stream_identity | one_attempt_id_per_connection | logical_request_id + attempt_id + provider_request_id | first_effective_event | record_exact_transition | event_type + timestamp + bytes_received | partial_output | incomplete_not_success | last_event_type + buffered_item_ids + incomplete_marker | tool_arguments | decode_only_after_terminal_boundary | call_id + fragment_sequence + completion_event | replay | policy_depends_on_side_effect_risk | replay_decision + new_attempt_id + prior_attempt_link | billing | retain_each_attempt_and_final_settlement | attempt_usage + terminal_state + charge_record | Exercise the critical transition The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result. CONNECTING -> STREAMING -> TERMINAL_SUCCESS | | \ | | -> DISCONNECTED_AFTER_PARTIAL -> INCOMPLETE | -> CLIENT_CANCELLED -> CANCELLED -> DISCONNECTED_BEFORE_FIRST_EVENT -> RETRY_ELIGIBLE? Never concatenate output from two attempt IDs. Never execute partial tool arguments. Verify behavior under failure Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request. Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for stream_identity: enforce one_attempt_id_per_connection and retain logical_request_id + attempt_id + provider_request_id. Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for first_effective_event: enforce record_exact_transition and retain event_type + timestamp + bytes_received. Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for partial_output: enforce incomplete_not_success and retain last_event_type + buffered_item_ids + incomplete_marker. Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for tool_arguments: enforce decode_only_after_terminal_boundary and retain call_id + fragment_sequence + completion_event. Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for replay: enforce policy_depends_on_side_effect_risk and retain replay_decision + new_attempt_id + prior_attempt_link. Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for billing: enforce retain_each_attempt_and_final_settlement and retain attempt_usage + terminal_state + charge_record. Failure amplification to prevent Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions. blind_reconnect — A reconnect can create a second generation while the first attempt is still running or already billed. partial_success — Displaying truncated output as complete hides missing citations, JSON fields, tool arguments, or safety endings. cross_attempt_splice — Joining fragments from different attempts creates text the model never emitted as one response. partial_tool_execution — Parsing or executing an unfinished argument buffer can trigger an invalid or unintended side effect. Trip signals, denominators, and stop conditions Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins. Signal | Decision threshold | Action | disconnect_before_first_event_rate | workload_SLO_threshold | inspect_connect_and_header_phases | disconnect_after_partial_rate | workload_SLO_threshold | mark_incomplete_and_stop_auto_retry | stream_without_terminal_event | 0_success_classifications | reclassify_and_investigate | cross_attempt_fragment_count | 0 | disable_recovery_path | Modelflare boundary and limitations Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth. This method does not assume that a provider supports resumable generation or an event cursor. If the native API documents resumability, treat its cursor and retention behavior as a separate tested contract. Otherwise a new connection is a new attempt and may produce different content and cost. Continue through the topic cluster Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field. Parent decision guide (/articles/ai-api-streaming-guide) Next implementation guide (/articles/streaming-function-calls-sse) Current setup documentation (/docs) Frequently asked questions Can the UI simply reconnect? Only as a new attributable attempt under policy. It must not imply byte-for-byte continuation without a documented resume contract. When may a request retry? A read-only request before first effective output may be eligible within budget; partial output or tool side effects require stricter replay rules. What does the user see? Show an explicit incomplete or cancelled state and offer a deliberate restart rather than silently stitching output. Sources and verification date Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state. OpenAI: streaming Responses (https://developers.openai.com/api/docs/guides/streaming-responses) W3C Trace Context (https://www.w3.org/TR/trace-context/) Google SRE: Addressing Cascading Failures (https://sre.google/sre-book/addressing-cascading-failures/) ### Design AI API Timeouts by Request Phase Canonical URL: https://modelflare.dev/articles/ai-api-timeout-strategy Production guide: Design AI API Timeouts by Request Phase. It includes a deterministic artifact, failure boundaries, rollout checks, and source-qualified limitations. Decision first Design AI API Timeouts by Request Phase is an explicit production contract, not an isolated code change. Define the success signal, terminal failure state, and rollback condition before moving traffic. The artifact below separates evidence from assumption and keeps every attempt attributable. Start with connect, prove headers under a deterministic case, and make cancel a release gate rather than a follow-up task. The reusable artifact Use this table as the review record. A row passes only when its evidence comes from the same request, test window, or configuration snapshot. Checkpoint | Evidence to capture | Pass condition | connect | DNS,TCP,TLS_budget_and_failure_class | The limit is explicit and the request fails closed when it is exceeded. | headers | request_sent->response_headers | The limit is explicit and the request fails closed when it is exceeded. | first_output | headers->first_meaningful_output,separate_from_first_SSE_frame | The record joins to one logical request and one concrete attempt. | idle | maximum_gap_between_meaningful_stream_events | The limit is explicit and the request fails closed when it is exceeded. | total | absolute_request_deadline_inherited_by_all_child_work | The limit is explicit and the request fails closed when it is exceeded. | cancel | client_cancel_reaches_queue,gateway,upstream,and_tool_work | The record joins to one logical request and one concrete attempt. | Worked example The example is synthetic and deterministic. Replace identifiers and thresholds with reviewed values from your own workload; never paste production secrets or customer prompts into a test fixture. total_deadline_ms: 60000 phases: connect_ms: 3000 response_headers_ms: 10000 first_meaningful_output_ms: 25000 stream_idle_ms: 15000 rules: - child_deadline_must_not_exceed_total - client_cancel_propagates_immediately - timeout_records_phase_and_attempt Implementation procedure Freeze the current request, response, configuration, and observable baseline before changing behavior. Run one deterministic positive case and retain the complete client-visible result. Run the paired negative or limit case so failure behavior is proved rather than inferred. Join every attempt to one logical request ID, then record timing, terminal state, and usage without sensitive content. Roll out through a bounded cohort with explicit stop conditions; do not widen traffic merely because the happy path passed. Re-read the durable state and public behavior after the change, and execute the prepared rollback if any invariant fails. Failure modes The following failures invalidate the result even when the outer HTTP request appears successful: Client cancellation does not reach queued work or the active upstream request, consuming capacity after the result is useless. The next request assumes hidden provider state that the selected route does not store or cannot retrieve. Independent retry layers multiply attempts and turn a small 429 episode into sustained overload. Prompts, credentials, raw tool arguments, or provider payloads are copied into high-cardinality telemetry. Modelflare boundary Modelflare can centralize OpenAI-compatible routing, keys, groups, usage records, and failure handling, but a configured route is not proof that every upstream supports every optional field. Verify the selected model and channel with the native protocol path. Keep provider prices in the current pricing source, preserve explicit zero values, and treat the final durable settlement—not an estimate—as billing truth. Use the parent guide (/articles/ai-api-streaming-guide) for the broader decision boundary and the setup documents (/docs) for current client configuration. Publication checklist The main question is answered before background material. Every request field, status, metric, and formula has one named owner. Examples use synthetic identifiers and contain no credential or customer data. All supported locales preserve the same headings, tables, code, limits, and caveats. Volatile API contracts, model support, and prices are rechecked at T-1 day; an invalidated claim moves the publication date. Before the scheduled time, the public API, localized routes, and sitemap still exclude the article. Sources and verification date Sources were checked on 2026-08-07. They establish the external contract or operating principle; they do not prove support on an untested provider route. Go: Context (https://go.dev/blog/context) RFC 9110: Retry-After (https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after) ### Design Circuit Breakers for AI APIs Canonical URL: https://modelflare.dev/articles/ai-api-circuit-breaker Design Circuit Breakers for AI APIs: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits. The operating rule Design Circuit Breakers for AI APIs should be implemented as a reliability control contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are closed, open, half_open, failure_window. Place circuit breakers around a concrete route and failure class, not around the entire AI feature. Count only attributable, policy-defined failures; open before the dependency consumes the caller’s deadline; probe recovery with bounded half-open traffic; and keep capacity protection separate from model-quality fallback. Draw the failure boundary first Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once. Reliability controls must be defined on attempts and terminal states, not on a vague notion of a request “working.” Preserve the caller deadline, distinguish overload from permanent errors, and make rejection, cancellation, retry, fallback, and settlement observable as separate outcomes. The owner record for this page is ai-api-circuit-breaker; its frozen controls are closed, open, half_open, failure_window. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label. State and control contract: Design Circuit Breakers for AI APIs Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response. Control point | Fixed decision | Evidence to retain | breaker_key | provider_route_plus_protocol_plus_failure_class | route_id + endpoint + normalized_error_class | closed_state | rolling_window_with_minimum_sample | eligible_attempts + failures + window_bounds | open_state | fail_fast_without_upstream_attempt | decision_timestamp + reopen_at + returned_error | half_open_state | bounded_probe_concurrency | probe_count + outcomes + state_transition | fallback | same_contract_eligible_route_only | eligibility_reason + route_choice + terminal_state | operator_override | time_bounded_and_audited | actor + reason + expiry + restored_policy | Exercise the critical transition The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result. state = CLOSED if eligible_failures(window) >= threshold and samples >= minimum: state = OPEN reopen_at = now + cool_down if state == OPEN and now < reopen_at: return fail_fast if state == OPEN and now >= reopen_at: state = HALF_OPEN if bounded_probe_succeeds(): state = CLOSED else: state = OPEN Verify behavior under failure Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request. Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for breaker_key: enforce provider_route_plus_protocol_plus_failure_class and retain route_id + endpoint + normalized_error_class. Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for closed_state: enforce rolling_window_with_minimum_sample and retain eligible_attempts + failures + window_bounds. Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for open_state: enforce fail_fast_without_upstream_attempt and retain decision_timestamp + reopen_at + returned_error. Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for half_open_state: enforce bounded_probe_concurrency and retain probe_count + outcomes + state_transition. Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for fallback: enforce same_contract_eligible_route_only and retain eligibility_reason + route_choice + terminal_state. Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for operator_override: enforce time_bounded_and_audited and retain actor + reason + expiry + restored_policy. Failure amplification to prevent Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions. global_breaker — One unhealthy provider route can unnecessarily disable unrelated protocols, models, tenants, or regions. mixed_denominator — Counting client cancellations, invalid requests, and provider failures together produces meaningless transitions. half_open_stampede — Allowing normal concurrency during recovery can immediately overload a dependency that has only partially recovered. fallback_cascade — Moving all traffic to the remaining route can overload it and spread a local failure. Trip signals, denominators, and stop conditions Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins. Signal | Decision threshold | Action | eligible_failure_ratio | reviewed_window_and_minimum_sample | open_route_breaker | open_fail_fast_latency | <_caller_remaining_deadline | repair_local_decision_path | half_open_probe_concurrency | <=_configured_probe_limit | reject_extra_probes | fallback_headroom | >=_required_reserved_capacity | shed_load_instead_of_failover | Modelflare boundary and limitations Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth. Circuit breakers do not create spare capacity, repair incompatible routes, or make non-idempotent retries safe. They are one stateful guard in a larger timeout, retry-budget, load-shedding, and observability design. Thresholds must come from the workload and failure cost, not from a copied percentage. Continue through the topic cluster Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field. Parent decision guide (/articles/ai-api-error-troubleshooting) Next implementation guide (/articles/ai-api-timeout-strategy) Current setup documentation (/docs) Frequently asked questions What should key a breaker? Use the smallest stable failure domain that can be isolated and recovered independently, usually route plus protocol and normalized failure class. Does open always mean fallback? No. Failing fast or shedding load can be safer when no compatible route has reserved headroom. How is half-open tested? Permit a bounded number of probes, record every transition, and return to open on the defined failure condition. Sources and verification date Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state. Microsoft: Circuit Breaker pattern (https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker) Google SRE: Addressing Cascading Failures (https://sre.google/sre-book/addressing-cascading-failures/) Microsoft: Cloud Design Patterns (https://learn.microsoft.com/en-us/azure/architecture/patterns/) ### Claude Opus 5.5: Specs, Price, and Place in the Field Canonical URL: https://modelflare.dev/articles/claude-opus-5-5-review-pricing Claude Opus 5.5 was released on 22 September 2026. The public contract is on the model page and in the what's-new note: API id claude-opus-5-5, a 1M-token context window, 128K max output, adaptive thinking that stays on, default effort medium, and a list price of $4 per million input tokens and $20 per million output tokens. Anthropic's pages do not publish a parameter count or a training-compute figure. This article stays with the published spec, the published price card, and the published benchmark table. Checked conclusion: on Anthropic's list price, Opus 5.5 is the current Opus tier. Per token it is cheaper than Opus 5, and much cheaper than Fable 5.1. On the coding and knowledge-work rows Anthropic printed, its scores lead Fable 5.1, Opus 5, and, on most of those rows, GPT-6 Astra. Two printed rows go the other way: Astra leads Terminal-Bench-Science 0.1 and Zapier's AutomationBench. Modelflare's pricing API, read on 30 September 2026, includes claude-opus-5-5 at the official $4 / $20 unit prices. That row enables claude-award 0.25, claude-stable 0.315, and claude-premium 0.5. A key for this tier sends claude-opus-5-5. Live rates are on the pricing page (/pricing). Official behavior below is cited from the Opus 5.5 model page (https://platform.claude.com/docs/en/models/opus-5-5/overview), what's new (https://platform.claude.com/docs/en/models/opus-5-5/whats-new-opus-5-5), and the launch page (https://www.anthropic.com/claude-opus-5-5). The earlier Fable 5.1 review (/articles/claude-fable-5-1-review-pricing) is the place for that model's migration notes. Published specifications Item | Claude Opus 5.5 | Model id, Claude API | claude-opus-5-5 | Amazon Bedrock id | anthropic.claude-opus-5-5 | Google Cloud and Microsoft Foundry id | claude-opus-5-5 | Release date | 22 September 2026 | Context window | 1M tokens, both the default and the maximum | Max output | 128K tokens | Thinking | Adaptive, always on | Default effort | medium | Latency class on the model page | Moderate | Reliable knowledge cutoff | June 2026 | Parameter count | Not on the public model page | "Reliable knowledge cutoff" is Anthropic's name for the date through which the model's stored knowledge is most extensive. It is separate from the context window. A file you attach is still in the request. Fast mode is a speed switch on the same id, documented as a research preview on the Claude API only. The request sets speed: "fast" with the fast-mode-2026-02-01 beta header. Anthropic prices that mode at $8 / $40 per million tokens and describes output as up to 2.5× faster. The what's-new page says this fast mode is absent on Amazon Bedrock, Claude Platform on AWS, Google Cloud, and Microsoft Foundry. The same page lists the effort ladder as the control for thinking depth: low, medium, high, xhigh, and max. A manual thinking budget is rejected. The minimum cacheable prompt is 512 tokens. Prices Official list prices Figures are US dollars per million tokens. Opus 5.5's cache, batch, and fast-mode rows come from the what's-new pricing section. Opus 5's cache-write cell is the single "cache writes" number on the launch comparison table, which does not split the 5-minute and 1-hour tiers. Fable 5.1's cache read is the official figure already used in this blog's Fable 5.1 review. Sonnet 5 and Haiku 4.5 input and output prices are the comparison row on the Opus 5.5 model page. An em dash means this article did not take that cell from a primary price row. Price per 1M tokens | Opus 5.5 | Opus 5 | Fable 5.1 | Sonnet 5 | Haiku 4.5 | Input | $4 | $5 | $10 | $2 | $1 | Output | $20 | $25 | $50 | $10 | $5 | Cache read | $0.20 | $0.50 | $0.25 | — | — | Cache write | $5 for 5 minutes; $8 for 1 hour | $6.25 | — | — | — | Batch input / output | $2 / $10 | — | — | — | — | Fast mode input / output | $8 / $40 | — | — | — | — | The list-price move from Opus 5 is 20% on fresh input and output, from $5 / $25 to $4 / $20. Cache reads, which dominate long agent loops, move from $0.50 to $0.20. That is 60% off the Opus 5 cache-read price, and it also sits under Fable 5.1's $0.25 cache read. Anthropic's separate workload claim is that a typical run at default settings costs about 40% less than Opus 5, because the model also uses fewer tokens. That 40% describes their harness. A real bill still follows effort, cache hit rate, and tool rounds. Batch processing is half the new list price. Fast mode is double the new list price, on the Claude API only. Modelflare catalog on 30 September 2026 GET https://modelflare.dev/api/pricing on 30 September 2026 includes claude-opus-5-5. The same response also includes claude-opus-5, claude-fable-5-1, claude-sonnet-5, claude-sonnet-5-5, claude-haiku-4-5, gpt-6-astra, and gpt-5.6-sol. It does not include a Haiku 5.5 id. Unit prices below are that response's token prices before a group ratio. For the Claude rows they match the official list. Astra and Sol are tiered: the second pair applies when input context is above 272,000 tokens, and the long-context output price is the tier's own output price. The example column multiplies Claude rows by claude-stable 0.315 and OpenAI rows by openai-stable 0.08. Claude ratios on this read are claude-premium 0.5, claude-stable 0.315, and claude-award 0.25. OpenAI ratios are openai-premium 0.13, openai-stable 0.08, and openai-award 0.03. Opus 5.5 enables only the three Claude groups above. claude-tmp-cheap is a public Claude group, and this model row does not include it. Model | Catalog id | Unit input / output | User price on the stable group | Opus 5.5 | claude-opus-5-5 | $4 / $20 | $1.26 / $6.30 | Opus 5 | claude-opus-5 | $5 / $25 | $1.575 / $7.875 | Fable 5.1 | claude-fable-5-1 | $10 / $50 | $3.15 / $15.75 | Sonnet 5 | claude-sonnet-5 | $2 / $10 | $0.63 / $3.15 | GPT-6 Astra | gpt-6-astra | $10 / $50, then $20 / $75 | $0.80 / $4.00, then $1.60 / $6.00 | GPT-5.6 Sol | gpt-5.6-sol | $5 / $30, then $10 / $45 | $0.40 / $2.40, then $0.80 / $3.60 | Opus 5.5 on claude-stable is $4 × 0.315 and $20 × 0.315. Use the official table to compare labs. Use the catalog table to read a Modelflare bill you can send today. Haiku 4.5 is in the same catalog at a $1 / $5 unit price. On claude-stable that is $0.315 / $1.575. The model-page comparison still gives Haiku a 200K context window and 64K max output. Haiku 5.5 is not in the 30 September response. Haiku 4.5 remains the low end of the Claude line that is actually listed. Cache ratios on the catalog rows are a fraction of the input unit price: 0.05 for Opus 5.5, so the catalog cache read is $0.20 and the catalog cache write is $5. That write matches the official 5-minute tier. The official 1-hour write is $8 and is not a separate catalog price. Opus 5 and Sonnet 5 use 0.1, and Fable 5.1 uses 0.025. Settle cache, long context, and priority tiers from the pricing page and from the finished usage record. What changed versus Opus 5 and Fable 5.1 Behavior | Opus 5.5 | Opus 5 | Fable 5.1 | API id | claude-opus-5-5 | claude-opus-5 | claude-fable-5-1 | Context / max output | 1M / 128K | 1M / 128K | 1M / 128K | Thinking | Always on. disabled and a manual budget return 400 | On by default. disabled is accepted only at effort high or below | Always on | Default effort when the field is omitted | medium | high | high | Reliable knowledge cutoff | June 2026 | May 2026 | June 2026 | List input / output | $4 / $20 | $5 / $25 | $10 / $50 | Cache read | $0.20 | $0.50 | $0.25 | Forced tool choice | any and tool return 400 | Listed as a breaking change away from Opus 5 | Same 400 as Opus 5.5 | Cybersecurity safeguard | Hands the task to Opus 4.8 | The launch note does not put this fallback on Opus 5 | Fable-class safeguards | Biology and frontier-LLM safeguard | Hands the task to Opus 5 | The launch note pairs this fallback with Opus 5.5 | Same safeguard class | Opus 5.5 reads thinking blocks from Opus 5 and from earlier Opus, Sonnet, and Haiku models. It does not read thinking blocks from Fable or Mythos. On the Claude API, Fable 5.1 and Mythos 5.1 read Opus 5.5 thinking blocks, and the what's-new page says no other current model does. A block the target model cannot read is dropped before the model sees it, and a dropped block is not billed. Accounts created on or after 31 August 2026 get a stricter check: if the system prompt, the tool list, or an earlier message changed after an Opus 5.5 thinking block was produced, replaying that block returns 400. The migration guide's path is an append-only history, or a mid-conversation system message when instructions have to change. The launch page also says Opus 5.5 ships with the preserved-thinking control introduced on Fable 5.1, with EU AI Act watermarking, and with zero data retention, in line with earlier Opus models. How to read the intelligence scores The table is the one printed on the launch page. Anthropic says that, unless a footnote says otherwise, Opus 5.5 ran with adaptive thinking at max effort. Terminal-Bench 4.0 is the footnote: Opus 5.5 at xhigh, GPT-6 Astra at high, each as that model's highest reported score. Astra and Sol numbers are quoted as reported by OpenAI. Production safeguards were on. When they fired, cybersecurity tasks were completed by Opus 4.8, and biology plus frontier LLM-development tasks were completed by Opus 5. Anthropic says that pulls Opus 5.5's scores down. Benchmark | Opus 5.5 | Fable 5.1 | Opus 5 | GPT-6 Astra | GPT-5.6 Sol | Terminal-Bench 4.0 | 66.4% | 55.8% | 52.3% | 57.9% | 37.3% | FrontierCode v1.1 (Main) | 54.4% | 50.3% | 48.0% | 53.3% | 47.5% | CursorBench 4.0 | 57.8% | 51.8% | 46.6% | — | 41.7% | GDPval-AA v2.1 | 1846 | 1735 | 1708 | 1542 | 1588 | AutomationBench | 40.0% | 31.4% | 26.9% | 41.4% | 28.8% | Humanity's Last Exam, with tools | 67.7% | 65.6% | 63.6% | 57.2% | — | Terminal-Bench-Science 0.1 | 58.7% | 52.6% | 29.0% | 64.6% | 22.4% | OSWorld 2.0, partial | 81.8% | 80.7% | 74.0% | — | — | Chartography, with tools | 89.0% | 88.4% | 83.4% | — | — | Put the gaps next to the error bars they published. Terminal-Bench 4.0 is ±2.6 points for Opus 5.5 and ±1.6 to ±2 points for the other Claude models. The gap from 66.4% to Astra's 57.9%, and to Fable's 55.8%, is wider than those bars. Terminal-Bench-Science is ±3.5 to ±5 points per model. Astra's 64.6% against Opus 5.5's 58.7% sits near the width of those bars, with Astra ahead in the printed point estimate. AutomationBench was run by Zapier. Safeguard interventions counted as failures because the run had no fallback model. Anthropic says the practical score would be higher. The printed 40.0% against Astra's 41.4% is a close published result, with Astra ahead on the point estimate. GDPval-AA v2.1 is an Elo on real professional tasks across 44 occupations: 1846 for Opus 5.5, 1735 for Fable 5.1, 1708 for Opus 5, 1588 for Sol, 1542 for Astra. The same launch page prints a second pair of scores on the cost charts, taken at default effort medium: FrontierCode 54.6% and CursorBench 52.5%. The comparison table prints 54.4% and 57.8%. Keep them as two Anthropic figures at two effort settings. The chart caption also says that, at medium effort, FrontierCode beats Astra's top score of 53.3% at about a fifth of the cost per task, and that Terminal-Bench matches Astra at about 40% of the cost. Those are cost-per-task claims about Anthropic's harness. The sentence under the table is the one to keep: in Anthropic's own use, the gap between Opus 5.5 and Fable 5.1 is narrower than the scores, and benchmark margins at this level are a weaker guide to a real task. Customer quotes on the launch page are selected testimonials. They can suggest what to replay. They do not belong in the score table. Where it sits among current models Three checkable shifts landed on the same day. The work Anthropic describes as Fable 5.1 level for most tasks now has an Opus price card. List price is $4 / $20 against Fable 5.1 at $10 / $50. Cache reads are $0.20 against Fable's $0.25 and Opus 5's $0.50. Where a team's own replay agrees, Fable 5.1 stays the model for the tasks that still fail, and Opus 5.5 is the model that should run the loop. Where the replay disagrees, the table already shows the published exceptions: science-agent tasks and Zapier's business workflows, where Astra leads the printed numbers. The default moved with the price. Claude Code's model configuration, updated with this release, sets Opus 5.5 as the default for Pro, Max, Team, Enterprise, and the Anthropic API, and also for Claude Platform on AWS, Amazon Bedrock, and Google Cloud's Agent Platform. A default inside the agent product moves more traffic than a new row in a picker. The safeguard boundary moved down onto Opus. Earlier Opus models were where traffic went when a Fable classifier would not answer. Opus 5.5 launches with a similar set: cybersecurity tasks are completed by Opus 4.8, and biology plus frontier LLM-development tasks are completed by Opus 5. A log line that records the requested id claude-opus-5-5 can sit on a completion produced by a different model. Refusals return HTTP 200 with stop_reason: "refusal". Sonnet 5.5 is already in the 30 September catalog at the same $2 / $10 unit price as Sonnet 5. Haiku 5.5 is not in that response. The cheaper listed Claude cards are Sonnet 5.5, Sonnet 5 at $2 / $10, and Haiku 4.5 at $1 / $5. Haiku 4.5 remains on a 200K window. Sonnet 5.5's groups are in the Sonnet 5.5 review (/articles/claude-sonnet-5-5-review-pricing). This is a position you can audit from documents: list price, the Claude Code default, which model answers when a safeguard fires, and which rows of the vendor table Astra still leads. It is not a market-share claim. Integration constraints Requests that return 400 A client that only replaces the model string still hits four breaks documented for code that was running on Opus 5. The first three also apply to Fable 5.1. The shape below is the one the migration note tells you to send. The comments mark the variants that return invalid_request_error. model: claude-opus-5-5 thinking: {"type": "adaptive"} tool_choice: {"type": "auto"} output_config.effort: medium # 400: thinking type disabled, or enabled with budget_tokens # 400: tool_choice any, or tool_choice tool # 400 on the Claude API and Google Cloud: tool type computer_20251124 Depth replaces the old on/off thinking switch. Omit thinking, or send adaptive. Set effort explicitly. tool_choice auto and none remain valid, including on the token-counting endpoint. For schema-valid JSON, the migration note points to strict tool use or structured outputs, plus a prompt line that says when the tool applies. On the Claude API and Google Cloud, replace computer_20251124 with the computer_toolset_20260801 toolset and drop the old computer-use beta header. Amazon Bedrock still accepts computer_20251124 on Opus 5.5. Browser-use integrations that already use the toolset need no change for this break. For accounts created on or after 31 August 2026, a replay that edits the prefix in front of an Opus 5.5 thinking block is the fourth 400. Keep the conversation append-only. Calls that succeed and still change behavior Omitting effort now runs medium. The same omission on Opus 5 ran high. At a matched effort, Opus 5.5 tends to think more per turn than Opus 5, especially at xhigh and max. An effort value copied from an Opus 5 config is a new cost point. Leave room in max_tokens for the thinking blocks. Text written between tool calls comes back inside thinking blocks. At the default display value omitted, those blocks are empty. A UI that used to stream that text goes quiet, and the request still succeeds. Set a display value when the product needs the progress lines. A biology classifier runs beside the cybersecurity classifier. A prompt that pushes the model to copy its reasoning into the visible answer can be declined under reasoning_extraction. Chart and screenshot reading is sharper before any vision tool is added, according to the prompting guide, so older prompts that used to compensate for weak chart reading are worth replaying on their own. Which workloads to evaluate first Replay Opus 5.5 on the jobs the launch page is specific about: a multi-file change, a migration with the repository's own tests, a research memo graded against its sources, and a computer-use flow you can run twice. Record the requested model, the model that actually answered, effort, input tokens, cache-read tokens, output tokens, wall time, retries, and human edits. Anthropic's "about 40% less" and "about a fifth of the cost" are claims about their harness. They do not replace that sheet. Keep Fable 5.1, or an explicit fallback, on tasks the replay still loses. Plan for Opus 4.8 on traffic the cybersecurity safeguard will hand off, and for Opus 5 on biology and frontier-model-development traffic the safeguard will hand off. Keep Sonnet 5.5 or Sonnet 5 where the task fits a $2 / $10 card and does not need the Opus loop. On Modelflare, send claude-opus-5-5 for this tier. The 30 September row enables claude-premium at 0.5 when a failed call is expensive, claude-stable at 0.315 for ordinary production, and claude-award at 0.25 when the call can be retried. claude-tmp-cheap is not enabled on this model. On claude-stable the user price is $1.26 per million input tokens and $6.30 per million output tokens. Sources Launch page, 22 September 2026: https://www.anthropic.com/claude-opus-5-5 (https://www.anthropic.com/claude-opus-5-5) Model page, specs and the current comparison row: https://platform.claude.com/docs/en/models/opus-5-5/overview (https://platform.claude.com/docs/en/models/opus-5-5/overview) What's new, breaking changes and the price card: https://platform.claude.com/docs/en/models/opus-5-5/whats-new-opus-5-5 (https://platform.claude.com/docs/en/models/opus-5-5/whats-new-opus-5-5) Claude Code model configuration, default model: https://code.claude.com/docs/en/model-config (https://code.claude.com/docs/en/model-config) Modelflare pricing API, read 30 September 2026: https://modelflare.dev/api/pricing (https://modelflare.dev/api/pricing) Fable 5.1 review on this blog: Fable 5.1 review (/articles/claude-fable-5-1-review-pricing) ### GPT-6.1 Sol: Specs, Price, and Modelflare Access Canonical URL: https://modelflare.dev/articles/gpt-6-1-sol-review-pricing OpenAI released GPT-6.1 Sol on 29 September 2026. The API model ID is gpt-6.1-sol. The launch page describes it as an upgrade to gpt-6-sol: close to gpt-6-astra on complex coding, computer use, and professional work, at one-fifth of Astra's standard input and output prices. Cached input is $0.10 per million tokens, 5% of standard input and half of the gpt-6-sol cache read. For the hardest scientific research, the launch page still points to Astra. gpt-6-sol stays available. This article keeps three layers apart. The specification and the prices come from OpenAI's launch page and model page. The benchmark comparisons copy the deltas, costs, and conditions the launch page actually states. Modelflare did not rerun them. Groups and user prices come from GET https://modelflare.dev/api/pricing read on 30 September 2026. A vendor phrase such as "about one-fifth of the cost" describes OpenAI's own harness. What the sources support The short-context list price is $2 input, $0.10 cache read, $2.50 cache write, and $10 output per million tokens. Above 272,000 input tokens, input and cache are doubled and output is multiplied by 1.5 for the whole request, which is $4 / $0.20 / $5 / $15. The Modelflare price row from the same day writes that same pair in two pricing_rules tiers. request_rules on that row is empty, so the catalog unit prices are not multiplied again by Fast, Batch, Flex, or a regional premium. The public selectable groups are openai-award (0.03) and openai-stable (0.08). openai-premium is a public group, and this model is not attached to it. The model row also names openai-enterprise and non-public openai-first-topup. The public response does not include an enterprise ratio, so this article does not print an enterprise user price. Published specification Item | GPT-6.1 Sol | Model ID | gpt-6.1-sol | Release | 29 September 2026, OpenAI API | Context window | 1,050,000 tokens | Maximum input | 922,000 tokens | Maximum output | 128,000 tokens | Input to output | Text and images to text | Knowledge cutoff | 30 April 2026 | reasoning.effort | low, medium (default), high, xhigh, max | Efforts that are not accepted | none, minimal | Tool calling | Responses API | Chat Completions | Supported, without tools | Endpoints listed | Chat Completions, Responses, Batch | Stated as unsupported on the model page | Realtime, Assistants, fine-tuning, embeddings, image generation, video, speech | The model page also says the model supports US and EU data residency, and that Fast is unavailable with EU data residency. The Modelflare price row read for this article has no residency switch. Read residency from the model page, and keep it separate from a gateway parameter. Maximum input of 922,000 lines up with the 1,050,000 context window and the 128,000 maximum output: 1,050,000 − 128,000 = 922,000. Reasoning tokens occupy context and are billed as output tokens. max_output_tokens caps reasoning tokens, visible output, and formatting tokens together. When that cap is hit, the response status is incomplete and the reason is max_output_tokens. Official prices Short context and long context Amounts are US dollars per million tokens. The short tier is the model page. The long tier is the rule stated there: above 272,000 input tokens, input and cache are doubled, output is multiplied by 1.5, and the whole request uses the long tier. The Sol and Astra columns come from those models' pricing_rules in the Modelflare pricing response of the same day. Their short tiers match the published list prices. Per 1M tokens | 6.1 short | 6.1 long | Sol short | Sol long | Astra short | Astra long | Input | $2 | $4 | $2 | $4 | $10 | $20 | Cache read | $0.10 | $0.20 | $0.20 | $0.40 | $1 | $2 | Cache write | $2.50 | $5 | $2.50 | $5 | $12.50 | $25 | Output | $10 | $15 | $10 | $15 | $50 | $75 | The integer 272,000 stays on the short tier: the price-row condition is input_context_tokens <= 272000. From 272,001 upward the long tier applies, including the first 272,000 tokens. The 6.1 cache read is half of Sol. Standard input and output are one-fifth of Astra. The cache read is not one-fifth of Astra. Astra's short-context cache read is $1. Other official tiers The model page states three further official multipliers: Fast is 2× Standard, Batch and Flex are 50% below Standard, and regional processing adds 10% where it is available. The launch page also says that Ultrafast will be offered in Codex in the coming days, with up to 8× the token generation of standard speed. On the Modelflare price row of 30 September 2026, request_rules is empty, has_request_rules is false, and there is no Ultrafast model ID. Budget a Modelflare call with the two tiers above. Whether an official multiplier has reached a bill is answered by the usage record. How to read the benchmarks The launch page publishes deltas and costs, not a table of absolute scores that can be copied out. The table below keeps only comparisons the page states. The cost sentences describe OpenAI's harness. Modelflare did not rerun the tests. Evaluation | Comparison stated on the launch page | Condition | DeepSWE v1.1 | Matches Astra; 6.4 percentage points above Sol's best score | Lower reasoning effort, at about one-fifth of the cost | GDP.pdf | Above Opus 5.5 with fallbacks; approaches Astra | Across the reasoning settings tested; under half the cost of Opus, about one-fifth the cost of Astra | AutomationBench | 2.2 percentage points above Opus 5.5; 4.8 percentage points above Sol at the same setting | medium; about one-third the cost of Opus | OSWorld 2.0 offline set | 7 percentage points above Sol; within 2.1 percentage points of Astra | max; partial reward, v2026.08.08; under half the cost of Sol, about one-seventh the cost of Astra | Terminal-Bench Science 0.1 | More than doubles Sol's score; Astra remains highest, at 68.1% | max; average cost per task $5.47 for 6.1, $23.21 for Opus 5.5, $23.80 for Astra | Factuality | Share of answers with at least one factual error falls from 11.4% to 7.7% | low; within 1.9 percentage points of Astra | The factuality sample is de-identified conversations where a user had flagged an earlier error. The launch page says these prompts are deliberately difficult and are not typical traffic. The Terminal-Bench Science dollars are OpenAI's published average cost per task, not a Modelflare group price. Astra's 68.1% is still the highest scientific score, and the launch page says to keep the hardest scientific research on Astra. The alignment tests on the same page add a set of deliberately difficult failure rates. When the search tool is broken, the share of max-effort answers that do not tell the user is 2.1% for 6.1, 4.9% for Sol, 1.5% for Astra, and 28.7% for Luna. The launch page says no attempt to bypass an automated safety reviewer was observed, matching Astra and Sol. The detail is in the GPT-6.1 Sol system-card addendum. These rates are not failure rates on ordinary traffic. Modelflare prices and groups On 30 September 2026, gpt-6.1-sol has input_price 2, completion_ratio 5, cache_ratio 0.05, and create_cache_ratio 1.25. billing_mode is tiered_expr. Short-tier unit prices are $2 input, $0.10 cache read, $2.50 cache write, and $10 output. The long tier is $4 / $0.20 / $5 / $15. The endpoint types are openai, openai-response, and openai-response-compact. The public selectable groups that include this model are the two below. The group is chosen on the key. Both groups call the same model ID. Price table Group | Ratio | Tier | Input / 1M | Cache read / 1M | Cache write / 1M | Output / 1M | openai-award | 0.03 | Input up to 272,000 | $0.06 | $0.003 | $0.075 | $0.30 | openai-award | 0.03 | Input above 272,000, whole request | $0.12 | $0.006 | $0.15 | $0.45 | openai-stable | 0.08 | Input up to 272,000 | $0.16 | $0.008 | $0.20 | $0.80 | openai-stable | 0.08 | Input above 272,000, whole request | $0.32 | $0.016 | $0.40 | $1.20 | The catalog description of openai-award is price-priority routing for budget-sensitive, retryable development, testing, and flexible workloads, with a stated 65% daily cache-hit protection for eligible requests and compensation for a shortfall. The openai-stable description is everyday development, longer projects, and production use where stability matters, with the stated protection at 75% daily. Both sentences are the group-definition text. This article did not measure the hit rate separately. The model row also lists openai-enterprise. The same public response has no ratio for that group in group_definitions or group_ratio, so the table above leaves it out. Non-public openai-first-topup is ratio 0.015, described as unlocked when a single top-up reaches $20. It is not a public selectable group. When a key also has fallback groups, a request that leaves the campaign group is billed on the group that serves it. Public openai-premium is ratio 0.13, and this model's enable_groups does not include it. Three examples The first two rows fall in the short tier. The third row is 280,000 input tokens, so the whole request uses the long tier. The cache row assumes the usage record counts those input tokens as a cache hit. Request | List price | openai-award | openai-stable | 100,000 uncached input, 5,000 output | $0.2500 | $0.0075 | $0.0200 | 100,000 cache read, 5,000 output | $0.0600 | $0.0018 | $0.0048 | 280,000 uncached input, 10,000 output | $1.2700 | $0.0381 | $0.1016 | The first row is 0.1 × $2 + 0.005 × $10 = $0.25. The second is 0.1 × $0.10 + 0.005 × $10 = $0.06. The third is 0.28 × $4 + 0.01 × $15 = $1.27, and the group cells multiply by 0.03 or 0.08. Applying the short-tier unit prices to those 280,000 tokens would produce $0.0198 on award, which is not the price of this row. The live page is the gpt-6.1-sol pricing page (/pricing/gpt-6.1-sol). The full catalog is Models and pricing (/pricing). Retryable development and testing use openai-award. For everyday production and longer projects, the catalog describes openai-stable as that kind of default. The two ratios are 3% and 8% of the short-tier and long-tier catalog prices. They are not 3% and 8% of a price that has already been multiplied for Fast, Batch, Flex, or a regional premium. Calling it on Modelflare Create a key on openai-award or openai-stable. Send the model ID gpt-6.1-sol exactly. The canonical base URL is https://modelflare.dev/v1. The same API is also published at https://cf.modelflare.dev/v1 and https://origin.modelflare.dev/v1. The root domain is the canonical site. Tool calling uses Responses. The default effort is already medium. The request below sets it explicitly. export MODELFLARE_API_KEY="YOUR_MODELFLARE_API_KEY" curl -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "input": "Review this migration plan and list the three assumptions that most need a test." }' Chat Completions can send text without tools. The field name is reasoning_effort. curl -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-6.1-sol", "reasoning_effort": "medium", "messages": [ { "role": "user", "content": "Review this migration plan and list the three assumptions that most need a test." } ] }' POST /v1/responses/compact is listed on this model. It is a compact operation on gpt-6.1-sol, not a second model ID. Image input uses the ordinary content array on a user message. The difference between the two entrances is covered in Responses API versus Chat Completions (/articles/responses-api-vs-chat-completions). How usage meets the bill is covered in AI API cost tracking (/articles/ai-api-cost-tracking). Request differences from GPT-6 Sol The short-tier input and output list prices match gpt-6-sol, and the cache read moves from $0.20 to $0.10. Long-tier output is $15 on both, and the long-tier cache read moves from $0.40 to $0.20. gpt-6-sol remains in the catalog. Its public groups on this reading include openai-premium, which 6.1 does not. When switching to 6.1, pin the model name on the completed result. A result that lands on another ID is not this switch. Shapes that fail reasoning.effort: none reasoning.effort: minimal tools carried on a Chat Completions request none and minimal return HTTP 400. When moving from gpt-6-sol or gpt-6-luna, both of which accept none, compare again at low and remove none from the request. Tools, computer use, file search, and web search go through Responses. Chat Completions carries requests that have no tools. Checks before shifting traffic Pin gpt-6.1-sol. The model name on the completed result has to be that ID as well. Estimate the public price on openai-award or openai-stable, and choose the key group by the cost of failure. Replay one non-sensitive set at low, medium, high, xhigh, and max. none and minimal return 400. Put tools on Responses. The Chat Completions example carries no tools. Price one prompt at 272,000 input tokens and one at 272,001. The second uses long-tier unit prices for the whole request. Budget with the two catalog tiers. Fast, Batch, Flex, and the regional premium on the model page follow the amount that actually appears on the usage record. Read the cached-token count on the usage record before using the cache example above. Keep the hardest scientific research on gpt-6-astra. gpt-6-sol stays available. Sources Checked on 30 September 2026. Modelflare did not independently reproduce the benchmarks. Launch page: https://openai.com/index/introducing-gpt-6-1-sol/ (https://openai.com/index/introducing-gpt-6-1-sol/) Model page: https://developers.openai.com/api/docs/models/gpt-6.1-sol (https://developers.openai.com/api/docs/models/gpt-6.1-sol) The 6.1 section of the GPT-6 guide: https://developers.openai.com/api/docs/guides/latest-model (https://developers.openai.com/api/docs/guides/latest-model) System-card addendum: https://deploymentsafety.openai.com/gpt-6-1-sol (https://deploymentsafety.openai.com/gpt-6-1-sol) Modelflare pricing API: https://modelflare.dev/api/pricing (https://modelflare.dev/api/pricing) Pricing page: https://modelflare.dev/pricing/gpt-6.1-sol (https://modelflare.dev/pricing/gpt-6.1-sol) ### Claude Sonnet 5.5: Specs, Price, and Modelflare Access Canonical URL: https://modelflare.dev/articles/claude-sonnet-5-5-review-pricing Anthropic released Claude Sonnet 5.5 on 28 September 2026. The API model ID is claude-sonnet-5-5. It is the second model in the Claude 5.5 family. Anthropic places it beside Opus 5.5: well-scoped everyday tasks, bug fixes, and documents, slides, and spreadsheets go here; complex, open-ended work that needs sustained judgment stays on Opus 5.5. The list price matches Sonnet 5, at $2 per million input tokens, $10 per million output tokens, and $0.20 per million cache-read tokens. This article keeps three layers apart. The specification and the breaking changes come from the model page, the what's-new page, and the migration guide. The benchmark table is copied from the launch page. Modelflare did not rerun those evaluations. Groups and user prices come from GET https://modelflare.dev/api/pricing read on 30 September 2026. Vendor statements such as "up to 30% less" and "30%+ faster" describe Anthropic's own harness. What the sources support On the published list price, Sonnet 5.5 is the same card as Sonnet 5. Input and output are half of Opus 5.5. On the coding and knowledge-work rows printed on the launch page, it is well ahead of Sonnet 5. Terminal-Bench 4.0 at 70.6% is above the 66.4% printed there for Opus 5.5. CursorBench 4.0 at 55.5% and GDPval-AA v2.1 at 1844 sit next to Opus 5.5 and below it. Anthropic also says that, in their own use and in external testing, Opus 5.5 remains stronger at complex, open-ended work that needs sustained judgment. On 30 September 2026 the Modelflare pricing response already contained claude-sonnet-5-5. The unit prices match the official input, output, cache-read, and 5-minute cache-write rates. The public groups are claude-award, claude-stable, claude-premium, and claude-tmp-cheap. The main-site notice of 29 September 2026 named the groups open that day, claude-award and claude-stable. By this reading, the public list also included the latter two. Published specification Item | Claude Sonnet 5.5 | Claude API ID | claude-sonnet-5-5 | Amazon Bedrock ID | anthropic.claude-sonnet-5-5 | Google Cloud, Microsoft Foundry, Claude Platform on AWS | claude-sonnet-5-5 | Release date | 28 September 2026 | Retirement commitment | Not sooner than 28 September 2027 | Context / synchronous max output | 1M / 128K tokens | Batch API max output | 300K tokens, beta header output-300k-2026-03-24 | Input to output | Text and images to text | Thinking | Adaptive, on by default | Default effort on the Claude API | high | Default effort in the Claude apps and Claude Code | medium | Effort values | low, medium, high, xhigh, max | Reliable knowledge cutoff | June 2026 | Minimum cacheable prompt | 512 tokens | Tokenizer | Same as Sonnet 5, so the same text has the same token count | The model page labels latency as Fast. That is a comparison inside the current Claude lineup, not a measured tokens-per-second figure. The lowest setting that turns off up-front thinking is thinking: {"type": "between_tools"}, accepted only at low, medium, and high. xhigh and max go back to adaptive thinking. Setting temperature, top_p, or top_k away from the default returns 400. The launch page says Haiku 5.5 will follow in the coming weeks. This article does not price an ID that is not available yet. Official prices Amounts are US dollars per million tokens. The Sonnet 5.5 column is the model page. The Opus 5.5 column is the comparison table on the same launch page. The what's-new page says Sonnet 5.5 has the same prices as Sonnet 5, including prompt caching and batch processing. The batch discount is 50% on input and output. Per 1M tokens | Sonnet 5.5 | Opus 5.5 on the launch page | Input | $2 | $4 | Output | $10 | $20 | Cache read | $0.20 | $0.20 | Cache write | 5-minute $2.50; 1-hour $4 | $5 | Batch input / output | 50% of list | Not printed in that table | The launch page prints one Opus 5.5 cache-write figure, $5, without a 5-minute and 1-hour split. The Sonnet 5.5 model page splits the two. Against Opus 5.5 input and output, this card is half. Cache reads are $0.20 on both. Anthropic also publishes two workload statements: the same work usually takes fewer tokens, and most tasks cost up to 30% less; output generation is 30%+ faster than Sonnet 5. The cost charts add per-task comparisons at about one tenth, one fifth, and one fifteenth. Those results belong to their harness. A real bill still depends on effort, cache hits, tool rounds, and whether the call used batch. How to read the benchmarks The table below is the one printed on the launch page. A dash is an empty cell on that page. The Terminal-Bench 4.0 and CursorBench 4.0 cost charts use GPT-5.6 Sol because OpenAI had not published GPT-6 Sol on those two. This article does not copy a GPT-5.6 Sol chart point into the GPT-6 Sol column. Modelflare did not rerun the tests. Benchmark | Sonnet 5.5 | Sonnet 5 | Opus 5.5 | GPT-6 Sol | Terminal-Bench 4.0 | 70.6% | 10.3% | 66.4% | — | FrontierCode v1.1, Main | 46.2% (max); 52.1% (xhigh) | 42.4% | 54.4% | 49.3% | CursorBench 4.0 | 55.5% | 34.1% | 57.8% | — | GDPval-AA v2.1 | 1844 | 1449 | 1846 | 1487 | AA-Briefcase v1.1 | 1811 | 1359 | 1822 | 1483 | Humanity's Last Exam, with tools | 64.5% | 54.9% | 67.7% | — | OSWorld 2.1, partial | 80.1% | 57.0% | 81.8% | — | Chartography, no tools | 61.6% | 15.6% | 64.4% | 53.6% | The footnotes belong next to the scores. Opus 5.5 on Terminal-Bench 4.0 is at xhigh, which the launch page calls that model's highest score. On FrontierCode, Sonnet 5.5 at max scores below xhigh. The launch page explains that max more often runs a code review split across subagents, and that this produced a timeout or edits beyond the task, which the benchmark penalizes. Artificial Analysis ran GDPval-AA and AA-Briefcase on a pre-release deployment that had a bug which could degrade structured outputs. The launch page says the bug was later fixed and that any effect would understate Sonnet 5.5. The GPT-6 Sol cells for GDPval-AA, AA-Briefcase, and Chartography carry a further note: OpenAI had fixed a bug that hurt image understanding, and those outside scores may not yet have been updated. The cost charts on the same page are a different reading. Medium effort is the default in the Claude apps. On several benchmarks, Sonnet 5.5 at Low or Medium beats Sonnet 5's best score at about a tenth of the cost per task. High effort is the default on the Claude Platform. The launch page says that on FrontierCode at High it scores about 10 points above Sonnet 5 at the same setting, at about one fifteenth of the cost per task, and that it matches GPT-6 Sol's best score at about one fifth of the cost. On CursorBench at Low it exceeds Sonnet 5's best score at less than a tenth of the cost. On AA-Briefcase at Medium it beats Sonnet 5's best score at about one ninth of the cost. The printed 55.5% and 1844 are the points in the table. The cost sentences are another harness. Selected customer quotes on the launch page can suggest what to replay. They are not part of this table. The launch page also says this is the first Sonnet to beat Pokémon Red from screenshots alone. That is the vendor's product narrative, not a row in the table above. Modelflare prices and groups In the pricing response of 30 September 2026, claude-sonnet-5-5 has input_price 2 and completion_ratio 5, so output is $10. cache_ratio is 0.1, so a cache read is $0.20. create_cache_ratio is 1.25, so the catalog cache write is $2.50. The row has no billing_mode and no separate 1-hour write price. Do not multiply the official 1-hour $4, or the batch half-price, by a group ratio and treat the product as the posted charge. The usage record is the charge. The endpoint types are anthropic and openai. The group is chosen on the API key. The four public groups call the same model ID. Price table Group | Ratio | Input / 1M | Cache read / 1M | Catalog cache write / 1M | Output / 1M | Catalog description that day | claude-award | 0.25 | $0.50 | $0.05 | $0.625 | $2.50 | Price-priority routing for budget-sensitive, retryable development, testing, and flexible workloads | claude-stable | 0.315 | $0.63 | $0.063 | $0.7875 | $3.15 | Everyday development, longer projects, and production use where stability matters | claude-premium | 0.5 | $1.00 | $0.10 | $1.25 | $5.00 | Continuity and request success, billed at 50% of official list | claude-tmp-cheap | 0.06 | $0.12 | $0.012 | $0.15 | $0.60 | Limited-time promotional route | The arithmetic is the catalog unit price times the group ratio. On claude-stable, $0.63 is $2 × 0.315, and the cache write of $0.7875 is $2.50 × 0.315. The description is the text in the group definition. It is not a measured availability agreement. The same response also lists non-public claude-first-topup at ratio 0.03, described as unlocked when a single top-up reaches $50. It is not a public selectable group, so it has no row above. When a key also has fallback groups, a request that leaves the campaign group is billed on the group that serves it. Three examples Each shape is 100,000 tokens of the named input class plus 5,000 output tokens. The cache rows assume the usage record actually classifies those tokens as a read or a write. At small amounts, quota rounding can move the posted figure. Request | List price | claude-award | claude-stable | claude-premium | claude-tmp-cheap | 100,000 uncached input, 5,000 output | $0.2500 | $0.0625 | $0.07875 | $0.1250 | $0.0150 | 100,000 cache read, 5,000 output | $0.0700 | $0.0175 | $0.02205 | $0.0350 | $0.0042 | 100,000 catalog cache write, 5,000 output | $0.3000 | $0.0750 | $0.0945 | $0.1500 | $0.0180 | The list price on the first row is 0.1 × $2 + 0.005 × $10. Each group cell is that $0.25 times the ratio. The third row uses the catalog write price of $2.50, the official 5-minute tier, rather than the 1-hour $4. The live page is the claude-sonnet-5-5 pricing page (/pricing/claude-sonnet-5-5). The full catalog is Models and pricing (/pricing). When one failure is expensive, the catalog describes claude-premium as the continuity route. For everyday production and longer projects, the catalog describes claude-stable as that kind of default. Retryable development and testing use claude-award. claude-tmp-cheap is the limited-time promotional route at 0.06. Choose the group by the cost of failure, and read the ratio together with that description. Calling it on Modelflare Create a key on one of the four groups in the table. Send the model ID claude-sonnet-5-5 exactly. The canonical base URL is https://modelflare.dev/v1. The same API is also published at https://cf.modelflare.dev/v1 and https://origin.modelflare.dev/v1. The root domain is the canonical site. The Messages API is the entrance that sets effort. The request below leaves thinking adaptive and sets effort to medium. max_tokens has to leave room for thinking blocks. export MODELFLARE_API_KEY="YOUR_MODELFLARE_API_KEY" curl -sS https://modelflare.dev/v1/messages \ -H "x-api-key: $MODELFLARE_API_KEY" \ -H "anthropic-version: 2023-06-01" \ -H "Content-Type: application/json" \ -d '{ "model": "claude-sonnet-5-5", "max_tokens": 1024, "thinking": {"type": "adaptive"}, "output_config": {"effort": "medium"}, "messages": [ { "role": "user", "content": "Review this migration plan and list the three assumptions that most need a test." } ] }' Chat Completions is also listed on this ID. This example sends text only and does not set reasoning_effort. curl -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "claude-sonnet-5-5", "messages": [ { "role": "user", "content": "Review this migration plan and list the three assumptions that most need a test." } ] }' To choose an effort, use the Messages request above. The gateway currently turns Chat Completions reasoning_effort into a manual thinking budget with budget_tokens, and Sonnet 5.5 rejects a manual budget. When effort is omitted, the Claude API default is high. The difference between the two entrances is covered in Responses API versus Chat Completions (/articles/responses-api-vs-chat-completions). How usage meets the bill is covered in AI API cost tracking (/articles/ai-api-cost-tracking). Moving from Sonnet 5 The what's-new page lists five changes that make code already running on Sonnet 5 return 400, plus one change that leaves the request successful and changes the response shape. Requests that return 400 thinking: {"type": "disabled"} thinking: {"type": "enabled", "budget_tokens": N} thinking: {"type": "between_tools"} at effort xhigh or max tool_choice: {"type": "any"} tool_choice: {"type": "tool", "name": "..."} temperature, top_p, or top_k set away from the default tool type computer_20251124 on the Claude API and Google Cloud advisor tool using Opus 4.8, Opus 4.7, or Sonnet 5 as the advisor To turn off up-front thinking, send between_tools and keep effort at high or below. Keep tool_choice at auto or none. For schema-valid tool input, turn on strict, or move the schema to structured outputs, and say in the prompt when the tool applies. On the Claude API and Google Cloud, computer use moves to computer_toolset_20260801. Amazon Bedrock still accepts computer_20251124. The advisor tool accepts Mythos 5.1, Fable 5.1, Mythos 5, Fable 5, Opus 5.5, Opus 5, or Sonnet 5.5 itself. Advisor content comes back encrypted, and the client cannot read the text. Accounts created on or after 31 August 2026, 00:00 UTC, have one more 400 on the Claude API, Amazon Bedrock, and Google Cloud: replaying a Sonnet 5.5 thinking block after a change to the system prompt, the tool list, or an earlier message fails. Keep the conversation append-only. When instructions must change, use a mid-conversation system message. Behavior that changes on a successful response When effort is omitted, the API runs at high. The same word does not buy the same amount of thinking it did on Sonnet 5. The what's-new page says to sweep effort again rather than copy the old setting. A well-scoped tool loop can start at medium, and a longer or harder one can move to high. Latency-sensitive chat can start at medium or low. Progress text longer than a sentence or two between tool calls comes back inside thinking blocks. At the default display of omitted, the text of those blocks is empty, so a UI goes quiet between tools while the request still succeeds. Set display when the product needs those lines. With between_tools, that text comes back. A thinking block records the model that produced it. Sonnet 5.5 reads thinking blocks from Sonnet 5, Opus 4.8, Haiku 4.5, and earlier models. It does not read blocks from Opus 5, Opus 5.5, Fable, or Mythos. No other model reads Sonnet 5.5 thinking blocks. A block the target model cannot read is dropped before the model sees it, and a dropped block is not billed. A block Sonnet 5.5 produces also stays with the account that produced it. Replaying it from an unlinked account drops the block, and the request still succeeds. Refusal and fallback A declined request returns HTTP 200 with stop_reason set to "refusal". The categories named on the what's-new page are cyber, bio, frontier_llm, reasoning_extraction, and general_harms. The launch page says its cybersecurity capability is close to Opus 5, so this is the first Sonnet to ship with that class of cyber safeguards. Higher-risk cybersecurity tasks fall back to Sonnet 5. Biology safeguards match Sonnet 5. Routine software development and most life-sciences work sit outside those two narrow safeguards. The what's-new page says the Claude API beta default fallback retries cyber and frontier_llm on Sonnet 5, and does not retry bio, reasoning_extraction, or general_harms. That is Anthropic's documented behavior. Whether this Modelflare route performs the same fallback is answered by the model name on the response. Count refusals apart from transport failures. Whether a refusal is billed depends on its category. The launch page also says Sonnet 5.5 is available with zero data retention. Checks before shifting traffic Pin claude-sonnet-5-5. Also check the model name on the completed result. A safeguard fallback can answer as Sonnet 5. Confirm the key group is one of the four public groups in the table, and choose it by the cost of failure. Replay one non-sensitive set at low, medium, high, xhigh, and max. Record success, latency, input, cache reads, cache writes, output, and charge. Put effort in output_config.effort on the Messages request. Do not use Chat Completions reasoning_effort to set the thinking level for this model. Change clients that still send disabled, forced tool choice, a manual thinking budget, or computer_20251124 before raising volume. Keep conversations that replay thinking blocks append-only. Count HTTP 200 refusals apart from transport failures. Price one short prompt and one prompt that should hit cache, then read the posted amount. The official 1-hour write price and the batch half-price follow the usage record. Keep complex, open-ended work that needs sustained judgment on claude-opus-5-5. claude-sonnet-5 stays available. Sources Checked on 30 September 2026. Modelflare did not independently reproduce the benchmarks. Launch page, 28 September 2026: https://www.anthropic.com/claude-sonnet-5-5 (https://www.anthropic.com/claude-sonnet-5-5) Model page: https://platform.claude.com/docs/en/models/sonnet-5-5/overview (https://platform.claude.com/docs/en/models/sonnet-5-5/overview) What's new: https://platform.claude.com/docs/en/models/sonnet-5-5/whats-new-sonnet-5-5 (https://platform.claude.com/docs/en/models/sonnet-5-5/whats-new-sonnet-5-5) Migration guide: https://platform.claude.com/docs/en/models/sonnet-5-5/migration-guide (https://platform.claude.com/docs/en/models/sonnet-5-5/migration-guide) Modelflare pricing API: https://modelflare.dev/api/pricing (https://modelflare.dev/api/pricing) Pricing page: https://modelflare.dev/pricing/claude-sonnet-5-5 (https://modelflare.dev/pricing/claude-sonnet-5-5) ### Connect OpenCode to a Multi-Provider AI Gateway Canonical URL: https://modelflare.dev/articles/opencode-multi-provider-api-gateway Connect OpenCode to a Multi-Provider AI Gateway: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits. Choose the wire contract before the tool Connect OpenCode to a Multi-Provider AI Gateway should be implemented as a coding-agent integration contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are provider_id, provider_package, baseURL, explicit_model_map. Treat the OpenCode provider ID, provider package, base URL, credential entry, and explicit model map as one versioned contract. Choose the package that matches the endpoint actually used, test every declared capability on the selected model, and avoid advertising context, tools, or output limits copied from another provider. Separate client setup from gateway policy Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once. A useful integration page owns protocol selection, configuration precedence, proof cases, and team rollout. The live setup document remains the owner of copy-paste installation steps. This division follows a capability-tree model: the article explains why and how to verify; the documentation supplies the current commands. The owner record for this page is opencode-multi-provider-api-gateway; its frozen controls are provider_id, provider_package, baseURL, explicit_model_map. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label. Compatibility contract: Connect OpenCode to a Multi-Provider AI Gateway Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response. Control point | Fixed decision | Evidence to retain | provider_id | credential_and_config_ids_match | auth_listing + effective_config_provider_key | provider_package | package_matches_endpoint_contract | package_name + captured_endpoint | base_url | exact_versioned_api_root | resolved_baseURL + first_request_path | model_map | only_verified_model_ids_declared | display_id + upstream_model_id + probe_digest | capabilities | tools_and_limits_are_evidence_based | tool_probe + context_boundary + output_boundary | fallback | only_same_contract_routes_are_eligible | eligibility_matrix + selected_route + rejection_reason | Reference configuration or flow The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result. { "$schema": "https://opencode.ai/config.json", "provider": { "modelflare": { "npm": "@ai-sdk/openai-compatible", "name": "Modelflare", "options": { "baseURL": "https://modelflare.dev/v1" }, "models": { "": { "name": "" } } } } } Prove the complete coding-agent path Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request. Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for provider_id: enforce credential_and_config_ids_match and retain auth_listing + effective_config_provider_key. Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for provider_package: enforce package_matches_endpoint_contract and retain package_name + captured_endpoint. Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for base_url: enforce exact_versioned_api_root and retain resolved_baseURL + first_request_path. Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for model_map: enforce only_verified_model_ids_declared and retain display_id + upstream_model_id + probe_digest. Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for capabilities: enforce tools_and_limits_are_evidence_based and retain tool_probe + context_boundary + output_boundary. Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for fallback: enforce only_same_contract_routes_are_eligible and retain eligibility_matrix + selected_route + rejection_reason. Integration failures that a text demo hides Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions. provider_id_mismatch — A credential stored under one provider ID is invisible to a differently named configuration block. wrong_provider_package — Using a Chat Completions package for a Responses-only path, or the reverse, changes the wire contract. copied_capability_limits — Invented context, output, or tool flags create failures only after real workloads grow. fallback_without_eligibility — A cheaper or available route is unsafe when it cannot preserve the same protocol and required capabilities. Rollout evidence and stop conditions Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins. Signal | Decision threshold | Action | declared_model_probe_coverage | 100% | remove_unverified_model | provider_resolution_errors | 0 | rollback_config_and_credential_id | capability_contract_failures | 0 | disable_capability_or_route | fallback_contract_mismatch | 0 | remove_route_from_eligible_set | Modelflare boundary and limitations Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth. OpenCode’s provider schema is actively evolving. The example records the currently checked configuration family, while the live /docs/opencode page and official provider documentation own the exact syntax at publication time. T-1 verification must block publication if the schema or package contract changes. Continue through the topic cluster Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field. Parent decision guide (/articles/ai-coding-agent-api-gateway) Next implementation guide (/articles/openai-compatible-api-conformance-tests) Current setup documentation (/docs/opencode) Frequently asked questions Can any OpenAI-compatible provider package be used? Only when its endpoint contract matches the route. Chat Completions and Responses are not interchangeable package choices. Why declare models explicitly? A custom provider may not have a trusted catalog entry; explicit reviewed IDs prevent accidental discovery claims. Can fallback prioritize the cheapest model? Only after protocol, capability, policy, and workload eligibility have passed. Cost is not the first gate. Sources and verification date Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state. OpenCode: provider configuration (https://opencode.ai/docs/providers) OpenAI API reference overview (https://developers.openai.com/api/reference/overview) W3C Trace Context (https://www.w3.org/TR/trace-context/) ### JEV - A joint does not need a brain Canonical URL: https://modelflare.dev/articles/jev-system-one-joints Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-system-one-joints) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) Foundation models have crossed the line. Most people can no longer pose a task that sits beyond them. The questions you can think of, current models can mostly take. Comparing who chats a little better does not hold up a new product. The difference has moved to product definition. Jev is defined for this stage. It is TypeSafe AI's System One model. It does not generate free text. You give it a piece of state, text or program state, plus a set of closed questions. Each question comes back as a probability distribution. It does not sell chat. It sells the joints in a system. Nodes come in kinds. Some can be written down as rules, and then they are done. Some have to be reasoned out in the open and spoken through to a person. That is work for a brain. A batch remains in the middle. Rules cannot exhaust it. Calling a brain for it is waste. The joint sits on that batch. A bend is enough When you walk, the knee does not ask the brain. Hard ground, it bends a little. A steep slope, it holds. No explanation, and no draft. A joint does not need intelligence. A joint needs instinct. Instinct sits above rules. What can be exhausted goes to rules: the state matches, take that branch; it does not match, stop. What the rules still cannot decide, but a practiced person leans toward at a glance, goes to instinct. When the reason has to be laid out, and someone has to be spoken to, call intelligence in. Do not expect one question to do the work of both rules and a brain. Jev stays in that middle layer. The question is closed, and the shape of the answer is closed. The probability may spread. A spread means it is unsure. It only assigns probability inside the answers you gave it. The official line on a suitable question is tight: a judgment a person who knows the domain can make in a second while looking at the context. Anything that takes a long analysis does not belong as one question. Split it. This is the same kind of work as a general classifier from before GPT. The semantic understanding is better, and it is faster. It can read a sentence that takes the long way around, and it can read an intention tucked into program state. Do not take it to chat. The official name for the position is System One. It borrows Kahneman's System 1: fast, intuitive. The slow thinking that writes its steps out for a person to read is not in this position. The name comes from William Stanley Jevons. After coal and steam engines became more efficient, demand did not fall. It rose. The official use of the name says that once judgment is cheap, the number of nodes in a system that need a judgment grows with it. Only these three questions There are only three question types. Yes-no, called Noul. A sentence that true or false can close. What comes back is the probability of "yes", from 0 to 1. Code thresholds that number. Past the line, it runs. Short of the line, it stops, or it hands the case to a person. Single choice, called Choice. The options are fixed in advance, at most 255. Each option can be followed by a sentence saying what it means. What comes back is the whole distribution, plus the option with the highest probability. It does not add anything that is not on the list. Ordered levels, called Score. From 2 to 10 levels. Ranked from low to high, each level written as a concrete situation. Do not hand it only a number. The landing point may sit between two levels, and the probability of every level is still there. If the business needs an integer, code collects it. All three can be asked in the same request. The same state, each question on its own. A yes-no does not draft the single choice. Which question to use, and which to drop, is up to the code. Adding questions barely adds time. What you add is the question's own tokens. Input is billed by token. The extra cost is that small piece. Sampling is not the chat kind. The official account is a new architecture, parallel sampling. One pass collects every output. Not token by token. Chat models are slow because one word waits on the next, and the output gets expensive along the way. Jev does not grow one word at a time. The training method is called RLCD, Reinforcement Learning for Calibrated Decisions. The target is calibration: the confidence it reports has to line up with how often it is actually right. "Reads like a person" is not the target. Input is $0.042 per million tokens. Output is free. End to end is about 70ms to 500ms. Seventy milliseconds happens, and five hundred milliseconds happens. The official judgment on intelligence is stated alongside that: on System One tasks, about the same as current large models, and one to two orders of magnitude faster. That "about the same" holds only for closed judgments. Using it to write an article, or to talk an unclosed question all the way through, is not inside that claim. Where the rules stop Nodes of this shape are not rare. Classification, routing, scoring, review, risk control, plus the judgment nodes inside an agent, all have it. Which queue a ticket enters, which stretch of code the next hop takes, how hot a complaint is, whether one sentence is allowed through, whether an operation looks like the exception that should be stopped, whether a tool at a fork is handed over. All of these can be asked quickly, and all of them can be closed. Rules can eat the stretch that is easy to write. Once exceptions pile up, conditions tangle. Change one, and the ones beside it may fail with it, so later nobody dares to touch them. Some people throw the whole state at a model that can chat and ask it for a sentence of advice. A chat model can do that. The node only needs a lean. You still have to parse the sentence into a branch, and try again when that fails. Time is spent on the shape of the text. A brain is allowed. This node does not deserve one first. Jev takes this stretch. A closed question goes in, a distribution comes out. Whether it crosses the line, and whether it escalates, is written in the code beside it. How the flow moves is still the code's business. A refund splits into three questions: whether a refund is being asked for, whether the ask is a refund, a change of ticket, or only a question about the rules, and which level the anger lands on. Noul, Choice, and Score, looking at the same state. Sentences written into the policy stay with the rules. What the rules do not match is read from the distribution. The reply to the customer is not its job to write. An agent splits the same way. The sentence a person finally reads is left to a model that can speak. The judgments in the middle should not wake a brain at every step. Wake it once, and that step runs at the cost of a chat. Swap the judgment for a joint, and the reply that goes out can stay as it is. Writing needs another model It cannot write an article. It does not generate text. Put the topic and the tone into the state, and it still only answers the closed questions you wrote. Whether something is off topic, that it can judge. The sentences have to come from another model. Do not press an open question into a single choice. "What should we do next" has no cell in Choice until you have finished listing the ways. Split first. What can be exhausted stays with rules. What instinct can answer, write as these three questions. What needs reasoning laid out, and a person spoken to until the words are finished, that is when intelligence comes in. If it cannot be split, it is not a joint yet. Some people hear System One as a downgrade. Rules are lower. System One is the instinct above the rules. It is fast, and it does not write an explanation. It still has to understand the meaning, or the rules would have been enough. Above that is intelligence. The official intelligence claim is "about the same" on closed tasks, and the speed claim is one to two orders of magnitude faster. With that speed, a call can sit inside a loop instead of waiting out one chat. Once foundation models have flattened the tasks most people can pose, a product has to close into a definite interface. Jev closes into a joint. Chat is not for sale. The nodes where rules cannot be finished, and where calling a brain would be waste, are its place. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://evals.typesafe.ai/ (https://evals.typesafe.ai/) https://docs.typesafe.ai/primitives (https://docs.typesafe.ai/primitives) Series Next: JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) ### JEV - Why a Jev that cannot chat fits an agent Canonical URL: https://modelflare.dev/articles/jev-agents-without-chat Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-agents-without-chat) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) What an agent is often missing is not another stretch of long text. It is a judgment that can enter control flow directly. The usual ones are whether this alert stays closed, whether this invoice gets paid, whether this trace needs a person, and whether the next customer-service sentence should be escalated. These nodes are not there for the model to write an analysis. Code needs a value it can compare and branch on. That is what Jev does. TypeSafe's formulation means a function call of frontier intelligence. State comes in. A typed probabilistic decision goes out. It does not generate strings. The sentence for a person, and the judgment for a program, are not the same exit. Fix the output first The output of current large models is a string. Software that needs to continue still has to parse it, validate it, and guard against drift. An extra field appears. The paragraph looks complete, and the enum does not match. A person in a chat box notices and asks again. Code does not notice, and walks on with the error. Buried several calls deep, a smarter model cannot put it back. Jev fixes the output space in advance. You say what it is allowed to return, and it only assigns probability inside that space. A type error cannot happen, mathematically. It cannot hand back a shape you did not define. That is not the same thing as the judgment being correct. The probability can lean the wrong way, and the option can be the wrong one. Type safety closes the shape, not right and wrong. Once the shape is fixed, the branches after it can be written. You do not have to fish a value out of a paragraph first. There are only three questions. Noul is yes-no. The model gives a probability from 0 to 1. Near 1 is yes. Near 0 is no. Near 0.5 is unsure. It does not attach a separate confidence. The probability is the signal. Choice is single choice. You list the options first, with a cap of 255. What comes back is the chosen option, the distribution over every option, and a confidence. Code uses that distribution to decide whether it acts or hands the case to a person. Score is levels. The levels run from 2 to 10, and you write what each level means. What comes back is a score, the distribution over the levels, and a confidence. The score can land between two levels, as a position on the scale. A yes-no is written as a condition. Single choice and levels are different: the first looks at which option it landed on, the second thresholds the score. A good question is still narrow. A judgment that a person who knows the domain can make in a second after reading the material is the kind to give it. Whether the customer is asking for a refund is that kind of question. Reading the whole letter and then deciding the best action is not. The second sentence wants slow reasoning. Split it, then ask. The split questions face the same state, independent of each other, and come back together in one request. The weights stay in code. When policy changes, you change the numbers. You do not rewrite the whole prompt. The fork is what stalls people What actually stalls people is often a node of this shape. An alert arrives with records the machine already has, and the conclusion is close it, send it to an analyst, or isolate now. An invoice hangs on an order and a delivery record, and someone has to decide pay, hold, or send it back. A customer-service agent has finished, the tool calls are all in the trace, and someone has to decide whether this trace gets looked at, and how soon. The customer writes again, and the thread and the account state are both there. How the next sentence should continue, and whether it should escalate, is the same shape of question. The part rules can freeze is code. What is brittle is the exceptions that cannot be finished. The receipt matches, the reason is vague, and one step in the trace looks odd. Handwritten logic shatters on these. You stuff the whole policy into one prompt and let the model think it through once, and you write fewer conditionals. The exit turns back into a piece of text. For text to enter control flow, you write another layer of parsing. That layer can be wrong on its own. TypeSafe's workflow eval measures this way of connecting. The eval splits a task into many narrow questions. What code can decide, code decides. The model only answers the judgments code cannot settle. The four public flows are a security incident, agent-trace observability, invoice handling, and customer service. Under the same flow, a model on the workflow is more accurate than stuffing the whole policy into one prompt, and it costs less and takes less time. Averaged across the four tasks, the models under test move in that direction. The actions after that follow the probability, not a label that has been slammed shut. The result that leaves the system is still one discrete action. The engineering in the middle has to be done the same way every time. The eval measures closeness This eval does not argue with you about whether the flow itself was written wrong. It assumes the harness is right. The reference answer is not a gold label marked by hand, question by question. GPT-6 Astra and Claude Fable 5.1 answer every question under high thinking, and the two answers are averaged. Other models use the vendor's default reasoning. Only after the flow is fixed can they be compared. The eval compares how close a model comes to those two large models' judgments, plus speed and cost. So the chart does not prove that Jev understands the business better than Astra. Astra and Fable are the ruler here. Jev has to come close to their judgments, and it also has to open a gap on latency and cost. If the questions were split wrong, a closer ruler does not help. Splitting the questions is the work of the person who writes the flow. Jev sits far out on the frontier of "fast and cheap". The higher multiples on the official site, about 193.6 times faster and 444.6 times cheaper, are the high end of this eval. That multiple is not every call. The calls here are closer to the automation load you would actually ship. What it comes close to is the probability on those narrow questions after the split, not the writing of a long analysis. Jev does not write that long analysis. A hot path cannot wait Latency is hard for an agent. A person can still wait three seconds. Once layers wrap layers, they cannot. The same chain also has retrieval, a database write, and the next call, and every hop spends the same time. Inside 70 to 500 milliseconds, a judgment can sit on a hot path. Outside that range, the call stack treats it as blocking. Talking to a person, current frontier models commonly take from 3 seconds to more than 300 seconds end to end. That speed makes sense for a copilot, or for a coding agent with a person watching. As a condition on a hot path, that speed does not. Jev does not emit a sentence token by token. The probabilities that should come back arrive together. The speed comes from there. TypeSafe also uses Jev to check. Prompts, reasoning traces, and outputs from other models, Jev scores them, and it also acts as a guardrail and looks for jailbreaks. Generation stays with the chat model. Whether this gate is passed is decided by a model that does not generate strings. That is a joint in an agent, not chat. If the nod and the review still sit on long text, you have to find another program to read it. Jev closes the result into probabilities and levels, and the review can be written as code. The sentence written for the user is still the chat model's job. Jev does not take that sentence. It takes the routing in front, and the check behind. What an agent is missing here is not one more, longer explanation. It is a judgment whose type is already fixed, so code can walk with it. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://evals.typesafe.ai/ (https://evals.typesafe.ai/) https://docs.typesafe.ai/primitives (https://docs.typesafe.ai/primitives) Series Previous: JEV - A joint does not need a brain (/articles/jev-system-one-joints) Next: JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) ### JEV - How far "can't hallucinate" actually goes Canonical URL: https://modelflare.dev/articles/jev-what-cannot-hallucinate-means Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-what-cannot-hallucinate-means) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) TypeSafe says Jev "can't hallucinate". The sentence sounds like hallucination has been banned in one stroke. The same piece narrows it. What they wrote is that schema matching is a guarantee, not a statistic, so type errors can be written as 0%. One counterexample would be enough to refute that. They say it is mathematically impossible. The only layer that sentence can stand on is type. Jev gave up string generation. It hands back a structure fixed in advance, not a passage of text generated on the spot. Whether the probability is accurate, and whether the judgment is right, are two other matters. If a reader does not split them, "it did not write prose" gets heard as "the answer is correct". It cannot leave the table There are only three question types, and all of them are sealed. Noul asks whether one thing holds, and what comes back is a probability from 0 to 1. Choice options have to be written by the caller in advance, at most 255 on one table. What comes back is the chosen option, with a probability on every option and a confidence. Score levels also have to be written in advance, at least two and at most ten. The score can land between two levels. The answer carries the probability of each level and a confidence at the same time. That score is still a position on the scale, not a fresh paragraph. It cannot leave the option table. The model also cannot suddenly write a piece of prose and talk the answer into shape. The docs say this straight. What the model hands back is a distribution over the options or levels the caller submitted. It does not hand back a value outside that distribution. Code can use these values directly, without first digging fields out of a generated paragraph. Chat models usually do not hand work in this way. They hand in strings. A string might be a normal reply, and it might be a hallucination. Software still has to parse it and validate it. If validation fails, the program cannot use it directly. TypeSafe puts type safety and hallucination together, and treats type safety as the minimum condition for automation. If the type is wrong deep in a call chain, later branches connect to the wrong place. The official writing also says that current models, however smart, still hallucinate, and still make type errors. Staying inside the type does not stop the judgment from being wrong. Choosing the wrong item on a closed question is still wrong. On the public security-alert flow, the model has to choose among close it, hand it to an analyst, and contain it now. All three options are on the table. If the model chooses the wrong one, the type is still legal. The official account does not call System One always correct. They wrote that System 1 thinking gives the impression of being error-prone. They believe System One models can be made more reliable than the alternatives. They leave the specific reasons for later. That is their direction. It is not "always correct". Choice has another hard edge. Once the options pass 255, one table cannot hold them. A Wikipedia race that walks links has to pick among hundreds or thousands of links at a step. They switch to two stages: score independently, then make a choice. It occasionally slows down. That slowdown showed up in the Wikipedia-race demo. High cardinality is not infinite. "No hallucination" here still only means it will not hand back a link that is not on the table. Options cannot be added without a limit. Confidence is a separate question Calibration is a separate matter. The official account is that answers carry probability and confidence. When confidence is higher, accuracy is higher. Similar inputs return similar answers. On the other side, they write about chat models. Even if you prompt one to report a confidence, it is often overconfident, and unstable. If a task can be done right 95% of the time and the model does not say which 5% it is unsure about, that task cannot be automated. The confidence on Choice and Score is not a separate sentence of certainty the model reports. The number is computed from the shape of the distribution. Probability piled on one option or one level, and the number is high. Probability spread across several places, and the number is low. The caller acts when the number is high, and hands the case to a person when it is low. Noul does not attach a separate confidence. The yes-or-no probability is itself the signal. The official comparison table writes probability and confidence together. Applied to the three questions, Choice and Score have both, and Noul has only the probability. A confidence of 1.0 only means the distribution is pressed entirely onto one result. It describes the shape of this answer. When the distribution is pressed entirely onto the wrong option, confidence can still be high. A large number does not mean the result is right. A type error cannot come out. That is mathematically impossible. Calibration has no guarantee of that kind. What the official account gives is a correspondence: higher confidence, higher accuracy. If that correspondence breaks, the type still cannot escape, and confidence can no longer be a threshold. So this layer has a mechanism, and it has a claim. Whether errors are fewer when the number is higher, and whether similar inputs come back with similar distributions, both have to be looked at separately. The field being present every time only means the caller can read the number. Right is not the world Right and wrong have to be split once more. The workflow eval measures how close a model comes to a reference inside the same flow. That reference is not an objective answer from the world. They assume the flow's code is right, and they do not argue about whether the labels are right. The reference labels are the average of GPT-6 Astra and Claude Fable 5.1. Both use high thinking, and each answers every question in the flow. The other models are compared at each vendor's default reasoning. Jev coming close to that average means coming close to the consensus of those two models. It cannot be written as truer than the facts. If the consensus leans, an answer stuck to the consensus leans too. The official writing says that using the average of these two as the reference biases answers toward OpenAI's and Anthropic's models. During the comparison, the large models are wrapped in the System One LLM adapter, which forces them to emit structured decisions too, so they can be compared inside the same question types. The official account admits that asking with probabilities is usually slower and more expensive than giving a decision without probabilities. Their conclusion is that this is the most accurate way to take a decision from a large model. That accuracy is compared with a decision that does not carry probabilities. It does not mean the decision matched an external fact. The type-error numbers for large models on the chart come from OpenRouter, and they are biased. More complex queries may be routed to a stronger model. Jev's 0% is not the same kind of statistic. The official writing says their number is not an empirical result. Schema matching is guaranteed, so the chart can be filled with 0%. One side is an observation after routing. The other side is a number filled in from a guarantee. These are not the same error rate. The side-by-side with GPT-5.6 Terra used default reasoning. The official account picked that model because, as they read it, its intelligence is on average the closest to Jev. In the recorded run, the only disagreement is "Churn likelihood level". The official view is that the question is ambiguous and the answer cannot be said cleanly. On an ambiguous question, a distribution is more honest than a hard label. "Can't hallucinate", in the end, means it does not produce output outside the type. That is why they write type errors as 0%. Calibration has to be accepted on its own, by whether confidence can be used as a threshold. Whether the judgment is right, this flow eval cannot give the world's own answer. What it lines up with is the average of Astra and Fable. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://evals.typesafe.ai/ (https://evals.typesafe.ai/) https://github.com/typesafe-ai/system-one-adapter-python (https://github.com/typesafe-ai/system-one-adapter-python) https://docs.typesafe.ai/primitives/choice (https://docs.typesafe.ai/primitives/choice) https://docs.typesafe.ai/primitives/score (https://docs.typesafe.ai/primitives/score) https://docs.typesafe.ai/confidence (https://docs.typesafe.ai/confidence) Series Previous: JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) Next: JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) ### JEV - Are yes-no, choice, and score enough of a language? Canonical URL: https://modelflare.dev/articles/jev-three-question-types Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-three-question-types) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) Whether this language is enough depends on what you want it to say. Three closed questions buy speed, and they buy composability. The cost is that anything which needs to be unfolded, it cannot say. This is a language for code. It is not for a person to read. The docs and the eval site divide it the same way: Noul, Choice, and Score. There is no fourth kind on the page. The three kinds in the docs Noul only asks yes or no. It returns a probability. In the docs that number runs from 0 to 1. Near 1, it is a confident yes. Near 0, it is a confident no. If it stops near 0.5, that only means the two sides are equally possible. It does not mean "a middling degree". The docs name this trap: ask whether someone is strong at Python, and 0.5 is not a middling level. If you actually want to measure level, switch to Score and write what each level looks like. Noul also does not attach a separate confidence. The probability is the signal. Choice picks one item from a set that has been fixed. What comes back is the chosen item, plus the whole distribution and a confidence. Confidence looks at whether the distribution is sharp. If the probability is spread out, confidence is low, and code knows this question is unsteady. The cardinality cap on this question for Jev is 255. Above that, Jev takes two stages. Two stages means score independently first, then make an explicit choice. It is sometimes a little slower. The slowness is the extra step, not the same narrow question suddenly becoming expensive. Score is levels on a scale. In use it is an ordered scale of 2 to 10 levels. The docs say at least two levels, and the interface accepts at most ten. A level has to be written as a situation that can be matched. Write only "a bit high" or "all right", and the model has nothing to match. The score can land between two levels. That number is the sum of each level's position multiplied by its probability. A decimal between two levels is normal. It is not a broken calculation. It returns the score, and at the same time the distribution over the levels and a confidence. The same score might be everything pressed onto the middle level, or a split between the two ends. Look only at the score, and you will treat those as the same thing. Read the distribution and the confidence together. None of the three answers can leave the cells you gave in advance. Questions on the same material are independent of each other. Add one or take one away, and the other questions do not change. If a judgment depends on several things together, ask them separately and leave the weights in code. Later, when you adjust policy, you change these numbers. You do not rewrite a whole prompt. An expense report, taken apart The official eval page uses an expense report as a toy. On the left is a paragraph a person wrote, the kind of policy a team would write down. Every report has to carry a receipt. If the receipt cannot be read, ask the employee for another one. Then see whether the spend is a meal, travel, or equipment. If a meal is over $75 and the description does not match the receipt, a manager has to sign. The other reports are treated as approved. On the right, the same paragraph is split into a flow. Each sentence of policy is one question, or one code rule. Whether the receipt can be read is a yes-no. If it cannot, code asks for a new one, and that step does not ask the model again. Category uses a single choice, and the options are meal, travel, or equipment. Whether the amount is over $75, code compares on its own. Only the half sentence about the description not matching the receipt needs the model to be asked once more. Together with the amount, that decides whether a manager signs. Reports that do not need a manager's signature, code treats as approved. The model does not write a comment here. The direction on the eval page is one sentence: structure is always better than one large prompt. Averaged across the four example flows, every model on the workflow has higher accuracy than the whole policy as one prompt, and the spend and the time are lower. The prompt side hands the whole policy over and lets the model walk the logic inside one answer. The workflow side gives code whatever can be written as a rule, and leaves only the narrow judgments as questions. The steadiest real flow is usually not one large question. It is many narrow questions, independent of each other. Behavior depends on probability, not only on one discrete label. The category can look settled while the other options still hold a share of the probability. Code can take one more step at the threshold you wrote, without throwing the distribution away. Outside the flow, it is still one branch. The layer in the middle is domain engineering. Which question is unused this time, and which number counts as crossing the line, has to be written down for this business, and run the same way every time. Once the answers to the narrow questions are in code, the weights and thresholds are set in this layer. After string generation is given up Jev gives up string generation and takes parallel sampling in exchange. The questions in one request are computed together. Every output comes out at once. Not token by token. A chat model has to grow one token after another, each one looking at the one before. There is no letter-by-letter step here. Add a few ordinary narrow questions, and the time barely rises with them. The extra cost is mainly the question's own tokens. The official price is not the chat interface's price. Input is $0.042 per million tokens. Output is FREE. Without a long run of generated words billed by token, output is not charged. Latency lands at 70–500 milliseconds. That range is for a call chain, not for waiting inside a chat box. The return value goes straight to code. Noul's probability can enter an if. Routing uses Choice's label. A scoring threshold uses Score's position. No parsing first. If a chat model hands back a piece of Markdown, you have to split the words first, and one extra sentence can bend the format so the rest cannot connect. The three questions do not hand that text back, because Jev does not generate it. Explaining why this report was rejected does not fit in the three questions, and neither do an apology and a negotiation. A reason written for a person to read is not in the return value either. Those go to a model that can speak. Jev stops at the judgment. When it is not enough, the cases are concrete. If the options are names in an open world, company names and product names keep appearing, and the set cannot be fixed in advance. 255 is a cardinality cap, not an infinite list. When a reason must quote a sentence from the source, there is no such sentence to hand over. When a user is waiting for a human sentence, the interface wants readable words. Forcing Jev on then is treating a joint as a mouth. When policy can be closed into yes-no, single choice, and levels, what follows is a branch, and the judgment is assembled back into the same code. If what you want is speed, and this kind of composability, it is enough. What needs to be unfolded, it cannot say. Speed and composability are what that exchange buys. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://evals.typesafe.ai/ (https://evals.typesafe.ai/) https://docs.typesafe.ai/primitives (https://docs.typesafe.ai/primitives) Series Previous: JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) Next: JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) ### JEV - A lab could build this. Why it still might not. Canonical URL: https://modelflare.dev/articles/jev-why-incumbents-may-not-ship-it Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-why-incumbents-may-not-ship-it) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) On 15 September 2026, TypeSafe released Jev, and it was still in early access. This is their first System One model. It only makes judgments. It does not rely on small talk. The founder, Diogo Almeida, used to work on instruction following at OpenAI, on the research that became ChatGPT. Jev itself has no technical moat. Any model lab, with a small team, could make its own version in half a month. Technically, nothing blocks it. The official work did make something new. They built a new architecture. The sampler is parallel: one query finishes the judgment, without generating one token after another. They built a separate training method, called RLCD, reinforcement learning on calibrated decisions. The confidence the model reports has to line up with the share of later decisions that are right. Those are their implementation. What someone else would imitate is the product shape: do not generate strings, and emit calibrated probabilities only on closed questions. The question is fixed in advance, the model hands back a judgment with a probability, and code branches on that number. Half a month imitates that shape. It does not copy TypeSafe's research as it stands. The person imitating it does not have to take that research apart first. A moat is hard to claim. Once the shape is fixed, the implementation can be swapped out as a whole. After the swap, the system it plugs into still receives the same thing: state in front, a judgment behind. A chat model's product is a string. A string can be written very long, and software that actually needs that string has to parse it once more. Jev does not produce strings. Without that stretch of text that can be lengthened, output has no bill that rises with length. The incumbents' price list The price makes it plain. Jev's input is $0.042 per million tokens, close to free, and output is free. The official account says output is too cheap to meter. They also admit they cannot prove this price is not subsidized. Over the long run, they expect the price to go down, not up. The incumbents' price list is the other end. Input runs from $0.20 to $10 per million tokens, and output is about five times more expensive again. A chat model's money sits on long output. An answer for a person to read, and the steps an agent continues to execute, both have to be written as text. Once the text is long, the money lands on the output side. Even if input is pressed very low, one long write is still the bulk of the call. The two sides do not meet. Incumbents sell tokens. A model whose output is free, and whose input is close to free, eats the token business. The conflict is structural. The more widely a judgment model with free output is used, the fewer people buy the tokens that used to mean "let a large model classify, route, and score". An expense report is checked against a receipt, and someone has to judge whether the description matches the evidence. A security alert fires, and someone has to judge whether this machine is isolated now. Joints of that kind used to be handed whole to a chat model, which wrote a reason, after which the conclusion was dug out of the text. The dug-out text is priced as output, on the more expensive tier. A closed question hands the judgment over directly. What comes back is a set of probabilities, the length is fixed in advance, and there is no text that can keep being written downward. The reason can go unwritten, and that tier of charges has nowhere to land. What the incumbent damages is its own report. The more widely the model is used, the faster the old column falls. This is a business that enters from below. Cheap judgment travels with the flow and replaces the small decisions that used to call a large model. What gets replaced is not the call above that writes a long piece. It is the small decision that has to be made halfway through the flow. Calls that still have to be written out slowly can still be sold. The official expectation that their own price keeps going down does not fill the column the incumbent lost. The name Jev comes from William Stanley Jevons. The official analogy is coal and the steam engine. After the steam engine raised the efficiency of burning coal, coal was not burned less. It was burned more. Efficiency goes up, and use does not shrink. It grows. Cheap judgment calls use into existence. Asking a model once used to be not cheap, so people handed it only the few most valuable steps, and wrote the rest as dead rules, or looked themselves. Once judgment is cheap enough to be asked again and again inside code, questions that were not worth asking all arrive together. What goes up is the count. The use that gets called into existence is not the kind of token incumbents most want to sell today. The length written on a single call drops, and output is not charged. However high the count stacks, it does not enter the table they use today, the one that makes money on long output. The person who defines this kind of model The hard part is the person. Only someone who holds a business view and a model view at the same time sees joints all through a system. What a joint needs is not what a brain needs. A brain needs a stretch of words that can keep going. A joint does not want words. It wants a result that can be branched on. If the probability is sound, the program continues. If the probability is unsure, it is left to a person. Someone who only builds models easily sees this as an output format. Someone who only does the business easily feels that calling the large model once more is enough. The two views have to sit on one person before that person defines this kind of model. Such people are few. The item does not appear on a roadmap by itself. Using this kind of model well, a person meets the same difficulty. Instinct can only answer questions that instinct can answer. Whether something is so, and which of a fixed few items to pick, instinct can answer. Writing a stretch of content that was not there, or thinking the steps through on a question that has not been framed, instinct cannot hold. That is another kind of model's work. Instinct is fast, and it is cheap. When the program needs to ask, it asks once, and the time and the money can bear it. However many times it asks, the questions are still the ones instinct can hold. So it stays inside a program that keeps running. The large labs can build it. The people and the compute are there. Closing output from a string into a closed question is something engineering can get through, and that is not the same as reproducing TypeSafe's research in half a month. What half a month produces is a usable shape. TypeSafe's implementation is still in their own hands. That column is still taking money in. Once judgment is changed into a closed question, and output is made free, the column loses a stretch. The lost part is visible in a quarterly report. The place further forward is left to projects that are still writing output long. Context keeps growing, agents run a few more loops, and what comes out extra is exactly the token this price list recognizes. They can build it. It will not be scheduled at the front. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://evals.typesafe.ai/ (https://evals.typesafe.ai/) Series Previous: JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) Next: JEV - The two demos I ran (/articles/jev-two-demos) ### JEV - The two demos I ran Canonical URL: https://modelflare.dev/articles/jev-two-demos Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-two-demos) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) Today I ran two demos, trying Jev. Neither one asked it to chat. I put in a piece of business state, asked closed questions, and looked at what came back. State is a slice of the business, enough to judge from. I did not turn it into a story. A chat model given this kind of input often starts with a paragraph of understanding, then advice. I did not want advice. I wanted to see whether it could stay in the middle of a flow as a joint, emit numbers, and let code continue. The first demo I put that state in and asked the closed questions in one pass. I clicked. It came back at once. The official range is 70 to 500 milliseconds. I did not time it, so I cannot say where I landed inside the range. What I felt was that I never got as far as impatience. With models that write long replies, I am used to leaving the window and doing something else. This time I never got the window left. The result was already on the page before I switched away. I counted the money separately. Output is not charged. Input is $0.042 per million tokens. When the state is short, the input cost of one call is something I have to go looking for on the account before I can see it. As a joint, it is cheap. A node like this is walked many times. A large model that also bills output, I start saving by the second sentence. Here, asking one more question does not carry that burden. The questions were those three. For yes-no I used Noul. I asked whether this order should be lifted out of the automatic flow and given to a person first. It does not reply with a bare "yes" or "no". It replies with the probability of "yes", between 0 and 1. I look at which end it sticks to. If it sticks to one end, I treat the question as settled. If it wavers in the middle, the question is still open. For classification I used Choice, a single choice. The options and the meaning of each one were written by me. I did not pile them toward 255. The official cap is 255, and this question did not need it. Too many items, and I cannot see them myself. With few items I can judge. Weight clearly piled on one option, and the others very thin: only that kind of first place do I let into a branch. If the first two bite close together, I will not accept it just because there is a first place. A first place that is only a little higher has not separated yet. For risk I used Score, to see which level it landed on. Score's scale is 2 to 10 levels. I cut this question inside that range, from light to heavy. I did not invent another scale. It returns the probability of every level, and a position weighted by those probabilities. I do not dare to use that position alone. If the probability is piled on one level, the position is that level. If the probability is split across two neighboring levels, the position falls in the middle. The position looks finer, and both levels are still there. I will not write that kind of position into a later condition. It does not write an explanation. Look at the probability, not at an explanation. When I used a chat model for the same kind of judgment, the conclusion was wrapped in a paragraph. I had to peel the words off before I dared to fill a field, and I was afraid of taking politeness for a position. Today there is no such layer of words. Inside it is type and probability. What this demo confirmed for me is that it fits as a joint. I do not have to finish reading a short essay before I connect the code. I took the tradeoff from the distribution, not from a feeling of the moment. A yes-no stuck to an edge, I let through automatically. Category weight piled on one item, I let the code connect. The risk question's distribution did not gather, and I did not freeze it in this demo. I left it for the next one. The second one splits the same thing The second demo used the same state. I did not go find a new one. I split the questions, to check the sentence that a person does not have to stand in the middle of every loop. High confidence, I was ready to give to code. Low confidence, I kept to look at. After trying it, I was willing to let go of the high ones. Yes-no stuck to an edge, and category piled on one side, I let the program walk on its own. The high ones go straight into a branch. The rule is written in code. Confidence is enough, and the program continues. Confidence is not enough, and the program stops in front of me. When I read the log, those high loops had not waited for my nod. I did not confirm them one by one. The low one was the risk question. I had not asked it tightly enough, and the state was also missing material that could separate two neighboring levels. The distribution was spread, and confidence would not rise. While I looked, I actually wanted to pick a level for it, so the whole flow could finish. That thought cannot be followed. Once it is spread, I do not want to close it for it. If I casually pick a level and write it in, later steps will treat that level as fact. That is me supplying a decision it did not make, and dressing it up as an answer already finished. So the question stopped. It stayed for me to look at, and it did not enter an automatic branch. I did not run it again to try my luck. The material was still the same, and the distribution would most likely still be spread. This question should not be forced. Later I can thicken the state, or write the levels as wordings that actually stand apart. If I force a level out of it now, what comes back is an average. If I put that number into a branch, what follows will treat it as a risk already decided. I did not write "I feel it is more serious" beside it. My feeling cannot override a spread distribution and then be sent to run automatically. If the distribution really leaned, the weight would pile to one side on its own. If it cannot pile over, I have no standing to pile it for it. I stopped the second demo there. The official side-by-side Only after the two demos did I open the side-by-side example in the official playground. One side is Jev. The other is GPT-5.6 Terra. Terra used default reasoning. I was not there to judge who is smarter. I only looked at which question the disagreement landed on. I checked them through. Only "Churn likelihood level" differed on the two sides. The other questions lined up. The question itself is vague. How high churn might be is not easy to close into a clean level. When it is vague, what it gives is a distribution. The other side, to finish an answer, has to emit a word. I did not record the disagreement as a win or a loss. Giving a distribution is more honest than forcing out a word. That word can sit inside a sentence a person says. It does not belong in a branch that will execute on its own. I looked back at my own low risk question. Mine and this side-by-side are the same kind. There is no second model beside it, and I still should not pinch a spread probability into one level. Not being able to chat, I first took as a defect. After using it, I do not see it that way. There is no opening remark in the return. I do not have to delete an opening remark. When I used to dig a judgment out of a chat interface, I had to skip the courtesy first, and also guard against it changing its mind later. Today there is no text to clean up. For me that is a relief. It is one less thing that is easy to get wrong. What I ran today is these two demos. The side-by-side in the playground, I only opened and looked at. It does not count as a third one I made. The first one settled, for me, that it can sit on a joint, and that high confidence can be given to code. In the second, I stopped the question whose distribution was spread, and I did not dress it up as already answered. It is fast, and at this price I also find repeated calls cheap. What comes back is not an article. It is a distribution. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://console.typesafe.ai/playground?share=shr_13a74b495fb786c4bd7964f11597301e7c9 (https://console.typesafe.ai/playground?share=shr_13a74b495fb786c4bd7964f11597301e7c9) Series Previous: JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) Next: JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) ### JEV - People step from the middle of the loop to the edge Canonical URL: https://modelflare.dev/articles/jev-human-at-the-edge Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-human-at-the-edge) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) For the past two years, almost every serious AI system I have seen has assumed a person inside the loop, what people call human in the loop. The output cannot be counted on. The system finishes a passage and does not itself know whether the passage can be used directly, so a person has to catch it. Catching it, kept up, turns into a person looking at every judgment. That is what I ran into. Calling a person only when something has gone wrong is rare. A person sitting in the loop the whole time is common. When there are few questions, that glance still looks like responsibility. When there are many, the person is only passing them through, and after the pass the work goes on anyway. Those two years I took this as standard equipment for a serious system. Output shaped like that, and a person has to sit in the middle. I do not read it as laziness. Later I separated two things. Catching it is a person standing at the side, waiting for the moment that is unsure. A person in every loop means every question has to pass through their hands. I used to treat these as one thing, and then I could not take the person out of most loops. If I do not even know which question is unsure, I can only look at all of them myself. What the model handed me was a passage. The passage can be written very completely, and it can look a lot like a conclusion, but it cannot enter control flow. I have to read first, read which way it wants to take the matter, and then set the branch myself. I cannot remove that step. Code wants a value, and the passage is not yet a value. The person stays in the middle of the loop because the branch has to wait until I have finished reading. What I reviewed was not the occasional exception. Release or escalate, I had to finish a reason first, one piece after another. After enough reading, I treat a complete piece of writing as thinking that is already finished. The person was still sitting in the middle of the loop, and the judgment had followed the passage. A judgment small enough to enter code Jev turns that around. The judgment is pressed small enough, and closed enough, to enter the control flow of code directly. The question is written shut before it is asked: yes-no, or single choice, or else a Score. The model only answers the question that has already been closed. It does not tell the whole matter as a passage. The answer enters a branch. A person is no longer in the middle. The smallness I mean is the question. The business itself has not gotten smaller. The business stays as large as it should be. What gets smaller is the sentence asked of the model. That sentence has to be able to enter a branch in code. What cannot enter a branch is still a note a person has to read, and the person is back in the middle of the loop. I hold to this when I write the question. If the options can be written, and the boundaries do not fight, the question can be asked. If I only write "use your judgment", that is not a closed question. Either I still judge it myself, or it becomes a longer passage. I keep those. I do not ask the model. More precisely, what it turns around is not the person. It is "every judgment needs a person to glance at it". Every question carries a confidence. When the confidence can be trusted, the system sends only the least sure slice to a person. The person steps from the middle of the loop to the edge. On the part that is sure, the person no longer appears. I keep four shallow pictures in mind: whether an alert is closed, whether an invoice is paid, whether the next customer-service sentence escalates, and whether an agent run needs a person to look. They are the same kind of thing. The business is split into closed small questions first, and code walks from the result. The person used to stand in the middle of these loops. After the split, most branches do not have to call a person. I only handle what bounces back This is how I use it. High-confidence yes-no and single choice go straight to a branch. I do not open the content. A Score that lands on a middle level, with a flat distribution, is what bounces to me. One that only lands on a middle level, with a distribution that is not flat, I also do not look at. Code continues. There is no third way. I do not let it write a reason first and then decide, from that, whether I look. Done that way, a person still passes every question. What bounces back is only a small slice, and I can take it. When I used to pass every question, I could finish looking and still not say what I had judged. When it bounces over, look at the distribution, not at a long piece. A long piece kidnaps a person. Once the reason is spread out, the conclusion has usually already been chosen, and the words write downward along that conclusion. I may have another judgment, and I still have to take the passage apart first. I cannot point to the sentence it got wrong. I am only less sure than it is, and I feel awkward pushing back, and by the end of the taking-apart I often nod along. A distribution does not argue with me. How heavy each level is, is laid out. The flat place I can see at a glance. If I change the decision, I change it, without first having to beat a passage written very full. I am still in this system, and I am absent from most of the loops. The time it bounces back is the time that belongs to me. What I have in hand is a distribution that did not separate. I pick a side myself. Code walks by what I picked. What has already separated finishes inside the code, and it is not my turn to look. I no longer write a passage to negotiate with the model, and I do not wait for it to send me an explanation. Confidence has to work as a threshold I dare to use it this way on one condition. Confidence has to actually work as a threshold. If the model can do the questions but cannot say which time it is unsure, a person still cannot leave the middle of the loop. The official account says what they train with RLCD is calibration: higher confidence is more accurate, and similar inputs give similar answers. I trust that direction. The first sentence makes a threshold meaningful. The second sentence I care about more. If similar questions are one way today and another way tomorrow, the threshold I set will not stay fixed. That is why I dare to move the person out of the middle of the loop. If confidence is decoration, the thing reverses. The person has not stepped to the edge. The person has been replaced by a machine that is better at performing certainty. It gives every question a very full number, and the number has nothing to do with whether it is accurate. If I use it as the condition for letting something through, I let go when I should not. In the way I use it now, it does not do that. High confidence, I give to code. What bounces back, the distribution is flat, with no polite sentence of uncertainty attached. There are also times a person has to step back in. What has not been closed well should not be automated. An option is missing a piece, or two boundaries are tangled, and confidence only fills a number in on a bad question. I go back and change the question first. I do not prop the gap up with the model. Consequences too large to buy with a probability threshold, I leave inside the loop. All four pictures have a loop of that kind, and however high the confidence, I do not give it to a threshold. That is not the model failing. That loop should not be automated. After the person steps to the edge, the questions I look at are far fewer. The ones I am sure of, I no longer look at. The least sure slice is still in my hands. Loops that should not be given to a threshold, a person stays inside. Sources Introducing System One Models & Jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) Workflow evals (https://evals.typesafe.ai/) Series Previous: JEV - The two demos I ran (/articles/jev-two-demos) Next: JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) ### JEV - Past that line, I can no longer tell who is smarter Canonical URL: https://modelflare.dev/articles/jev-after-the-capability-line Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-after-the-capability-line) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) The contest among foundation models is in its third year. Most of those still at the table have crossed a line. The line is not low. My definition of it is one sentence. Ninety percent of users can no longer pose a task outside a model's reach. The work is still there. People do not stop asking. Those asks, the models can take. Replying to mail, collecting minutes, fixing one error, turning a pile of material into a few items: I stopped using these to rank them a long time ago. They sit inside the line. I place the landing between Opus 4.5 and 4.6. Above that, it blurs into one patch. The two years before, I still chased releases and switched in to compare, seriously, who was stronger. This year I stopped comparing. The boards are still updating. The work in my hands has not gained another tier. After the switch, only a feeling is left Before the line, I could tell. Whether a long piece would come apart, I knew by switching models. Whether earlier conditions were still there once the steps piled up, I could feel within the week. After the line, I cannot feel the difference in capability. Switching models often leaves only an impression that it got a little better. Better at what, I cannot say. The same kind of work needed my rework last month, and this month it goes through. Meeting notes go in, and the things to do come out. Two vendors come out a line or two apart, and both look usable to me. Code is the same. Paste an error, and both can give a change. Sometimes I feel one of them is cleaner, with a smaller diff. I cannot point to the update where they pulled apart. If I have to pick, I pick the smaller change. There is no day in the middle I can point at and say, from here it could do this. The increments have kept coming. They are too fine for my days to remember. So I switched models less for this. Who is smarter at the next tier, I cannot tell. These I can still tell apart The first thing I can feel is the shape of what it emits. What emits text, I treat as someone to talk with. I hand it a messy record and have it collect the few things to do. I read it once, change a few words, and throw it back if I am not satisfied. That round trip is familiar. Voice is another path. I speak to it, and it speaks back. I do not use it much. I turn it on while I am on the way. The moment I need to change words, I go back to text I can copy. I recognize this shape. What emits actions is more different. It operates a computer and clicks a flow through on its own. I watch its hands. Click the wrong place, and however complete the words after that are, they do not help. There is another kind that only gives a probability. Jev is that kind. It does not emit text, and it does not operate a computer. I have met all of these. I can tell them apart because the way I take over is different. Temper and taste also separate people. Hand over the same material, and some write it very full, adding a sentence at the edge, afraid I will not understand. Others stop here and leave the judgment to a person. Who I keep often depends on whether that temper fits my hand. The one that stops in the right place, I keep in the window I open every day. A higher score beside it, and I still rarely switch windows for the score. Cost and latency are felt together. Which stretch of the cost curve it sits on, and which stretch of the latency curve, two uses are enough to know. Chatting with a person, I can stand waiting a little longer. A person has to pause anyway. Put it on a flow that runs again and again, and waiting becomes a blockage. Slow enough that I can go pour a glass of water, and it only belongs in a conversation. Expensive enough that I do not dare call it once more, and it cannot enter the steps that are small and dense. If it is cheap, and it comes back fast, then I dare to spread it out. If what comes back is a passage I have to finish reading, cheap does not help. I still cannot put it in a loop. Refusal is also a difference. What it refuses, and how it refuses, shows after a while of use. Some refuse directly, in very few words. I know the path is closed, I change the ask, and one changed sentence is enough to continue. Some do not say so. They walk around the thing and hand back work that has gone crooked. I will still think I did not say it clearly, and grind another round on a bad result. What I look at now is whether I dare to hand this work to it next time. A new model states its place first GPT-6 made the world recognize computer use again. Operating a computer was put back at the front of a release. What I care about is not that models could use a mouse before. Past the line, that is no longer new. I used to watch a release for another tier of height. This time I changed the question. A company releasing a new model has to think about its own niche first: who it is for, and what it is particularly good at. It does not have to stand up and say that it is the next, smarter tier. Jev is an extreme I have actually touched recently. It does not compete with Opus at writing. Writing, it does not take. What it occupies is the joint. The material reaches the point where a decision has to be made, the question is closed, and the options were given in advance. What comes back is a probability. It comes back at once. On the chat side I still switch to another window and wait. Here I do not have to switch away. The result is already back. This kind of output is not charged. What I throw over is not "write me a paragraph". It is a judgment already circled: does this path go, and how sure is it. After the probability comes back, what takes over is a program, not me sitting there to finish reading and then decide. I have no desire to chat with it. The place is not reserved for chat. Chat models can go further up, and what they emit is still text. Jev removes the step of generating text. This is not smarter chat. It is another kind of product. Between model companies, I now look at the route, and at whether the product definition can be carried through. How a model is designed now comes down to those two. It is not the first name on a board that decides. Names on the board change after a while. What a model intends to emit, once the shape is fixed, the training and the interface follow, and the pricing follows too. Other routes, this piece only carries one sentence. Some push the engineering to an extreme. Some turn and build a vertical suite. The rest simply do not speak, and keep pushing a general model. The particular suites from DeepSeek and Kimi are not unfolded here. They are written in JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes). Past that line, I can no longer tell who is smarter. When I switch, it can still feel a little better. That feeling is not enough for me to choose with. What I can tell apart is the product definition. Do you emit text, emit actions, or emit only a probability. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://openai.com/index/gpt-6-astra/ (https://openai.com/index/gpt-6-astra/) Series Previous: JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) Next: JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) ### JEV - Some questions do not belong to it Canonical URL: https://modelflare.dev/articles/jev-questions-not-to-ask Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-questions-not-to-ask) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) After using it, what I am clearest about is not what Jev can do. It is which kinds of talk I should not ask it. I take it as a joint, not as a mouth. Instinct can only answer questions that instinct can answer. What has to be unfolded, or what has to lay the reasoning out for a person to see, is the wrong object if you ask Jev. Speaking something through to a person is also the wrong ask. It has no strings. Ask, and there is nowhere to put the answer. What I should not ask If you are also coming over from a chat model, the easiest things to ask wrong are the kinds below. I should not ask it to explain why. The habit I brought from chat is to open by asking it to explain. That habit does not work on it. When I ask it to explain, what arrives is only a judgment. A reason has to be written as sentences. It cannot write them. The return has no "because". If someone needs to see how the cause arrived, it cannot give that. If I really need a stretch of reasons, I write it myself, or I ask a model that can speak, and have it write from the judgment already made. I have asked it, wrongly, to write a reply for a person to read. What has to be sent is a passage. Which sentence comes first, and how heavy the tone is, lives in the words. What it gives is options and probabilities. Those do not match the passage that has to be sent. However high the probability, what comes back is one of the items I gave in advance, not a passage it writes now. I later reversed the order. It only judges which kind this thing belongs to, and whether this step should escalate. The sentences go to a chat model, or I write them myself. I have asked it to think up a name on its own inside an open set. Usable names are many, and only a small slice is written into the question. It can only pick among the items I gave. What is not among the items, it cannot produce. If I cannot name the candidates myself, it is not its turn yet. I narrow with rules first, or I pull a small handful out with retrieval, and then I close the question. Later I changed naming to this: I fix a few candidates myself, then let it pick. A brand-new one outside the list, I leave to the step that can speak. There is another kind that looks like heavy work which ought to be given to a model. The evidence has to be turned back and forth, and the reasoning has to be kept. Materials push against each other. You want it to pick a side, and you also want it to keep how the match was made. It does not have that back and forth. One request is a fast judgment from what I put in the state. The process does not become text that can be turned through. The back and forth I mean is often a conclusion that depends on the next page, and that page is not in the state yet. I put that dependence in code. Ask one closed judgment first, then I go get the evidence, and when it comes back I change the state and ask again. The sequence of which page was looked at first, and what was ruled out, it cannot write. If a trace has to be kept, I give it to a model that can speak. What I should ask What I should ask, I now only recognize a few shapes. Whether it is this kind, I ask directly. The categories have to be the ones I fixed in advance, and it only has to point at one. Severity, I do not let it report a fine number. I draw the levels, and I let it say which level it lands on. Escalation, whether this step should be handed to a person, is also a closed question, and I ask it in the same pass. A shape outside these, I do not stuff into it. I also keep a coarse ruler. If a person who understands the matter can point, by instinct, while looking at this state, then it looks like its question. If pointing still takes a long deduction, I hand it straight to a model that can speak. I prepare the state. I close the question. I choose the fields that enter the context, and I do not bring what is irrelevant to this question. The boundaries of options and levels are written in the question, not in a place where I hope it will improvise. It is not responsible for rounding an open question, and it does not turn the next page of material for me. Turn a page, and the state has changed. That is the next request, and I still have to prepare it. I have stepped on a question that was not closed well. The distribution flattens. Several options crowd together, and none stands a head above. I first took that as it being slow. It was not. I had given a joint the piston of a mouth. Options overlap, or a situation that will actually occur was not written into the list, and the probability can only spread. Then change the question, or hand the step back to a model that can speak. Changing the question means merging the overlapping items, and giving the uncovered case a "none of these". Handing it back means letting the model that can speak say the situation clearly first, and then I decide whether to close it again. Do not keep tuning it. While the question is still open, the distribution is still spread. Choice and Score I try not to fill Choice up to 255. The official cap is 255. Beyond that, they themselves have to score first and then choose, and it also gets slower. 255 is not a number I am here to fill. The more edge cases I list as options, the more the question looks like a list that was never closed. If my question naturally has hundreds of exits, I first ask whether rules can cut the candidates, and whether retrieval can cut them. A small handful has no fixed count. I can still tell, at a glance, how these items differ, and only then is the cut far enough. If a glance cannot tell them apart, I am sending the request too early. Cut it to a small handful, and then I take it to choose. I treat Score as ordered levels, not as a continuous exact decimal. When I write the question, I usually draw the levels from 2 to 10. What I want is a level, not a false precision like 7.63. I write the levels in plain words. Whether this level and the next one would be handled differently, I think through first. If they would, I split them. If both levels end by triggering the same code, I merge them into one level. No branch in the code is waiting for two digits after the decimal. It sometimes lands a number between two levels, and I still read it as a level, with the threshold written in code. If the meaning of a level cannot be written clearly, I change the description of the level. I do not pick at the decimal. While watching the demos I watched the official Doom demo and the Wikipedia race, and I did not use them to conclude my own questions. Watching Doom, I am easily carried off by the screen. The screen looks like a game being played. What goes in is structured state, not the picture. On the Doom side the scale is about 10 queries a second, about $7 an hour. That scale only shows that a realtime joint holds, and that a loop can turn at that density. It does not show that it can play the game. The picture was not eaten. The next step is code outside, asking with the state. The Wikipedia race shows that choosing links at high cardinality is useful. It does not show that it browses the web better than a model that can reason. Under the non-reasoning comparison, it takes fewer steps. That is because the other side did not turn reasoning on, so the demonstration looks better. Turn reasoning on, and the demonstration looks less good. The step count changes, and who is quicker changes. I will not bring "fewer steps" back to my own task and treat it as better at finding a path. Whether the path continues, and where it stops, is still decided by the program outside. I take this as: do not judge everything from a toy. Whether a demonstration is pleasant to watch does not decide, for me, which sentence I should ask next. When I use it myself, the split is fairly hard. What can be exhausted, and whose conditions are stable, I still write as rules. I do not go ask it. What instinct can answer in one point, that is when I ask it. Before I ask, the question has to be closed, and I have to be able to prepare the state. When it is time to open up and say the content, I give it to a chat model. Ask the wrong thing, and the slowness and the cost are my problem. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://evals.typesafe.ai/ (https://evals.typesafe.ai/) https://docs.typesafe.ai/primitives (https://docs.typesafe.ai/primitives) Series Previous: JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) Next: JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) ### JEV - A stripped-down model, a suite, and one that does not speak Canonical URL: https://modelflare.dev/articles/jev-three-product-shapes Co-author: folkbench.com (https://folkbench.com/?utm_source=modelflare&utm_campaign=jev-three-product-shapes) (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.) Ten pieces in this series: JEV - A joint does not need a brain (/articles/jev-system-one-joints) JEV - Why a Jev that cannot chat fits an agent (/articles/jev-agents-without-chat) JEV - How far "can't hallucinate" actually goes (/articles/jev-what-cannot-hallucinate-means) JEV - Are yes-no, choice, and score enough of a language? (/articles/jev-three-question-types) JEV - A lab could build this. Why it still might not. (/articles/jev-why-incumbents-may-not-ship-it) JEV - The two demos I ran (/articles/jev-two-demos) JEV - People step from the middle of the loop to the edge (/articles/jev-human-at-the-edge) JEV - Past that line, I can no longer tell who is smarter (/articles/jev-after-the-capability-line) JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) JEV - A stripped-down model, a suite, and one that does not speak (/articles/jev-three-product-shapes) The class group is passing boards around again. Someone throws DeepSeek's score. Someone throws Kimi's score. Underneath, someone drags Jev in and asks which of the three is stronger. I looked once and left. I did not reply. What the group wants is a name, and I do not have that kind of name on hand. These three are not ranks on the same board. Force a ranking, and the noise stays in the group. It has little to do with how I work this week. I use them by niche now, not by an overall board. The success I recognize is that narrow: a model leads in its own niche, and it does not have to be first at everything. Past the line from Opus 4.5 to 4.6, that habit is fixed. I think 90% of users can no longer pose a task beyond the model. Lift general intelligence another tier, and my days barely move. Compare general intelligence again, and I cannot feel it. Whether something speaks or not, that I choose every day. The interface also affects whether I can keep using it. Price is more direct. It has to be cheap enough that I dare to write it into a loop before I put the model into code. The ones that speak, I use apart DeepSeek, for me, is the path pushed to the simple extreme. Engineering taken as far as it will go, rate filled up, price cut in half. I open it, and there is not much on the page to wander through, and no industry workbench waiting. DeepSeek puts its effort into running fast and cutting the price. I dare to run the same thing several more rounds. Switch to a more expensive one, and I start calculating whether to save. Overnight batch work, I also throw to it. People are asleep, the calls are still running, and with an expensive model I feel uneasy. I use it for work that wants throughput. I push a batch of logs through to scan, and I squeeze interface drafts back and forth, many rounds of the same kind of question. I watch two things: do not stall, and do not get expensive. The cursor spins too long, or one round starts to hurt on money, and the pipeline breaks on its own. It can write sentences too. I usually change them in passing and use them. I rarely stop to taste them. I also do not talk taste with it on this work. Taste has to be ground word by word. It was not priced this way so a person could grind sentences. I put Kimi on the other side. Not to prove who is smarter. The front-end ability is filled up. A long piece of material goes in, a table of contents appears, the citations are still there, and the source can be spread open to read against. If I assembled that myself, an afternoon would be gone. It also built suites specifically for finance and law. The data sources are connected inside, the skills to use are sitting there, and there is a ready surface when the work is handed over. Regulations and financial reports, I turn through on that surface. I do not open a pile of other pages. When finance or law is spread out as long material, I do not want to build the environment myself first. What I want is to sit down and turn through it, not to spend half the time as my own front end. I open it when I need to read long documents, and when I need a workbench that is already there. What I accept after using them is one thing. This is a product. It is not a model getting smarter and therefore growing a website on the side. Which house the data connects from, and which clause of a regulation gets turned to, do not grow by themselves because a model is smarter. What a report looks like when it is handed over, someone has to make the flow into an interface first. When speech is fast and cheap, I stack calls, and what I use is DeepSeek. When I actually need to read material and hand work over, I open Kimi's front. Both can speak. I do not substitute one for the other. The one that does not speak, I set inside the flow I do not take Jev to chat. State goes in, the closed questions are asked, and what comes back is probability. Which item to pick, or which level it lands on, including a plain yes or no, the type has to be written shut in advance. If the question has not yet been closed, I will not call it. Output is free. Input is $0.042 per million tokens. The speed is the fast kind, 70 to 500 milliseconds. At this price I dare to ask again and again inside one flow, without mental arithmetic on the bill at every question. I treat it as one call inside the flow. I ask, and I go. Set inside my own flow as a joint. Code cleans the state up first, then throws a closed question over. The probability comes back, and the branch is walked by the program after it. For me it handles classification and routing, it handles levels, and it handles whether this step should call a person over. After classification I go on and ask which level it lands on, and after the level comes out I ask whether to call a person. All closed questions, all the same kind of call. Reasons and explanations written for someone else to read are not its job. Text that really has to be seen by a person, I connect a model that can speak. If this kind of judgment is given to a model that can chat, it often comes back as a whole paragraph, with an attitude and with advice. I then have to write another layer, and dig the paragraph into a branch. One more layer on the joint, and I do not think it is worth it. I have opened the eval only once. Inside the workflow, its judgments are compared with Astra and Fable 5.1. What is compared is closeness, and cost, not a win or loss at chat. That is another coordinate system. Once I understood that, I closed the page. A release should say who it is for The GPT-6 wave made computer use something people recognize again. The model goes and clicks the screen itself, opens software, and finishes a thing on the desktop. People were doing this before. After Astra came out, operating a computer became something people can point at again, no longer only an extra outside the chat box. Set next to these three, that is even more settled. The moment a model is released, the people releasing it have to say clearly who it is for. People ready to hand over the whole machine can use computer use. Another group sits in front of a chat box and wants the speaking finished. DeepSeek and Kimi both speak, but one is for stacking throughput and one is for a front that is already built, and those are already not the same use. Jev is not on the speaking side. The code is mine. It is only responsible for emitting a probability, so the program after it can walk. If I made a chat page for it, I would use it wrong. The moment there is a chat box, I want it to write the reason out. I will not write a piece called "who is strongest in 2026". That sentence does not help this year. The two inside the chat box, and this one that does not speak, I separated in use a long time ago. I am not preparing to write that kind of piece either. This is how I take them. Work that wants throughput, I give to DeepSeek. I do not spend myself on a tug over style, and I do not want to switch up to an expensive one. Long material, if it also comes with a workbench already built, I open Kimi. When code reaches the step that has to make a judgment — classification and routing, a level, and whether to call a person — I set Jev there, and I have it not open its mouth. The overall board can update if it likes. I take them by niche. The rank, I do not look at anymore. Sources https://typesafe.ai/blog/introducing-system-one-models-and-jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev) https://evals.typesafe.ai/ (https://evals.typesafe.ai/) https://openai.com/index/gpt-6-astra/ (https://openai.com/index/gpt-6-astra/) Series Previous: JEV - Some questions do not belong to it (/articles/jev-questions-not-to-ask) ### Validate LLM JSON Beyond Structured Outputs Canonical URL: https://modelflare.dev/articles/llm-json-output-validation Production guide: Validate LLM JSON Beyond Structured Outputs. It includes a deterministic artifact, failure boundaries, rollout checks, and source-qualified limitations. Decision first Validate LLM JSON Beyond Structured Outputs is an explicit production contract, not an isolated code change. Define the success signal, terminal failure state, and rollback condition before moving traffic. The artifact below separates evidence from assumption and keeps every attempt attributable. Start with transport, prove parse under a deterministic case, and make rejection a release gate rather than a follow-up task. The reusable artifact Use this table as the review record. A row passes only when its evidence comes from the same request, test window, or configuration snapshot. Checkpoint | Evidence to capture | Pass condition | transport | HTTP_and_endpoint_terminal_state_are_complete | The value is exact, preserved, and compared at the wire boundary. | parse | UTF-8_JSON_parses_once_with_no_trailing_content | The limit is explicit and the request fails closed when it is exceeded. | schema | Draft_2020-12_subset_and_additionalProperties_policy | The value is exact, preserved, and compared at the wire boundary. | business | cross-field_invariants_and_allowed_identifiers | Identity, scope, and policy are checked immediately before execution. | side_effect | validation_completes_before_any_mutation | Replaying the same operation cannot duplicate a side effect or charge. | rejection | safe_error_class_retained_without_echoing_sensitive_output | The owner, source, checked date, and known limitation are recorded. | Worked example The example is synthetic and deterministic. Replace identifiers and thresholds with reviewed values from your own workload; never paste production secrets or customer prompts into a test fixture. pipeline = parse_one_json -> validate_schema -> validate_business -> authorize cases: valid_object: accept '{"category":': reject_parse_incomplete '{"category":"ops","extra":1}': reject_schema_extra_field '{"category":"ops","requires_human":true,"priority":"low"}': reject_business_rule '{"category":"restricted"}': reject_authorization side_effect_gate = accepted_and_authorized_only Implementation procedure Freeze the current request, response, configuration, and observable baseline before changing behavior. Run one deterministic positive case and retain the complete client-visible result. Run the paired negative or limit case so failure behavior is proved rather than inferred. Join every attempt to one logical request ID, then record timing, terminal state, and usage without sensitive content. Roll out through a bounded cohort with explicit stop conditions; do not widen traffic merely because the happy path passed. Re-read the durable state and public behavior after the change, and execute the prepared rollback if any invariant fails. Failure modes The following failures invalidate the result even when the outer HTTP request appears successful: Valid JSON Schema is mistaken for application validity; authorization and business rules are skipped. A parser attempts to decode incomplete streamed arguments before the terminal event has closed that call. A retried tool call repeats a payment, message, mutation, or other irreversible operation. Prompts, credentials, raw tool arguments, or provider payloads are copied into high-cardinality telemetry. Modelflare boundary Modelflare can centralize OpenAI-compatible routing, keys, groups, usage records, and failure handling, but a configured route is not proof that every upstream supports every optional field. Verify the selected model and channel with the native protocol path. Keep provider prices in the current pricing source, preserve explicit zero values, and treat the final durable settlement—not an estimate—as billing truth. Use the parent guide (/articles/structured-outputs-openai-compatible-api) for the broader decision boundary and the setup documents (/docs) for current client configuration. Publication checklist The main question is answered before background material. Every request field, status, metric, and formula has one named owner. Examples use synthetic identifiers and contain no credential or customer data. All supported locales preserve the same headings, tables, code, limits, and caveats. Volatile API contracts, model support, and prices are rechecked at T-1 day; an invalidated claim moves the publication date. Before the scheduled time, the public API, localized routes, and sitemap still exclude the article. Sources and verification date Sources were checked on 2026-08-07. They establish the external contract or operating principle; they do not prove support on an untested provider route. JSON Schema Draft 2020-12 (https://json-schema.org/draft/2020-12) OpenAI: Structured Outputs (https://developers.openai.com/api/docs/guides/structured-outputs) ### Grok 4.7: Specs, Benchmarks, Capabilities, and Modelflare Pricing Canonical URL: https://modelflare.dev/articles/grok-4-7-api-guide xAI released Grok 4.7 on 21 September 2026. The API model ID is grok-4.7. It is xAI's current flagship for coding, agentic tool use, and knowledge work. Modelflare listed that same ID in the public pricing catalog on the same day. This article keeps three layers apart. The specification is what xAI documents. The benchmark table is what xAI reported at launch. The Modelflare prices are what GET /api/pricing returned on 21 September 2026. A vendor score is not a Modelflare test, and a catalog row is not a promise that every key, route, or prompt will behave the same way. xAI has not published a parameter count, a layer count, or a training-token total for Grok 4.7. "Larger than Grok 4.6" is the company's qualitative description of the new base model. The comparison below uses the figures that are actually public: context, token prices, reasoning levels, modalities, and the launch scores. What was published The launch post describes Grok 4.7 as a new, larger base model than Grok 4.6, trained with a longer reinforcement-learning run. The task mix is harder and weighted toward problems that take many hours. xAI says the model is better at checking its own work and at managing long context, and that it was trained to understand the Grok Bot harness used for conversational work. Two sentences in that post should not be collapsed into one. The headline says the model is "twice as fast, at half the price of comparable models." The body says it is served at the same price and the same speed as Grok 4.6. The token table supports the second sentence against Grok 4.6: both list $2 per million input tokens and $6 per million output tokens. The headline is a comparison with other models in that table, whose list prices are higher. xAI does not publish a tokens-per-second number next to "twice as fast," so this article does not treat that phrase as a measured latency result. grok-4.7 is on the xAI API, in Cursor, and is the default model in Grok Build. A separate Fast serving path exists only in Cursor and Grok Build. It is not on the public xAI API, and it was not in the Modelflare catalog checked for this article. Specification comparison Prices below are xAI's published list prices in USD per million tokens. "Short" means a prompt under 200,000 tokens. "Long" means a prompt that has reached 200,000 tokens. On the price table, crossing that line reprices every token in the request, not only the tokens past the line. Item | grok-4.7 | grok-4.6 | grok-4.5 | Context window | 500,000 tokens | 500,000 tokens | 500,000 tokens | Short input / cached input / output | $2.00 / $0.50 / $6.00 | $2.00 / $0.50 / $6.00 | $2.00 / $0.30 / $6.00 | Long input / cached input / output | $4.00 / $1.00 / $12.00 | $4.00 / $1.00 / $12.00 | $4.00 / $0.60 / $12.00 | Reasoning effort | low, medium, high (default), xhigh | low, medium, high (default), xhigh | low, medium, high (default) | Grok 4.7 details checked on the model page the same day: Inputs are text and images. Output is text. The model does not generate images. The page states no fixed text-output limit. Client timeouts, account limits, and the remaining context window still apply. The knowledge cutoff is May 2026. Later events are invisible unless a search tool is enabled. Documented interfaces are the Responses API and Chat Completions. Documented tools include function calling, web search, X search, and code execution. Reasoning cannot be switched off. If you omit the effort, the default is high. On the Responses API, grok-4.7 returns reasoning.encrypted_content even when the request does not ask for it. Later turns should send those reasoning items back unchanged. Chat Completions does not gain that behavior. Grok 4.5 stops at high. Grok 4.6 and Grok 4.7 add xhigh. Cached-input list price also moved: Grok 4.5 is $0.30 on short prompts and $0.60 on long prompts; Grok 4.6 and Grok 4.7 are $0.50 and $1.00. Standard input and output list prices are unchanged across these three IDs. Capability dimensions Coding and long agent work The launch post aims Grok 4.7 at work that runs for a long time: multi-step coding, checking a result before calling it done, and holding a long context together. These coding scores all come from xAI's launch table: CursorBench 4.0, which stresses longer coding tasks: 46.3% at xHigh. Grok 4.6 High is 40.4%, GPT-5.6 Sol Max is 41.7%, and Fable 5.1 Max is 51.8%. DeepSWE v1.1: 71.0%, marked by xAI as a high-effort score rather than xHigh. Grok 4.6 is 65.2%, Sol is 72.7%, and Fable 5.1 is 70.0%. Terminal-Bench 4.0: 38.0%. Grok 4.6 is 20.3%, Sol is 37.3%, and Fable 5.1 is 57.9%. EEBench, electrical engineering: 64.0%. Grok 4.6 is 53.0%, Sol is 39.4%, and Fable 5.1 is 56.4%. Read the rows as a map of where the vendor says the model moved, not as the score your repository will get. On CursorBench, xAI calls the result frontier price-performance: the score is above Sol and Grok 4.6, below Fable 5.1, at a much lower list price than either Sol or Fable. DeepSWE is close to Fable and still below Sol, and it is not an xHigh number. Terminal work improved sharply versus Grok 4.6 and sits near Sol, while Fable 5.1 remains well ahead on that row. EEBench is the clearest lead in the table. Professional knowledge work xAI also reports office and professional tasks: AA Briefcase v1.1, multi-hour office work: 1,657. Grok 4.6 is 1,546, Sol is 1,487, and Fable 5.1 is 1,678. Harvey Legal Agent Benchmark: 19.6%. Grok 4.6 is 15.8%, Sol is 2.5%, and Fable 5.1 is 6.7%. Leading this table still means about one in five items. It is not evidence that the model can carry legal work unsupervised. HealthBench Professional, clinical reasoning: 56.7%. Grok 4.6 is 48.5%, Sol is 60.5%, and Fable 5.1 is 62.1%. On this row Grok 4.7 is ahead of its predecessor and behind both comparison models. The post says Grok 4.7 is better at documents and presentations, and that it improves on Grok 4.6 on GDPval while landing in the same range as other frontier models. The prose does not print the GDPval point values, so this article does not supply any. There is no single intelligence index in the Grok 4.7 launch text. Grok 4.6's earlier post published an Artificial Analysis Intelligence Index. This launch replaces that one number with the task table above. Use the row that matches the work. A model can lead electrical engineering and trail clinical reasoning in the same release. Reasoning effort reasoning_effort accepts low, medium, high, and xhigh. high is the default. xAI describes low as the lighter setting, medium as more thinking where latency is less sensitive, high as the setting for difficult multi-step problems, and xhigh as the deepest setting, with the highest latency, for problems where answer quality dominates response time. More effort usually means more reasoning tokens, and output tokens are the expensive side of the list price. xhigh is not automatically the right production default. Compare task success, latency, and total tokens on your own prompts before you leave it on. For a multi-turn Responses conversation, keep the encrypted reasoning items in the next request. Dropping them discards the context the model used to reach the previous answer. Chat Completions does not return that encrypted field. Tools, images, and memory Function calling is the tool path you implement. You declare the function, the model proposes a call, your application runs it, and you send the result back. Web search, X search, and code execution are hosted by xAI. The model can call them on its own, and each invocation is priced separately from tokens. Images can be sent as input. The model-page summary does not list a separate image price. Image inputs consume context and appear in the request's token usage. Image generation stays on the Grok Imagine models, which are a different API. Five hundred thousand tokens is the window, not a budget you should fill. xAI recommends a stable prompt_cache_key on the Responses API, and the x-grok-conv-id header on direct Chat Completions, so a conversation is more likely to land on the same server and hit cache. Without that affinity, a cache-cold server bills the full input price. Long loops are also expected to compact older turns. Compaction changes what the model can see. Check the summary against the original constraints, IDs, decisions, and open errors. The May 2026 cutoff is a hard limit on memorized knowledge. Anything newer requires a search tool, and search is not included in the token price. Safeguards xAI says Grok 4.7 uses a new safeguard stack and is the strongest model it has tested on refusals and jailbreak resistance. Two figures are printed in the launch post: LatchBio's biosafety benchmark: 62.4%. HackerBench v0.3, which xAI describes as its benchmark of risky and malicious cyber tasks: 3.3% of risky dual-use prompts are allowed through. xAI also says legitimate security work is rarely blocked. The post says selected cybersecurity partners have invite-only access to red-team capabilities for defense research. That access is not part of the public grok-4.7 API described here. These figures are vendor evaluations. They are not an independent audit, and they are not a reason to skip human review on medical, legal, security, or other high-stakes answers. Official prices Token rates The xAI price page (https://docs.x.ai/developers/pricing), checked on 21 September 2026, lists USD per million tokens: Prompt size | Input | Cached input | Output | Under 200,000 prompt tokens | $2.00 | $0.50 | $6.00 | At or above 200,000 prompt tokens | $4.00 | $1.00 | $12.00 | The long rate covers the whole request once the prompt reaches the threshold. The first 200,000 tokens do not keep the short rate. Three official examples Hosted tool fees are excluded. Request | Basis | Arithmetic | Total | 100,000 uncached input, 5,000 output | Short | 0.1 × $2 + 0.005 × $6 | $0.230 | 100,000 cached input, 5,000 output | Short | 0.1 × $0.50 + 0.005 × $6 | $0.080 | 220,000 uncached input, 10,000 output | Long | 0.22 × $4 + 0.01 × $12 | $1.000 | The jump from the first row to the third is not "a bit more text." Crossing 200,000 tokens doubles every rate, and the extra tokens are billed at the doubled rate. A fully cached 100,000-token prompt in the second row costs about one third of the uncached short prompt, because cached input is one quarter of the input rate and the output charge does not change. Hosted tools, Fast, and the US region Hosted tool invocations on the same price page: Web search (web_search): $5 per 1,000 calls. X search (x_search): $5 per 1,000 calls at the time of this check. Starting 21 September 2026 at 12:00 PT, xAI replaces that call price with $5 per 1,000 posts fetched and $10 per 1,000 user profiles fetched. Every returned post counts, including parents and quotes. Every returned profile counts. Code execution (code_execution and code_interpreter): $5 per 1,000 calls. File-attachment search: $10 per 1,000 calls. Collections search: $2.50 per 1,000 calls. The model chooses how many hosted calls to make. A search-heavy answer can cost more in tool fees than in tokens. The US regional endpoint https://us.api.x.ai/v1 keeps inference in the United States and multiplies input, cached input, and output by 1.1, including long-context rates. For grok-4.7 that is $2.20 / $0.55 / $6.60 under 200,000 prompt tokens, and $4.40 / $1.10 / $13.20 at or above that line. Grok 4.7 Fast is the same model on faster infrastructure. xAI's prose says it is billed at twice the standard token rates. The price table prints $4.00 / $1.00 / $12.00 below 200,000 prompt tokens, and $6.00 / $1.50 / $18.00 at or above 200,000. Doubling the standard long-context row of $4 / $1 / $12 would be $8 / $2 / $24. The printed Fast long-context row does not match that doubling. Budget from the table, and recheck the price page before you rely on Fast. Fast is sold through Cursor and Grok Build. It is not on the public xAI API, it is not included in Grok Build's free tier, and the Modelflare catalog checked on 21 September 2026 had no Fast model ID. The Grok IDs present were grok-4.5, grok-4.6, and grok-4.7. Launch benchmarks The scores below are copied from xAI's launch post of 21 September 2026. Folkbench compares routes on availability, latency, and price, and keeps its notes on this model at folkbench.com/models/grok-4-7 (https://folkbench.com/models/grok-4-7). Column headers keep the effort setting xAI printed. The other models' scores are the ones xAI placed in the same table. Modelflare did not re-run them. Evaluation | Grok 4.7 xHigh | Grok 4.6 High | GPT-5.6 Sol Max | Fable 5.1 Max | List input price, USD / 1M | 2 | 2 | 4 | 10 | List output price, USD / 1M | 6 | 6 | 20 | 50 | CursorBench 4.0 | 46.3% | 40.4% | 41.7% | 51.8% | DeepSWE v1.1 | 71.0% (high, not xHigh) | 65.2% | 72.7% | 70.0% | EEBench | 64.0% | 53.0% | 39.4% | 56.4% | AA Briefcase v1.1 | 1,657 | 1,546 | 1,487 | 1,678 | Terminal-Bench 4.0 | 38.0% | 20.3% | 37.3% | 57.9% | Harvey Legal Agent Benchmark | 19.6% | 15.8% | 2.5% | 6.7% | HealthBench Professional | 56.7% | 48.5% | 60.5% | 62.1% | The Grok 4.7 column is not one effort setting. DeepSWE is explicitly a high-effort score. The other Grok 4.7 cells sit under the xHigh header. Grok 4.6 is High. Sol and Fable are Max. A higher score can come from a deeper and more expensive reasoning setting. That is worth knowing, and it is not a matched comparison at a single effort level. On this table Grok 4.7 leads EEBench and the Harvey benchmark. It is close to the leaders on CursorBench, DeepSWE, AA Briefcase, and on Terminal-Bench against Sol. It trails Fable 5.1 by a wide margin on Terminal-Bench, and it trails both Sol and Fable 5.1 on HealthBench Professional. Input list price is half of Sol and one fifth of Fable. Output list price is 30% of Sol and 12% of Fable. "Half the price" fits the Sol input cell and does not fit every other cell. The table is the precise comparison. Modelflare prices and how the groups differ On 21 September 2026, Modelflare's public pricing response included grok-4.7. The catalog list rate matches xAI's short-context list price: input $2 per million tokens, a completion ratio of 3 so output is $6, and a cache ratio of 0.25 so cached input is $0.50. That row does not publish a second long-context tier. Do not multiply xAI's $4 / $1 / $12 long rate by a Modelflare group ratio and treat the product as the bill. The charge stored on the finished request is the figure that was applied. The group is selected on the API key. Both groups call the same model ID. Price table Group | Ratio | Input / 1M | Cached input / 1M | Output / 1M | Catalog description | grok-award | 0.03 | $0.06 | $0.015 | $0.18 | Price-priority routing for budget-sensitive, retryable development, testing, and flexible workloads | grok-stable | 0.11 | $0.22 | $0.055 | $0.66 | Everyday development, longer projects, and production applications for individual developers who want stability | The arithmetic is the catalog list rate times the group ratio. Award is $2.00 × 0.03, $0.50 × 0.03, and $6.00 × 0.03. Stable is $2.00 × 0.11, $0.50 × 0.11, and $6.00 × 0.11. Against the short-context list price, those ratios are 3% and 11%. They are not 3% or 11% of the long-context list price, of the US regional price, or of hosted tool fees. The Chinese catalog text for grok-award adds a sentence the English description does not: the very low price does not guarantee stability or model quality. Keep that next to the 0.03 ratio. grok-stable is the catalog's description for ordinary development and production. Neither sentence is a measured uptime agreement, a latency target, or a claim that the groups run different weights. Three Modelflare examples The token shapes match the official examples. They are priced on the catalog's single rate. Cached rows assume the usage record counts those input tokens as cache hits. Very small charges can also move when quota is rounded to the billing unit, so the request log remains the invoice. Request | grok-award | grok-stable | 100,000 uncached input, 5,000 output | $0.0069 | $0.0253 | 100,000 cached input, 5,000 output | $0.0024 | $0.0088 | 220,000 uncached input, 10,000 output, at the single catalog rate | $0.0150 | $0.0550 | The 220,000-token row is intentionally not 0.03 × $1.00 or 0.11 × $1.00. One dollar is xAI's long-context list charge. If a later catalog revision adds a long-context tier, this row stops being the quote. The live page is the grok-4.7 price page (/pricing/grok-4.7). The index is Models and prices (/pricing). Calling the model through Modelflare, rather than opening a separate xAI account only for this ID, means: The model ID stays grok-4.7. The client does not target a renamed alias. The canonical base URL is the OpenAI-compatible https://modelflare.dev/v1. An OpenAI SDK usually needs that base URL, a Modelflare key, and the model ID. The key is placed in grok-award or grok-stable. The same account can call other catalog models when the key's group allows them. The price split is explicit. Award is the cheap route for work you can retry, with the stability and quality caveat above. Stable is the route the catalog describes for everyday development and production. The listed endpoint types for this ID are Chat Completions, Responses, and Responses compaction. This price does not include a measured latency advantage over calling xAI directly, hosted search or code-execution fees, the Fast serving tier, or a service level beyond the group description. How to call Grok 4.7 on Modelflare Create an API key and select grok-award or grok-stable. The group lives on the key. A key in an unrelated group can fail on grok-4.7 even though the model is listed publicly. Send the model ID exactly. The canonical base URL is https://modelflare.dev/v1. The same API is also published at https://cf.modelflare.dev/v1 and https://origin.modelflare.dev/v1. The root hostname is the canonical site. Chat Completions: export MODELFLARE_API_KEY="YOUR_MODELFLARE_API_KEY" curl -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "grok-4.7", "reasoning_effort": "high", "messages": [ { "role": "user", "content": "Review this migration plan and list the three assumptions that most need a test." } ] }' Responses: curl -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "grok-4.7", "reasoning": {"effort": "high"}, "input": "Review this migration plan and list the three assumptions that most need a test." }' On Chat Completions, reasoning_effort accepts low, medium, high, and xhigh. On Responses, set reasoning.effort to the same values. Image input uses the ordinary OpenAI content array on a user message. Your own functions use the ordinary tools field. POST /v1/responses/compact is listed for this model and is forwarded as compaction on grok-4.7, not as another model ID. Hosted xAI tools, including web search, are a provider behavior. Try them on the route you will actually use, and read the usage record, before you depend on them or on xAI's invocation fees. For the difference between the two request shapes, see Responses API versus Chat Completions (/articles/responses-api-vs-chat-completions). For the cost method behind the tables, see AI API cost tracking (/articles/ai-api-cost-tracking). Checks before you switch traffic Pin grok-4.7. Do not accept a silent substitution of Grok 4.6 or Grok 4.5. Confirm the key's group is grok-award or grok-stable, and choose it from retry tolerance rather than from the ratio alone. Replay one fixed, non-sensitive set at low, medium, high, and xhigh. Record success, latency, input tokens, cached input, output tokens, and the charge. Run Grok 4.6 on that same set before you replace it. The list price is similar. The score table is not uniform. Price one prompt just under 200,000 tokens and one just over it, then read the recorded charge. The xAI sheet and the Modelflare catalog row do not currently publish the same long-context rule. If the workflow needs images, function calls, streaming, or cancellation, test those paths. One text-only success does not cover them. Keep a rollback route. A completed HTTP response only shows that one call finished. Sources Checked on 21 September 2026. Introducing Grok 4.7 (https://x.ai/news/grok-4-7) Grok 4.7 developer guide (https://docs.x.ai/developers/grok-4-7) xAI models and pricing (https://docs.x.ai/developers/models) xAI API pricing (https://docs.x.ai/developers/pricing) xAI release notes (https://docs.x.ai/developers/release-notes) Reasoning effort (https://docs.x.ai/developers/model-capabilities/text/reasoning) Modelflare public pricing catalog, GET https://modelflare.dev/api/pricing, model grok-4.7 Earlier guide: Grok 4.6 API, pricing, and 500K context (/articles/grok-4-6-api-guide) ### Connect Claude Code to an Anthropic API Gateway Canonical URL: https://modelflare.dev/articles/claude-code-anthropic-api-gateway Connect Claude Code to an Anthropic API Gateway: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits. Choose the wire contract before the tool Connect Claude Code to an Anthropic API Gateway should be implemented as a coding-agent integration contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN, Anthropic_Messages, model_aliases. Connect Claude Code through a gateway that preserves the native Anthropic Messages contract. Set the base URL and authentication source explicitly, map primary and background model aliases deliberately, then verify streaming, tool use, usage, and failure behavior on the actual route. Do not treat an OpenAI-shaped compatibility response as proof. Separate client setup from gateway policy Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once. A useful integration page owns protocol selection, configuration precedence, proof cases, and team rollout. The live setup document remains the owner of copy-paste installation steps. This division follows a capability-tree model: the article explains why and how to verify; the documentation supplies the current commands. The owner record for this page is claude-code-anthropic-api-gateway; its frozen controls are ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN, Anthropic_Messages, model_aliases. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label. Compatibility contract: Connect Claude Code to an Anthropic API Gateway Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response. Control point | Fixed decision | Evidence to retain | wire_contract | native_anthropic_messages | captured_messages_request + stream_events + stop_reason | base_url | reviewed_gateway_origin | ANTHROPIC_BASE_URL_effective_value_without_credentials | authentication | scoped_token_or_dynamic_helper | credential_source + owner + expiry + last_rotation | model_aliases | primary_and_background_mapped_explicitly | effective_alias_map + native_probe_per_alias | tool_use | tool_use_and_tool_result_correlated | tool_use_id + policy_decision + terminal_result | removal | original_environment_restorable | unset_or_restore_steps + direct_probe | Reference configuration or flow The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result. export ANTHROPIC_BASE_URL="https://modelflare.dev" export ANTHROPIC_AUTH_TOKEN="" export ANTHROPIC_MODEL="" export ANTHROPIC_SMALL_FAST_MODEL="" Prove the complete coding-agent path Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request. Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for wire_contract: enforce native_anthropic_messages and retain captured_messages_request + stream_events + stop_reason. Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for base_url: enforce reviewed_gateway_origin and retain ANTHROPIC_BASE_URL_effective_value_without_credentials. Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for authentication: enforce scoped_token_or_dynamic_helper and retain credential_source + owner + expiry + last_rotation. Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for model_aliases: enforce primary_and_background_mapped_explicitly and retain effective_alias_map + native_probe_per_alias. Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for tool_use: enforce tool_use_and_tool_result_correlated and retain tool_use_id + policy_decision + terminal_result. Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for removal: enforce original_environment_restorable and retain unset_or_restore_steps + direct_probe. Integration failures that a text demo hides Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions. protocol_emulation — Translating native Messages through an unverified OpenAI-compatible path can lose stop reasons, content blocks, or tool semantics. auth_precedence_surprise — Multiple credential sources can cause Claude Code to use a different token than the reviewer expects. background_model_gap — The main prompt can pass while background or lightweight work fails on an unmapped alias. unbounded_shared_token — A static token shared across developers prevents precise revocation, quota, and attribution. Rollout evidence and stop conditions Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins. Signal | Decision threshold | Action | native_messages_probe_rate | 100%_for_each_required_alias | block_unverified_alias | tool_correlation_errors | 0 | disable_tool_use_and_rollback | credential_owner_coverage | 100% | stop_new_onboarding | unexpected_direct_provider_calls | 0 | repair_environment_and_restart_client | Modelflare boundary and limitations Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth. Anthropic’s gateway documentation describes configuration principles and third-party examples; it does not endorse or certify Modelflare. Modelflare support must be proven through its current native Messages route and the exact model aliases selected for the workload. Continue through the topic cluster Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field. Parent decision guide (/articles/ai-coding-agent-api-gateway) Next implementation guide (/articles/api-key-security-cost-controls) Current setup documentation (/docs/claude-code) Frequently asked questions Why not use the OpenAI-compatible endpoint? Claude Code expects Anthropic Messages semantics. Use the native route unless a complete translation contract has been independently proved. Should the token be stored in a shell profile? Prefer a scoped secret source or helper appropriate to the environment; never commit or print the token. Why test the background alias? Claude Code may use different models for foreground and background work, so the primary probe is incomplete by itself. Sources and verification date Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state. Anthropic: Claude Code LLM gateway configuration (https://docs.anthropic.com/en/docs/claude-code/llm-gateway) Anthropic: Claude Code setup (https://docs.anthropic.com/en/docs/claude-code/getting-started) OWASP: Secrets Management Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html) ### Reconstruct Streaming Function Arguments Safely Canonical URL: https://modelflare.dev/articles/streaming-function-calls-sse Production guide: Reconstruct Streaming Function Arguments Safely. It includes a deterministic artifact, failure boundaries, rollout checks, and source-qualified limitations. Decision first Reconstruct Streaming Function Arguments Safely is an explicit production contract, not an isolated code change. Define the success signal, terminal failure state, and rollback condition before moving traffic. The artifact below separates evidence from assumption and keeps every attempt attributable. Start with identity, prove buffer under a deterministic case, and make replay a release gate rather than a follow-up task. The reusable artifact Use this table as the review record. A row passes only when its evidence comes from the same request, test window, or configuration snapshot. Checkpoint | Evidence to capture | Pass condition | identity | response_id,output_index,item_ID,and_call_id | The record joins to one logical request and one concrete attempt. | buffer | one_append-only_UTF-8_buffer_per_call | The limit is explicit and the request fails closed when it is exceeded. | ordering | event_sequence_validated;cross-call_order_not_assumed | The value is exact, preserved, and compared at the wire boundary. | completion | arguments_parsed_only_after_the_call_terminal_event | The value is exact, preserved, and compared at the wire boundary. | validation | JSON_parse,schema,authorization,business_rule | Identity, scope, and policy are checked immediately before execution. | replay | idempotency_key_derived_from_application_request_and_call_ID | Replaying the same operation cannot duplicate a side effect or charge. | Worked example The example is synthetic and deterministic. Replace identifiers and thresholds with reviewed values from your own workload; never paste production secrets or customer prompts into a test fixture. state = Map() on argument_delta(event): call = state.get_or_create(event.call_id) call.utf8_buffer.append(event.delta) on arguments_done(event): call = state.require(event.call_id) call.complete = true args = JSON.parse(call.utf8_buffer) validate_schema(args) authorize(current_user, call.name, args) execute_once(request_id + ":" + event.call_id, call.name, args) fragment_sequence: - [call_A, '{"order'] - [call_B, '{"city":"Pa'] - [call_A, '_id":"ORDER-1001"}'] - [call_B, 'ris"}'] terminal_parse: call_A: { order_id: ORDER-1001 } call_B: { city: Paris } Implementation procedure Freeze the current request, response, configuration, and observable baseline before changing behavior. Run one deterministic positive case and retain the complete client-visible result. Run the paired negative or limit case so failure behavior is proved rather than inferred. Join every attempt to one logical request ID, then record timing, terminal state, and usage without sensitive content. Roll out through a bounded cohort with explicit stop conditions; do not widen traffic merely because the happy path passed. Re-read the durable state and public behavior after the change, and execute the prepared rollback if any invariant fails. Failure modes The following failures invalidate the result even when the outer HTTP request appears successful: Parallel calls or streamed fragments are correlated by array position instead of a stable call identifier. A parser attempts to decode incomplete streamed arguments before the terminal event has closed that call. A retried tool call repeats a payment, message, mutation, or other irreversible operation. Prompts, credentials, raw tool arguments, or provider payloads are copied into high-cardinality telemetry. Modelflare boundary Modelflare can centralize OpenAI-compatible routing, keys, groups, usage records, and failure handling, but a configured route is not proof that every upstream supports every optional field. Verify the selected model and channel with the native protocol path. Keep provider prices in the current pricing source, preserve explicit zero values, and treat the final durable settlement—not an estimate—as billing truth. Use the parent guide (/articles/function-calling-responses-vs-chat-completions) for the broader decision boundary and the setup documents (/docs) for current client configuration. Publication checklist The main question is answered before background material. Every request field, status, metric, and formula has one named owner. Examples use synthetic identifiers and contain no credential or customer data. All supported locales preserve the same headings, tables, code, limits, and caveats. Volatile API contracts, model support, and prices are rechecked at T-1 day; an invalidated claim moves the publication date. Before the scheduled time, the public API, localized routes, and sitemap still exclude the article. Sources and verification date Sources were checked on 2026-08-07. They establish the external contract or operating principle; they do not prove support on an untested provider route. OpenAI: Function calling (https://developers.openai.com/api/docs/guides/function-calling) WHATWG Server-Sent Events (https://html.spec.whatwg.org/multipage/server-sent-events.html) ### Connect Codex to a Responses API Gateway Canonical URL: https://modelflare.dev/articles/codex-responses-api-gateway Connect Codex to a Responses API Gateway: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits. Choose the wire contract before the tool Connect Codex to a Responses API Gateway should be implemented as a coding-agent integration contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are model_provider, wire_api="responses", base_url, X-Client-Request-Id. For Codex, configure a user-level custom model provider that explicitly uses the Responses wire API, then verify more than visible text. The release gate must cover streaming, tool calls, typed output, request identifiers, usage, cancellation, and an unsupported-field case on the exact selected model route. Separate client setup from gateway policy Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once. A useful integration page owns protocol selection, configuration precedence, proof cases, and team rollout. The live setup document remains the owner of copy-paste installation steps. This division follows a capability-tree model: the article explains why and how to verify; the documentation supplies the current commands. The owner record for this page is codex-responses-api-gateway; its frozen controls are model_provider, wire_api="responses", base_url, X-Client-Request-Id. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label. Compatibility contract: Connect Codex to a Responses API Gateway Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response. Control point | Fixed decision | Evidence to retain | configuration_scope | user_level_provider_redirect_only | resolved_config_path + effective_provider | wire_api | responses | wire_api="responses" + POST_/v1/responses | credential_source | env_key_or_command_helper | variable_name_or_helper_path_without_secret | request_identity | client_and_provider_ids_joined | X-Client-Request-Id + x-request-id + logical_request_id | capability_probe | required_output_types_exercised | stream_events + function_call + usage + refusal_or_error | rollback | previous_provider_snapshot_restorable | backup_digest + restore_probe | Reference configuration or flow The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result. model = "" model_provider = "modelflare" [model_providers.modelflare] name = "Modelflare" base_url = "https://modelflare.dev/v1" env_key = "MODELFLARE_API_KEY" wire_api = "responses" Prove the complete coding-agent path Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request. Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for configuration_scope: enforce user_level_provider_redirect_only and retain resolved_config_path + effective_provider. Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for wire_api: enforce responses and retain wire_api="responses" + POST_/v1/responses. Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for credential_source: enforce env_key_or_command_helper and retain variable_name_or_helper_path_without_secret. Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for request_identity: enforce client_and_provider_ids_joined and retain X-Client-Request-Id + x-request-id + logical_request_id. Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for capability_probe: enforce required_output_types_exercised and retain stream_events + function_call + usage + refusal_or_error. Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for rollback: enforce previous_provider_snapshot_restorable and retain backup_digest + restore_probe. Integration failures that a text demo hides Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions. project_local_redirect_assumption — A project-local configuration may be intentionally unable to redirect model providers; verify the effective user-level config. chat_completions_substitution — A Chat Completions text success does not prove the Responses wire contract Codex expects. output_text_only_parser — Reading only convenience text can hide tool calls, refusals, or typed output items. credential_in_config — Embedding the secret in the provider file expands exposure and complicates rotation. Rollout evidence and stop conditions Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins. Signal | Decision threshold | Action | required_case_pass_rate | 100%_for_frozen_corpus | block_model_alias | unjoined_request_id_ratio | 0 | stop_and_fix_trace_join | stream_terminal_event_rate | 100%_of_successful_streams | rollback_provider_config | usage_reconciliation_delta | 0_for_deterministic_probe | hold_rollout_and_investigate | Modelflare boundary and limitations Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth. The configuration shown is a checked pattern, not a promise that every Codex release accepts the same keys forever. Re-read the current official Codex configuration reference and the live Modelflare setup document at T-1. Keep model IDs as reviewed deployment inputs rather than hard-coded editorial claims. Continue through the topic cluster Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field. Parent decision guide (/articles/responses-api-migration-checklist) Next implementation guide (/articles/openai-compatible-api-conformance-tests) Current setup documentation (/docs/codex) Frequently asked questions Why require wire_api="responses"? It selects the protocol contract Codex uses; a compatible Chat Completions endpoint is a different surface. Can project configuration redirect the provider? Do not assume so. Current Codex guidance restricts provider redirection to trusted user-level configuration. What is the minimum probe? Text alone is insufficient; include streaming, a tool call, typed output handling, request IDs, usage, cancellation, and one negative case. Sources and verification date Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state. OpenAI: custom model providers for Codex (https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers) OpenAI: streaming Responses (https://developers.openai.com/api/docs/guides/streaming-responses) OpenAI API reference: request IDs (https://developers.openai.com/api/reference/overview) ### DeepSeek V4.1 Flash Deep Dive: Architecture, Modelflare Pricing, and Rivals Canonical URL: https://modelflare.dev/articles/deepseek-v4-1-flash-review-pricing DeepSeek V4.1 Flash is a low-cost, open-weight, multimodal model built for long agent loops. Released on September 10, 2026, it combines a one-million-token context window, up to 384K output tokens, an unusually cheap official API, and an architecture that reduces prompt processing and KV-cache pressure. Its clearest advantage is not that it wins every benchmark. It is the amount of long-context agent work it can attempt per dollar while remaining competitive on coding, terminal, automation and tool-use evaluations. That conclusion needs boundaries. The benchmark numbers below were published by DeepSeek, not reproduced independently by Modelflare. DeepSeek's own report says the model still trails the largest closed systems on the hardest reasoning and edge cases. The correct first-party API model ID is deepseek-flash. DeepSeek V4.1 Flash is now live on Modelflare under the versioned model ID deepseek-v4.1-flash-0910, with both Chat Completions and Responses in the public deepseek-stable group. Last verified: 2026-09-10 UTC. Prices, aliases and catalog status can change. The direct answer: what DeepSeek V4.1 Flash is DeepSeek's release notice (https://api-docs.deepseek.com/news/news260910/) defines DeepSeek V4.1 Flash as the production replacement for the earlier Flash line. The official endpoint accepts deepseek-flash. The legacy names deepseek-v4-flash and deepseek-v4-flash-vision-exp are temporarily accepted by DeepSeek and routed to V4.1 Flash at Flash rates. Beginning September 14, 2026 at 04:00 UTC, DeepSeek says requests for deepseek-v4-pro will also route to V4.1 Flash until a later V4.1 Pro release. This migration policy belongs to DeepSeek's own API. Modelflare uses the explicit release-stamped ID deepseek-v4.1-flash-0910 rather than silently relabeling the earlier deepseek-v4-flash entry. The Modelflare operator confirmed the launch, and the model appeared in both public and origin pricing catalogs at 2026-09-10 15:21 UTC. Clients should use the exact ID shown by their provider instead of assuming that a first-party alias works unchanged through every gateway. The model's practical identity has five parts: multimodal image-and-text input; text output; a 552B-parameter MoE backbone; an additional 196B-parameter Engram memory; and asymmetric activation of 8B parameters per prompt token versus 16B per decoded token. The last point is central: input-heavy agent workloads should spend less compute during repeated prefill than a symmetric decoder-only design of similar scale. Specifications at a glance The following values come from the official release (https://api-docs.deepseek.com/news/news260910/), model card (https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) and technical report (https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf). “Backbone parameters” and “Engram parameters” describe different stores; neither number equals active compute per token. Item | DeepSeek V4.1 Flash | Official API ID | deepseek-flash | Modelflare model ID | deepseek-v4.1-flash-0910 | Model class | Multimodal Mixture-of-Experts Transformer | Input / output | Text and images in; text out | Transformer layout | 40 layers: 20-layer causal encoder + 20-layer decoder | Parameter stores | 552B backbone + 196B Engram conditional memory | Active parameters | 8B per prefill token; 16B per decode token | Context / maximum output | 1M / 384K tokens | Pretraining | 45T multimodal-corpus tokens; 7:1 text-only-to-multimodal token ratio | Main attention/cache design | CED + CSA2 + FP4 global KV; FP8 local SWA KV | Global KV footprint | 890 bytes per token, reported by DeepSeek | Public reasoning presets | low = 50, high = 75, max = 100 on the internal effort scale | Official API concurrency limit | 2,500 concurrent requests | License | MIT checkpoint license | The 384K output ceiling is three times the 128K maximum published for the current OpenAI and Anthropic comparison models in this article. It is a limit, not a recommendation: very long output increases latency, cost and the surface for drift. For long agents, the more useful question is whether the model can retain requirements and pass real acceptance checks after tool calls. Why the architecture is built for long agent loops A tool-using agent repeatedly adds observations, command results and edits to its history. Every new turn may require prompt prefill, attention over a growing context, KV storage and new output. DeepSeek V4.1 Flash attacks each cost separately rather than relying on a single smaller model. Agent cost | V4.1 Flash mechanism | Intended effect | Reprocessing long prompts | Causal Encoder-Decoder (CED) | Nearly halves asymptotic prefill work for long sequences in DeepSeek's analysis | Keeping global history in HBM | CSA2 cross-layer reuse + FP4 | Reduces duplicated global KV entries and bytes per entry | Persisting local attention state | SWA Bounded Replay | Reconstructs a bounded local window instead of storing every layer's full local cache | Recalling stable token patterns | Engram | Moves conditional n-gram memory into sparsely accessed tables | Producing tokens | DSpark speculative decoding | Drafts several candidate positions and chooses a verification length based on confidence and system load | Handling images | DeepSeek-ViT + projector | Converts visual inputs into embeddings processed with text from pretraining onward | These mechanisms mainly improve serving economics and throughput. They do not by themselves prove better reasoning. The model's quality also depends on data, post-training, the agent harness and how much inference effort is purchased. CED: cut prompt processing before touching the cache The technical report (https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf) splits 40 Transformer layers into a 20-layer causal encoder and a 20-layer decoder. The first two layers use only a 128-token sliding window. The remaining layers combine local sliding-window attention with compressed global context. For global attention, the decoder does not build an independent full KV history at every layer. Its global keys and values are projected from the final encoder hidden state. Local sliding-window states remain layer-specific. DeepSeek estimates long-sequence prefill complexity at approximately half the corresponding all-layer path: the dominant term changes from processing every token through all L layers to processing global context through roughly L/2, plus bounded local-window work. That explains the 8B/16B activation split. Prompt tokens activate about 8B backbone parameters during prefill; newly generated tokens activate about 16B during decode. Workloads with huge documents and short answers benefit more than workloads dominated by hundreds of thousands of generated tokens. CED reduces the first half of that bill; it does not make decoding free. CSA2, FP4 and SWA Bounded Replay: the memory story Compressed Sparse Attention 2 compresses cache along sequence, layer and precision dimensions. Each CSA2 layer is statically assigned one of three modes. Full computes main KV, indexer keys and fresh Top-K positions. Reindex reuses main KV and indexer keys but rescans them with its own query. Reuse reuses both the shared KV and a previous sparse selection. Every layer still has its own query and local sliding-window KV. The decoder's first Full layer selects Top-512 entries and builds a candidate pool; later Reindex layers choose their Top-512 positions from that reduced pool. Main global KV uses an approximately four-bit E2M1 representation with an E4M3 scale per 16 channels. DeepSeek retains FP8 for the more quantization-sensitive local SWA cache. Cache layer | Precision / lifetime | Reported result | What it does not guarantee | Runtime global KV | FP4, resident in HBM | 890 bytes/token; about 1/4 of V4 Flash and 437× smaller than V1 | Perfect sparse selection on every edge case | Runtime local SWA KV | FP8 | Preserves the 128-token layer-local window | Negligible memory at arbitrary concurrency | Persistent global KV | Host memory or SSD in DeepSeek's deployment design | Long-lived reusable global state | A fixed retention period on the public API | Persistent local SWA state | Reconstructed with Bounded Replay | About 1/8 of V4 Flash's total persistent cache footprint at equal sequence length | Exact replay of every discarded local state | The report says its deployment keeps global persistent KV for at least 72 hours and uses a distributed short-lived SWA pool. The public context-cache guide (https://api-docs.deepseek.com/guides/kv_cache/) describes cache creation as automatic and best effort, says construction can take seconds, and says entries are generally cleared after hours or days. Production clients should treat the public contract as authoritative: inspect hit and miss token fields instead of budgeting around a guaranteed 72-hour hit. DeepSeek identifies two robustness boundaries itself: CSA2 can select the wrong sparse positions, and Bounded Replay approximates discarded local state. Its tests found no systematic degradation in the evaluated settings, but extreme contexts and cache-resumption boundaries remain areas for further stress testing. Engram, DSpark and the multimodal path Engram separates some memorization from dense model computation. V4.1 Flash allocates 196B parameters across two conditional-memory modules at zero-indexed layers 1 and 14. Each uses token n-grams of lengths two, three and four, eight hash heads, context-aware gating and host-memory prefetch over RDMA. These are stored parameters, not another 196B parameters activated for every token. DSpark is a separately trained speculative decoder. Three Transformer blocks look through a 128-token sliding window and propose five positions in parallel. A lightweight dependency head relates draft tokens; a confidence head estimates how much of the draft will survive verification. The scheduler uses those probabilities and measured engine throughput to select a verification length under current load. This can improve system token throughput without changing the backbone's final verification role. The visual path uses a separately trained DeepSeek-ViT with 2D rotary position encoding. A 3×3 pixel-unshuffle step reduces visual-token count ninefold before projection into the language model. The vision encoder has 32 layers, hidden size 1,024 and 16 attention heads in the reported configuration. Its autoregressive training stage uses images scaled between 544×544 and 1,344×1,344, after an earlier contrastive stage over about 47B image-text pairs. Training and post-training DeepSeek reports 45T pretraining tokens with multimodal data integrated into language-model training. Text-only and multimodal pipelines are merged at a 7:1 token ratio after overlap replacement and deduplication. Sparse attention is trained from scratch at 64K sequence length, without a dense-attention warm-up; context extends to 1M at the 34T-token point. Stage | Published detail | Why it matters | Vision contrastive pretraining | About 47B image-text pairs, low-resolution phase | Learns broad visual representations before language integration | Vision autoregressive tuning | 236B tokens with captions, charts, OCR and related data | Adds fine-grained visual and document understanding | LLM pretraining | 45T tokens; multimodal data included from the start | Avoids treating vision as a late prompt adapter only | Context training | Sparse attention from 64K; extended to 1M at 34T tokens | Trains the deployed attention pattern rather than converting it only after training | Post-training | SFT, reinforcement learning and on-policy distillation | Aligns reasoning, tools and agent behavior | The report does not claim a new post-training algorithm. It attributes gains to synthetic task and environment creation, improved data filtering, verifiable rewards and the architecture/data scale. This matters when comparing “model architecture” claims: CED and CSA2 explain efficiency, while observed agent performance is the product of the whole training and scaffold stack. Reasoning effort: quality has a visible token bill DeepSeek exposes an internal effort continuum from 1 to 100 and maps the public API presets low, high and max to 50, 75 and 100. Compatibility inputs map minimal and low to low; medium, high and xhigh to high; and max or ultra to max. Thinking defaults to high. In DeepSeek's reported sweep, raising effort from 25 to 100 increased the average result across eight reasoning evaluations from 67.1% to 76.3%, DeepSWE from 66.0% to 74.2%, and Terminal-Bench 2.1 from 82.4% to 90.6%. Output tokens grew by about 2.5×. The 60–80 range captured most of the quality at less than half the maximum token budget; the final move to 100 lengthened agent trajectories by 1.6–1.8× for smaller gains. This is one of the model's most useful controls. Use low for classification, extraction and retryable exploration; high for normal coding and research; reserve max for tasks where another attempt or a missed edge case costs more than the extra tokens. Treat these as starting hypotheses and measure accepted-task cost. In thinking mode (https://api-docs.deepseek.com/guides/thinking_mode/), temperature and presence/frequency penalties are ignored, and top_p has a minimum of 0.95. With tools, clients must return the complete reasoning_content from the assistant message alongside tool results; omitting it causes a 400 response. That state rule is more important than copying a familiar OpenAI parameter set. Benchmark comparison with OpenAI and Anthropic This table reproduces selected results from Table 3 of DeepSeek's technical report. All models are shown at the report's Max setting. DeepSeek used different required or official scaffolds for some evaluations, a 1M DeepSeek Harness context for coding, and a 512K Claude Code context for visual-agent tasks. The numbers are provider-reported evidence, not an independent head-to-head test and not a production SLA. Evaluation | V4 Flash | V4 Pro | V4.1 Flash | GPT-5.6 Sol | Claude Opus 5 | GPQA Diamond, Pass@1 | 89.9 | 92.4 | 90.9 | 94.1 | 93.4 | Terminal-Bench 2.1, Pass@1 | 82.7 | 87.9 | 90.6 | 88.8 | 89.1 | Terminal-Bench 3.0, Pass@1 | 7.6 | 11.8 | 30.0 | 34.4 | 43.3 | Terminal-Bench 4.0, Pass@1 | 7.0 | 12.4 | 31.2 | 39.9 | 51.8 | DeepSWE v1.1, resolved | 54.4 | 62.7 | 74.2 | 73.0 | 74.0 | CyberGym, Pass@1 | 76.7 | 83.3 | 88.1 | 84.5 | — | HLE with tools, Pass@1 | 51.5 | 60.0 | 63.9 | — | 63.6 | AutomationBench, Pass@1 | 37.7 | 43.2 | 54.8 | 45.8 | 50.3 | Agents' Last Exam, Pass@1 | 25.2 | 25.7 | 31.8 | 26.7 | 28.6 | Chartography with tools, Pass@1 | — | — | 78.9 | 79.9 | 84.0 | The strongest reading is specific. V4.1 Flash makes a large step over V4 Flash on agent tasks and is competitive with GPT-5.6 Sol and Claude Opus 5 on DeepSWE, Terminal-Bench 2.1, automation and tool-using HLE. It does not lead GPQA, later Terminal-Bench versions or Chartography. The report itself warns that near parity on common tasks does not mean frontier parity on the hardest reasoning and edge cases. GPT-6 Astra and Claude Fable 5.1 are omitted from this score table because the DeepSeek report does not publish the same per-row comparison data for them. Adding scores from separate provider pages would create a false common harness. Teams should evaluate all candidates on the same repository tasks, tools, timeouts, network policy and acceptance checks. API compatibility is broad but not identical The official service exposes OpenAI Chat Completions (https://api-docs.deepseek.com/), OpenAI Responses (https://api-docs.deepseek.com/guides/responses_api/) and an Anthropic-compatible API (https://api-docs.deepseek.com/guides/anthropic_api/). That breadth lowers migration work, but compatibility describes request shape rather than identical lifecycle semantics. Surface | DeepSeek V4.1 Flash behavior | Migration consequence | Chat Completions | Streaming, JSON output, function calls; prefix/FIM in non-thinking mode | Existing OpenAI clients are the simplest starting point | Responses | Stateless; functions and images supported | Persist the full conversation yourself | Responses state fields | previous_response_id, conversations, store, background and context management are unsupported or ignored | A 200 response does not prove those features took effect | Hosted tools | Web search, file search, code interpreter, computer use and MCP built-ins are ignored; custom-tool support is limited | Execute tools in the application and test every requested field | Reasoning summaries | Summary and encrypted reasoning content are unsupported | Do not depend on OpenAI-style reasoning handoff fields | Anthropic format | Anthropic message shape is accepted | Preserve DeepSeek's reasoning and tool-state rules rather than assuming Claude semantics | Vision | Images are accepted in user input and tool outputs | Validate payload size, detail mode and image count | DeepSeek's Responses guide also lists ignored fields such as metadata, prompt templates, truncation, service tier, safety identifier, prompt-cache key/retention and stream options. Silent ignore is a compatibility hazard: schema acceptance can hide missing behavior. Build a conformance request for each feature your application depends on. The OpenAI-compatible API guide (/articles/openai-compatible-api-guide) explains why endpoint shape and capability need separate checks. OpenAI's current models provide a broader native Responses tool ecosystem and managed state features. Anthropic's native API has its own mature thinking-block and tool-use contract. DeepSeek's advantage is that one model accepts all three common dialects; its cost is that the intersection is smaller than the full native surface of either rival. Vision limits and image economics The official vision guide (https://api-docs.deepseek.com/guides/vision/) supports JPEG, PNG, GIF and WebP through base64, external URLs or the Files API. Images are automatically resized toward roughly 1,300×1,300 and use at most 1,024 input tokens each. low detail uses a 512×512 view; high and original retain more detail; auto currently behaves as original. Limit | Official value | Request body | 48 MiB | Inline or externally fetched image | 32 MiB each | Image referenced by file ID | 64 MiB each | Images per request | 600 | Combined image bytes | 64 MiB without file IDs; 200 MiB with file IDs | Long edge | 8,192 px; 4,096 px when sending 15 or more images | Visual-token ceiling | 1,024 tokens per image after processing | At the maximum 1,024 visual tokens, one uncached image contributes about $0.0001536 off-peak or $0.0003072 peak at the published input rates, before accompanying text and output. This arithmetic is useful for scale estimates, but image resizing, cache behavior and actual visual-token usage can change the charge. More images also consume context that could otherwise hold text or tool history. V4.1 Flash is suited to document screenshots, charts, OCR and visual agent observations. It is not an image-generation model. For generated images, use a model and endpoint designed to return pixels rather than text. Official API pricing compared The DeepSeek price page (https://api-docs.deepseek.com/quick_start/pricing/) introduced the following rates at 04:00 UTC on September 10. Weekday peak windows are 01:00–04:00 and 06:00–10:00 UTC; all other weekday times and the weekend use off-peak rates. DeepSeek's context cache is automatic, so “cached input” means a reported cache hit, not a cache flag that forces one. OpenAI rates come from its current model comparison (https://developers.openai.com/api/docs/models/compare); Anthropic rates come from its pricing page (https://platform.claude.com/docs/en/about-claude/pricing). Values are USD per one million tokens. Anthropic also charges cache writes; they are omitted here because the table compares new input, cache reads and output only. Model | Context / max output | New or miss input | Cached/read input | Output | DeepSeek V4.1 Flash, off-peak | 1M / 384K | $0.15 | $0.003 | $0.60 | DeepSeek V4.1 Flash, peak | 1M / 384K | $0.30 | $0.006 | $1.20 | DeepSeek V4.1 Flash on Modelflare | 1M / 384K model limit | $0.193548 | $0.003871 | $0.774194 | GPT-5.6 Luna | 1.05M / 128K | $0.20 | $0.02 | $1.20 | GPT-5.6 Terra | 1.05M / 128K | $2.00 | $0.20 | $12.00 | GPT-5.6 Sol | 1.05M / 128K | $4.00 | $0.40 | $20.00 | GPT-6 Astra | 1.05M / 128K | $10.00 | $1.00 | $50.00 | Claude Sonnet 5 | 1M / 128K | $2.00 | $0.20 | $10.00 | Claude Opus 5 | 1M / 128K | $5.00 | $0.50 | $25.00 | Claude Fable 5.1 | 1M / 128K | $10.00 | $0.25 | $50.00 | Price is not value by itself. Different tokenizers can bill different token counts for the same text; Anthropic notes that newer tokenization can produce roughly 30% more tokens on some workloads. Models can also require different output lengths, retries and human repairs. Compare cost per accepted task, not just the row with the smallest input number. Modelflare's current public catalog exposes one rate between DeepSeek's two time bands: it is about 64.5% of the first-party peak rate, or 35.5% lower during peak windows, while it is about 29.0% above DeepSeek's off-peak rate. The listed gateway tariff therefore has no time-band adjustment today; flexible workloads that can reliably use DeepSeek's off-peak window may still pay less through the first-party API. OpenAI applies higher rates when input context exceeds 272K: the whole request uses 2× input and cached-input prices and 1.5× output prices. This boundary matters in the long-context scenario below. DeepSeek uses time windows instead; Anthropic's listed one-million-token models do not use the same 272K boundary in the cited price table. Two reproducible cost scenarios Scenario A is one request with 100K uncached input and 10K output. The formula is 0.1 × input rate + 0.01 × output rate. It assumes no cached input, tools, retries, taxes or provider-specific add-ons. Model | Scenario A cost | DeepSeek V4.1 Flash, off-peak | $0.021 | DeepSeek V4.1 Flash on Modelflare | $0.0271 | GPT-5.6 Luna | $0.032 | DeepSeek V4.1 Flash, peak | $0.042 | Claude Sonnet 5 | $0.300 | GPT-5.6 Terra | $0.320 | GPT-5.6 Sol | $0.600 | Claude Opus 5 | $0.750 | GPT-6 Astra / Claude Fable 5.1 | $1.500 | Scenario B is the marginal cost of a warm-cache request with 900K cached input, 100K new input and 20K output. It deliberately excludes the earlier request that established the cache. Formula: 0.9 × cache-read rate + 0.1 × input rate + 0.02 × output rate. OpenAI's whole-request long-context modifiers are applied because input context totals one million tokens. Model | Scenario B marginal cost | DeepSeek V4.1 Flash, off-peak | $0.0297 | DeepSeek V4.1 Flash on Modelflare | $0.0383 | DeepSeek V4.1 Flash, peak | $0.0594 | GPT-5.6 Luna | $0.1120 | Claude Sonnet 5 | $0.5800 | GPT-5.6 Terra | $1.1200 | Claude Opus 5 | $1.4500 | GPT-5.6 Sol | $2.1200 | Claude Fable 5.1 | $2.2250 | GPT-6 Astra | $5.3000 | The comparison holds token quantities equal; it does not claim equal quality. DeepSeek's cache is best effort, while Anthropic cache creation has its own write price and lifetime. A fair workload test records cache hit/miss tokens, all attempts, total output including billable reasoning, elapsed time, acceptance rate and human correction minutes. The cost-tracking guide (/articles/ai-api-cost-tracking) provides the accounting frame. Open weights do not make self-hosting small The Hugging Face release (https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) is MIT-licensed and documents vLLM and SGLang serving paths. In the snapshot checked for this article, the repository contained 48 safetensor shards and about 510.3 GB of Git-LFS files at commit dba1be0a40aa45a94ad051997016db3960a90277. That byte count is a download artifact, not measured GPU RAM. The system still has a 552B backbone, 196B host-addressed Engram memory, expert routing, FP4/FP8 cache paths, RDMA prefetch and persistent KV storage. DeepSeek's release invites teams planning a large-scale deployment with 2,000 GPUs plus a storage cluster to contact the company. That example signals the scale of the full serving system; it is not a stated minimum for every deployment. Smaller research deployments may use quantization or different parallelism, but they should not be inferred from the provider's API economics. Open weights give teams checkpoint access, inspectability and control over deployment policy. They do not automatically reproduce DeepSeek's kernel fusion, cache hierarchy, DSpark scheduling, batching or public API latency. Compare a self-host plan with measured throughput, failure recovery, idle capacity, storage bandwidth and operations labor. DeepSeek V4.1 Flash on Modelflare DeepSeek V4.1 Flash is live on Modelflare. The operator-confirmed launch is independently visible in the public Modelflare pricing catalog (https://modelflare.dev/api/pricing) and the origin catalog. At 2026-09-10 15:21 UTC, both exposed the new deepseek-v4.1-flash-0910 record; the production configuration showed an enabled deepseek-stable ability and an enabled route advertising the same model. Modelflare item | Live value | Model ID to send | deepseek-v4.1-flash-0910 | Availability | Live in the public catalog and enabled routing group | Public group | deepseek-stable | Input price | $0.193548 / 1M tokens | Cached-input price | $0.003871 / 1M tokens | Output price | $0.774194 / 1M tokens | Endpoint types | /v1/chat/completions and /v1/responses | The prices are derived from the live entry's $0.322580645 base input coefficient, 0.6 public group ratio, 0.02 cache ratio and 4× completion ratio. For the two worked workloads above, Modelflare costs about $0.0271 and $0.0383 respectively. Use the live DeepSeek V4.1 Flash pricing page for the current model ID and tariff. Evidence levels still matter. The public catalog and enabled production route establish product availability, and the operator explicitly confirmed launch. By 15:30 UTC, production records contained six completed Chat Completions calls under the exact new model ID, totaling 216 prompt tokens and 173 completion tokens. That verifies authenticated Chat Completions dispatch and billable output. It is still too small a launch sample to measure latency or availability, and it does not independently prove Responses or multimodal behavior through this route. Teams should run their own small acceptance request on the exact endpoint before moving production work. Where DeepSeek V4.1 Flash has a real advantage The model has a coherent advantage when long inputs, frequent tool turns and budget all matter. The architecture reduces prefill and cache pressure; the API price turns those savings into an unusually low public tariff; the 384K output ceiling leaves headroom for long patches or reports; and the checkpoint keeps a self-host path open. Workload | Why V4.1 Flash is a strong candidate | What to measure | Repository agents | Strong publisher-reported DeepSWE and terminal results at low token rates | Accepted changes, test pass rate, retries and repair time | Long research synthesis | 1M input, cheap cache hits and 384K output | Citation correctness, requirement retention and total output | Visual document agents | Native images, OCR/chart training and common agent APIs | Field accuracy, crop/detail sensitivity and visual-token usage | High-volume automation | Low peak and off-peak prices; 2,500-request concurrency ceiling | Queue time, 429 rate, tool failures and cost per success | Controlled deployment | MIT checkpoint and documented vLLM/SGLang paths | Hardware utilization, cache/storage bandwidth and operational cost | The strongest production claim is therefore cost-efficient long-context agency, not “best model overall.” If the model completes a real task in one accepted run where a cheaper small model needs several repairs, V4.1 wins on outcome. If a frontier closed model prevents a costly failure, its higher token price can still be economical. Where OpenAI or Anthropic remains the safer choice Choose an OpenAI model when the application depends on the full Responses ecosystem, managed state, hosted tools, or an OpenAI-specific contract that DeepSeek ignores. GPT-5.6 Luna is a particularly close price competitor for shorter, uncached workloads; Terra, Sol and Astra trade much higher list prices for different capability tiers and native platform features. Choose an Anthropic model when Claude's native tool/thinking contract, cache controls or measured performance on your hardest agent work matters more than token list price. In DeepSeek's own table, Claude Opus 5 leads Terminal-Bench 3/4 and Chartography; Claude Fable 5.1 is positioned for the most demanding long-horizon work, though it is substantially more expensive per new and output token. For safety-critical or costly actions, route selection should be based on evaluated failure modes and permission boundaries, not brand or benchmark averages. Use the routing guide (/articles/reliable-ai-api-routing) to separate retryable work, fallbacks and actions with side effects. Deployment checklist Use deepseek-flash for the first-party API and record any gateway-specific mapping separately. Start at high effort, then measure low and max on the same accepted-task set; do not map names across providers as equal compute. When tools are used in thinking mode, return the full reasoning_content exactly as required. Audit every Responses field you depend on; rejected and silently ignored parameters need different handling. Log cache-hit and cache-miss tokens, output/reasoning tokens, retries, latency and final acceptance. Design cache economics around observed hits rather than a fixed retention assumption. Enforce image byte, count, dimension and detail limits before sending multimodal requests. Keep application-side authorization, tool idempotency and output validation independent of model quality. Recheck time-window pricing, model aliases and concurrency limits before launch. Run a real authenticated request on the exact route; a model catalog or HTTP 200 alone is not capability proof. Frequently asked questions Is DeepSeek V4.1 Flash better than GPT-6 Astra or Claude Fable 5.1? Not as a universal claim. V4.1 Flash has a large price and open-weight advantage and strong provider-reported agent scores. DeepSeek itself says the biggest closed models retain an edge on the hardest reasoning and edge cases. No same-harness Astra/Fable comparison is available in the V4.1 report. How many parameters does DeepSeek V4.1 Flash have? The report lists a 552B-parameter backbone plus 196B Engram parameters. It activates about 8B backbone parameters per prefill token and 16B per decoded token. Saying only “552B total” omits Engram; saying “748B active” is also wrong. What are its context and output limits? The official limits are one million context tokens and up to 384K output tokens. Image embeddings, text, tool history and reasoning all consume relevant budgets. Why is it called Flash? The design focuses on lower serving cost: asymmetric prefill/decode activation, compressed sparse attention, FP4 global KV, bounded persistent replay and speculative decoding. “Flash” does not promise the lowest latency for every prompt or queue condition. Does the public cache last 72 hours? Do not assume that. The technical report describes a deployment design with at least 72-hour global KV persistence. The public API guide calls caching best effort and says entries are generally cleared after hours or days. Billing should use observed hit/miss fields. Is the Responses API a drop-in replacement for OpenAI Responses? It accepts a familiar shape, but it is stateless and ignores multiple OpenAI fields and hosted tools. Test the precise capability contract you need. Can it generate images? It understands image input and returns text. It is not an image-generation model. Can I call V4.1 Flash through Modelflare today? Yes. Use deepseek-v4.1-flash-0910 in the deepseek-stable group with Chat Completions or Responses. This is Modelflare's release-stamped ID; the first-party deepseek-flash alias is a separate provider contract. Sources, method and update log This article prioritizes first-party sources and separates documented facts, DeepSeek-reported evaluations, calculated examples and observed Modelflare catalog state. No independent model benchmark was run. Cost math is preserved in the companion source artifact, and every scenario states its token vector and exclusions. The wordless cover is an AI-generated editorial illustration in the existing Modelflare visual style. Primary references: DeepSeek V4.1 Flash release (https://api-docs.deepseek.com/news/news260910/), technical report (https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf), model card (https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash), pricing (https://api-docs.deepseek.com/quick_start/pricing/), thinking (https://api-docs.deepseek.com/guides/thinking_mode/), Responses (https://api-docs.deepseek.com/guides/responses_api/), Anthropic compatibility (https://api-docs.deepseek.com/guides/anthropic_api/), vision (https://api-docs.deepseek.com/guides/vision/), context caching (https://api-docs.deepseek.com/guides/kv_cache/), OpenAI model comparison (https://developers.openai.com/api/docs/models/compare), Anthropic pricing (https://platform.claude.com/docs/en/about-claude/pricing), and the Modelflare public catalog (https://modelflare.dev/api/pricing). 2026-09-10 update: initial publication-day review. It records the first-party model ID, effective peak/off-peak pricing, September 14 alias migration notice, current API limits, technical-report architecture and benchmark data. A later same-day update adds the confirmed Modelflare launch under deepseek-v4.1-flash-0910, live gateway prices and worked gateway costs. Recheck changeable facts before implementation. ### GPT Image 2.5 Deep Dive: Flare vs Sunburst, Pricing, and Rival Models Canonical URL: https://modelflare.dev/articles/gpt-image-2-5-review-comparison GPT Image 2.5 is an image-generation family, not one API model. OpenAI released it on September 8, 2026 with two IDs: gpt-image-2.5-flare for fast, high-quality everyday work and gpt-image-2.5-sunburst for the highest quality and tighter control in demanding edits. Flare is the sensible first test for throughput; Sunburst is the first test when identity, product geometry, layout, or a long edit chain keeps failing. That does not make either model the automatic winner over Google Nano Banana, FLUX, Midjourney, or Adobe Firefly. GPT Image 2.5 has a strong unified generation-and-editing API, flexible sizes, transparent output, and partial-image streaming. Google offers broader grounded multimodal inputs, FLUX offers exact dimensions and open-weight options elsewhere in its family, Midjourney centers aesthetic exploration and personalization, and Firefly centers Adobe production workflows. The right unit of comparison is the cost per accepted image in a real workflow, including retries and human repair. What GPT Image 2.5 actually names OpenAI's launch announcement (https://openai.com/index/introducing-chatgpt-images-2-5/) uses “ChatGPT Images 2.5” for the product experience and introduces two separate API models. There is no documented generic API ID named gpt-image-2.5. Applications must select Flare or Sunburst, or pin their September 8 snapshots. Role | Alias | Dated snapshot | OpenAI positioning | Fast default | gpt-image-2.5-flare | gpt-image-2.5-flare-2026-09-08 | Small model for fast, high-quality everyday generation | Quality default | gpt-image-2.5-sunburst | gpt-image-2.5-sunburst-2026-09-08 | Base model for maximum quality and editing precision | The aliases can improve over time; snapshots protect reproducibility. Store the exact model ID, request settings, and returned usage beside every accepted asset. A marketing name without those fields is not enough to reproduce cost or output. Flare versus Sunburst OpenAI's prompting guide (https://developers.openai.com/api/docs/guides/image-prompting) describes Flare as the smaller, speed-optimized model with image quality comparable to GPT Image 2, and Sunburst as the base model with higher image quality than GPT Image 2. Both improve precise editing and subject preservation. The launch page says Flare reaches higher quality than GPT Image 2 at up to 50% lower latency, but that is a provider claim, not a fixed service-level guarantee for every prompt. Decision | Start with Flare | Start with Sunburst | Existing GPT Image 2 workflow already passes quality checks | Yes; test whether latency falls without losing acceptance | Only if a difficult case still fails | High-volume social, product exploration, visual search, or prototypes | Usually | Use for the subset that fails | Exact product, face, typography, or multi-step edit preservation | Benchmark it after a quality baseline exists | Yes; establish acceptable quality first | Cost-sensitive production | Measure accepted-image cost; the token rate is the same | Keep only when its quality gain avoids enough retries or repair | Migration sequence | Flare -> Sunburst for failures | Sunburst -> Flare after quality passes | This sequence is more useful than a permanent model ranking. Start with the cheaper operational path only after quality criteria are explicit, and retain Sunburst where its higher capability changes the acceptance rate. What changed from GPT Image 2 The release focuses on sharper detail, more natural lighting and texture, better preservation of reference subjects, and more reliable localized edits across multiple turns. OpenAI also highlights improved infographic accuracy and layout. Those are documented improvements, but this article did not run a paid head-to-head benchmark and does not present the provider gallery as independent evidence. The API contract also expands. GPT Image 2.5 adds xhigh and max above the earlier low, medium, and high quality levels. It accepts custom dimensions within a 4K pixel budget and supports both the Image API and the Responses API image-generation tool. The Responses path is useful when an application needs conversational, multi-step editing; it also adds the main language model's token usage to the image bill. The most important migration improvement is procedural: OpenAI now recommends preserving a representative baseline, testing the complete edit sequence, measuring consistency across repeated attempts, and tuning one setting at a time. That is also the correct way to compare another provider. API, output, and editing contract The image-generation guide (https://developers.openai.com/api/docs/guides/image-generation) documents generation and edits, one or more reference images, masks, transparent backgrounds, custom dimensions, and streaming partial images. The same visible quality label does not imply the same visual result or latency across Flare and Sunburst. Boundary | GPT Image 2.5 contract | APIs | Images API for direct generation/editing; Responses API image tool for conversational and multi-step flows | Quality | auto, low, medium, high, xhigh, max | Common sizes | 1024x1024, 1536x1024, 1024x1536; custom WIDTHxHEIGHT is supported | Custom-size limits | Each edge <= 3,840 px; multiples of 16; aspect ratio at most 3:1; 655,360 to 8,294,400 total pixels | Experimental size zone | More than 3,686,400 pixels (2560x1440) | Formats | PNG by default; JPEG and WebP with optional compression | Transparency | background="transparent" with PNG or WebP; inspect the decoded alpha edge | Streaming | Zero to three partial images; every partial adds 100 image-output tokens | Multiple outputs | n can request more than one image in the Image API | Masks | A guide to the edit, not a promise of pixel-exact boundaries | Repeated edits can still alter details that should remain fixed. Restate preserved properties, inspect every step, and composite a verified local edit back onto the original when a region must remain pixel-identical. GPT Image 2.5 pricing without a misleading flat number Both models use the same published rates: $5 per million text-input tokens, $1.25 per million cached text-input tokens, $8 per million image-input tokens, $2 per million cached image-input tokens, and $30 per million image-output tokens. Equal rates do not mean every request has the same bill. Size, quality, references, partial images, retries, and the selected API path all affect usage. The official calculator produces the following output-only estimates for an explicit 1024×1024 request. They exclude prompt input, reference-image input, partial images, retries, and any Responses API main-model tokens. Quality | Output tokens | Image-output cost | low | 196 | $0.00588 | medium | 439 | $0.01317 | high | 1,756 | $0.05268 | xhigh | 3,122 | $0.09366 | max | 7,024 | $0.21072 | auto cannot be budgeted from one fixed row because the selected setting depends on the request. Use the response usage as the billing evidence. For a production budget, calculate (all input + all output + partials + retries) / accepted images, then add human review time. A nominally faster model can cost less per accepted result even at the same token rate, while a higher-quality model can win when it prevents expensive repair loops. GPT Image 2.5 compared with current alternatives This matrix compares documented contracts as checked on September 9, 2026. It does not turn provider claims into a shared benchmark. Prices use different units and include the scope shown; they are orientation points rather than a normalized quality score. Model or family | Strongest documented fit | Output/control | Automation and cost boundary | GPT Image 2.5 Flare | Fast general generation and editing | Custom sizes to 4K budget, transparent output, multi-turn editing, partial streaming | API; 1024 square output from $0.00588 at low to $0.21072 at max, before input and retries | GPT Image 2.5 Sunburst | Premium assets and difficult preservation-sensitive edits | Same API controls, with quality-first positioning | Same token rates; measure whether fewer rejected edits justify the slower path | Gemini 3.1 Flash Image / Nano Banana 2 | Grounded, high-volume multimodal creation | Text, image, video, and PDF input; 0.5K/1K/2K/4K; very wide ratios; Web and Image Search grounding | API and Batch; Google lists $0.067 for a standard 1K image, plus input/text/thinking and optional search | Gemini 3 Pro Image / Nano Banana Pro | Complex layouts, localization, knowledge-grounded professional design | Thinking, Search grounding, 1K/2K/4K, text and image output | API and Batch; $0.134 for standard 1K/2K image output and $0.24 for 4K, plus other tokens/search | FLUX.2 Pro / Max | Exact dimensions, multi-reference product work, and controlled production | Up to 4 MP, arbitrary aspect ratio, up to 8 API references; Max adds grounding, Flex specializes in typography | API; Pro text-to-image from $0.03 and Max from $0.07, with megapixel/reference billing | Midjourney V8.2 | Aesthetic exploration, style references, moodboards, and personalization | 1K SD or native/upscaled 2K HD; web/Discord editor and a newer edit model | Subscription $10-$120/month; generally no public API and automated access is prohibited | Adobe Firefly Image 5 | Adobe-centered brand and production workflows | Native 4 MP, text-to-image, instruct edit, custom models, Creative Cloud ecosystem | Firefly API and plans use their own contracts/credits; this review does not invent a comparable flat API price | Google's image guide (https://ai.google.dev/gemini-api/docs/image-generation) makes Nano Banana 2 especially distinct when video or PDF should become image context, or when Search grounding is part of the workflow. BFL's FLUX.2 overview (https://docs.bfl.ai/flux_2/flux2_overview) is distinct when exact dimensions, many references, hex colors, or local/open-weight options in the broader FLUX family matter. Midjourney's current documentation (https://docs.midjourney.com/hc/en-us/articles/32199405667853-Version) makes it a creator product rather than an API substitute. Adobe's Firefly API (https://developer.adobe.com/firefly-services/docs/firefly-api/) is strongest as a production-system choice when its model, custom assets, and Adobe tooling need to stay together. Which model to choose by workflow Choose Flare first for interactive applications, social assets, catalog exploration, and high-volume generation where the acceptance threshold is already known. Escalate only failed categories to Sunburst instead of paying the slowest path for every image. Choose Sunburst first for final campaign creative, product geometry, face or character preservation, dense layouts, and long edit chains. After it passes, rerun the same sample on Flare. Keep Sunburst only where the difference survives repeated trials. Compare Nano Banana 2 or Pro when the image depends on current web context, image search, video/PDF inputs, very wide aspect ratios, or text-plus-image responses. Grounding can improve relevance, but a grounded image can still contain incorrect labels or data; verify every factual visual. Compare FLUX.2 when exact pixel dimensions, up to eight reference images, color control, or self-hosting/open weights elsewhere in the family are requirements. Pro and Max are hosted commercial products; the locally runnable Klein and Dev variants have different capability and license boundaries. Choose Midjourney V8.2 when a human creator values rapid aesthetic search, moodboards, style references, and personalization more than API automation. Do not build an automated product dependency around unofficial wrappers. Choose Firefly Image 5 when Adobe editing, custom brand models, production governance, and its licensed-data posture are decision criteria. “Commercially safe” is Adobe's positioning, not a universal legal warranty for every prompt or jurisdiction; teams still need rights and policy review. A fair benchmark for your own images Use 30-100 representative tasks, not one showcase prompt. Split them into generation, single-image edit, multi-reference composition, exact text, localization, transparent asset, and multi-turn preservation. Freeze prompts, reference files, dimensions, quality, output format, and acceptance rules before seeing model names. Measure | Record per attempt | Why it matters | Acceptance | Pass/fail plus reason | Prevents aesthetic preference from hiding task failure | Instruction accuracy | Missing, added, misplaced, or malformed elements | Measures whether the requested image was delivered | Preservation | Identity, product geometry, labels, lighting, and unchanged regions | Reveals edit drift across the complete chain | Text/localization | Character, spelling, grammar, and leftover-language errors | Separates attractive layout from usable copy | Latency | Median, p95, timeout, and queue time | Averages hide slow-tail workflow pain | Cost | All input/output, partials, failed attempts, and retries | Produces actual cost per accepted image | Human work | Review and repair minutes | Often dominates the API price difference | Randomize review order and hide the model name where possible. Retain every output, including refusals and failed edits. Report confidence intervals or at least sample counts; do not average unrelated criteria into one “quality score” unless the weights were fixed before evaluation. The source package includes evaluation-scorecard.csv for this record. Limitations, safety, and provenance GPT Image 2.5 remains probabilistic. Higher quality does not guarantee a better result for every prompt. Exact spelling, small faces, counts, spatial relationships, historical details, infographics, and translated copy still require review. A mask constrains intent without guaranteeing a pixel-exact boundary, and repeated edits may drift. OpenAI applies checks before and after generation. Its system card (https://deploymentsafety.openai.com/chatgpt-images-2-5/safety-evaluations) says the greater realism can raise deepfake risk and describes layered prompt, input, and output safeguards. It also documents C2PA metadata plus invisible watermarking. Google says Gemini-generated images include SynthID. Provenance signals help disclose origin, but they do not prove factual accuracy, consent, or rights. Safety behavior affects production metrics: a blocked prompt is not a successful zero-cost image, and blindly retrying the same request wastes budget. Record refusals separately, revise user-correctable inputs, and never weaken safeguards just to improve an acceptance chart. Current Modelflare boundary The Modelflare source tree checked on September 9 declares gpt-image-2 in its image-generation contract. It does not yet declare gpt-image-2.5-flare or gpt-image-2.5-sunburst. This source package therefore does not claim either new ID is available, priced, routed, or verified through Modelflare. Before integrating, check the live Models & Pricing (/pricing) catalog and the Image Workbench (https://docs.modelflare.dev/models/image/), then verify the exact model, endpoint, quality values, custom-size rules, streaming events, usage fields, billing, and edit behavior with an authorized real request. A catalog row alone would still not prove a completed image workflow. For the general contract discipline, use the OpenAI-compatible API guide (/articles/openai-compatible-api-guide), routing guide (/articles/reliable-ai-api-routing), and cost tracking guide (/articles/ai-api-cost-tracking). Questions teams ask Is gpt-image-2.5 a valid API model ID? Not in the checked official documentation. Choose gpt-image-2.5-flare or gpt-image-2.5-sunburst, or their dated snapshots. Is Flare always cheaper than Sunburst? They share token rates, but final cost depends on usage, retries, and acceptance. Measure the response usage and rejected-image rate. Is Sunburst always better? OpenAI positions it for higher quality, but a slower, higher setting is wasteful when Flare already passes the workflow's acceptance checks. Does GPT Image 2.5 support 4K? It supports a total budget up to 8,294,400 pixels with a 3,840-pixel edge limit. Outputs above 2560x1440 are experimental, so “4K” needs exact dimensions and testing rather than a blanket promise. Which competitor is closest? Nano Banana 2/Pro is the closest conversational, grounded multimodal comparison; FLUX.2 is strong for controlled production and reference-heavy work; Midjourney is a creator workflow; Firefly is an Adobe production workflow. The closest model changes with the task. Can this article prove image quality? No. It compares current documented contracts and supplies a reproducible evaluation method. It contains no paid generation benchmark. Sources and update record Primary sources checked on September 9, 2026: OpenAI: ChatGPT Images 2.5 launch (https://openai.com/index/introducing-chatgpt-images-2-5/), Flare model (https://developers.openai.com/api/docs/models/gpt-image-2.5-flare), Sunburst model (https://developers.openai.com/api/docs/models/gpt-image-2.5-sunburst), image generation (https://developers.openai.com/api/docs/guides/image-generation), image prompting (https://developers.openai.com/api/docs/guides/image-prompting), and system card (https://deploymentsafety.openai.com/chatgpt-images-2-5/safety-evaluations). Google: Gemini image generation (https://ai.google.dev/gemini-api/docs/image-generation), Nano Banana 2 model (https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-image), Nano Banana Pro model (https://ai.google.dev/gemini-api/docs/models/gemini-3-pro-image), and Gemini API pricing (https://ai.google.dev/gemini-api/docs/pricing). Black Forest Labs: FLUX.2 overview (https://docs.bfl.ai/flux_2/flux2_overview) and pricing (https://bfl.ai/pricing?category=flux.2). Midjourney: V8.2 version contract (https://docs.midjourney.com/hc/en-us/articles/32199405667853-Version), plans (https://docs.midjourney.com/hc/en-us/articles/27870484040333-Comparing-Midjourney-Plans), and automation/API boundary (https://docs.midjourney.com/hc/en-us/articles/32013696484109-Community-Guidelines). Adobe: Firefly API overview (https://developer.adobe.com/firefly-services/docs/firefly-api/) and Image 5 generation/editing guide (https://developer.adobe.com/firefly-services/docs/firefly-api/guides/how-tos/cm-generate-image/feature-guide). Update record: September 9, 2026: Initial documented-contract review, pricing calculation, competitor matrix, evaluation scorecard, and Modelflare source boundary. Recheck model aliases, pricing, independent evidence, and Modelflare availability immediately before publication. ### How to Test an OpenAI-Compatible API Canonical URL: https://modelflare.dev/articles/openai-compatible-api-conformance-tests Production guide: How to Test an OpenAI-Compatible API. It includes a deterministic artifact, failure boundaries, rollout checks, and source-qualified limitations. Decision first How to Test an OpenAI-Compatible API is an explicit production contract, not an isolated code change. Define the success signal, terminal failure state, and rollback condition before moving traffic. The artifact below separates evidence from assumption and keeps every attempt attributable. Start with text, prove stream under a deterministic case, and make errors a release gate rather than a follow-up task. The reusable artifact Use this table as the review record. A row passes only when its evidence comes from the same request, test window, or configuration snapshot. Checkpoint | Evidence to capture | Pass condition | text | non-stream_response,finish_state,empty_and_Unicode_input | The value is exact, preserved, and compared at the wire boundary. | stream | SSE_event_order,terminal_marker,usage_placement,disconnect | The record joins to one logical request and one concrete attempt. | tools | single_and_parallel_calls,argument_schema,call_correlation | The record joins to one logical request and one concrete attempt. | schema | valid_object,refusal,truncation,unsupported_keyword | The limit is explicit and the request fails closed when it is exceeded. | usage | input,output,cache_read/write,missing_categories | Reserved, observed, and final values reconcile through one durable ledger. | errors | 401,403,429,499,5xx_outer_and_embedded_failures | The owner, source, checked date, and known limitation are recorded. | Worked example The example is synthetic and deterministic. Replace identifiers and thresholds with reviewed values from your own workload; never paste production secrets or customer prompts into a test fixture. corpus_version: 1 cases: - id: text/non_stream_zero_values request: { stream: false, temperature: 0 } assert: [http_status, response_shape, explicit_zero_preserved] - id: stream/disconnect action: cancel_after_first_content_delta assert: [client_cancelled, upstream_cancelled, terminal_state_recorded] - id: tools/two_calls assert: [stable_call_ids, arguments_validated, results_correlated] - id: errors/rate_limit assert: [status_429, retry_after_parsed, attempt_budget_respected] Implementation procedure Freeze the current request, response, configuration, and observable baseline before changing behavior. Run one deterministic positive case and retain the complete client-visible result. Run the paired negative or limit case so failure behavior is proved rather than inferred. Join every attempt to one logical request ID, then record timing, terminal state, and usage without sensitive content. Roll out through a bounded cohort with explicit stop conditions; do not widen traffic merely because the happy path passed. Re-read the durable state and public behavior after the change, and execute the prepared rollback if any invariant fails. Failure modes The following failures invalidate the result even when the outer HTTP request appears successful: A single text response is treated as proof of protocol compatibility across streaming, tools, schema, usage, and errors. A zero or false value disappears during parsing or re-serialization, changing the caller's explicit intent. The integration reads one convenience field and silently drops typed output items, tool calls, refusals, or partial results. Independent retry layers multiply attempts and turn a small 429 episode into sustained overload. Modelflare boundary Modelflare can centralize OpenAI-compatible routing, keys, groups, usage records, and failure handling, but a configured route is not proof that every upstream supports every optional field. Verify the selected model and channel with the native protocol path. Keep provider prices in the current pricing source, preserve explicit zero values, and treat the final durable settlement—not an estimate—as billing truth. Use the parent guide (/articles/openai-compatible-api-guide) for the broader decision boundary and the setup documents (/docs) for current client configuration. Publication checklist The main question is answered before background material. Every request field, status, metric, and formula has one named owner. Examples use synthetic identifiers and contain no credential or customer data. All supported locales preserve the same headings, tables, code, limits, and caveats. Volatile API contracts, model support, and prices are rechecked at T-1 day; an invalidated claim moves the publication date. Before the scheduled time, the public API, localized routes, and sitemap still exclude the article. Sources and verification date Sources were checked on 2026-08-07. They establish the external contract or operating principle; they do not prove support on an untested provider route. OpenAI API reference (https://developers.openai.com/api/reference) WHATWG Server-Sent Events (https://html.spec.whatwg.org/multipage/server-sent-events.html) JSON Schema Draft 2020-12 (https://json-schema.org/draft/2020-12) ### GPT-6 Astra vs Claude Fable 5.1: Specs, Benchmarks, Pricing, and Model Position Canonical URL: https://modelflare.dev/articles/gpt-6-astra-vs-claude-fable-5-1 OpenAI released GPT-6 Astra on September 3, 2026. OpenAI describes it as its most intelligent and aligned model and the most capable model it has broadly deployed. Two days earlier, Anthropic released Claude Fable 5.1, a frontier model aimed at long-running coding, research, and knowledge work. Both models target difficult reasoning and agent workflows, but they make different trade-offs. This article answers the practical comparison question: what do GPT-6 Astra and Claude Fable 5.1 actually offer, where is the measurable progress, and which model should a team choose now? The factual cutoff for this review is September 4, 2026, Asia/Shanghai. Provider claims, provider-reported benchmark scores, and the live Modelflare catalog are kept separate. One important clarification comes first. Neither OpenAI nor Anthropic publishes a verified total-parameter count, active-parameter count, layer count, or training-FLOPs figure for these models on the official pages reviewed here. The “parameters” comparison below therefore means public model specifications, not an invented parameter number. Executive answer GPT-6 Astra is the broader end-to-end flagship. It is the better starting point when the work must move through a browser or desktop application, coordinate several tools, edit and test software, or operate under strict authorization and safety boundaries. Its API adds async tool calling, mid-turn steering, and effort changes that preserve the cached prompt prefix. Claude Fable 5.1 remains a top-tier alternative rather than a model that became obsolete overnight. It is especially attractive for long-horizon coding, research, documents, spreadsheets, and slides. Its official cache-read price is $0.25 per million tokens, one quarter of Fable 5’s rate and lower than GPT-6 Astra’s $1 cache-read rate. For a cache-heavy agent that repeatedly rereads the same context, that difference can matter more than a headline benchmark win. There is no universal “best” model. A useful rule is: choose GPT-6 Astra for broad computer-use, tool orchestration, coding, science, and safety-sensitive agent workflows; choose Claude Fable 5.1 for long research and artifact production, cache-heavy sessions, or an existing Claude Messages integration; choose neither by name alone when latency, throughput, or cost per successful task is the real constraint—measure the complete workflow, including retries, tool turns, and human review. Public specifications: what is known and what is not The numbers in this table come from the OpenAI GPT-6 Astra model page (https://developers.openai.com/api/docs/models/gpt-6-astra), the OpenAI model guidance (https://developers.openai.com/api/docs/guides/latest-model?model=gpt-6-astra), and Anthropic’s Fable 5.1 overview (https://platform.claude.com/docs/en/models/fable-5-1/overview) and pricing documentation (https://platform.claude.com/docs/en/about-claude/pricing). Public specification | GPT-6 Astra | Claude Fable 5.1 | Practical reading | Provider model ID | gpt-6-astra | claude-fable-5-1 | Use the exact ID; a product nickname is not an API contract. | Release date | September 3, 2026 | September 1, 2026 | Both are current-generation releases at this review date. | Published parameter count | Not disclosed | Not disclosed | Do not repeat third-party parameter numbers as official facts. | Context window | 1,050,000 tokens (1.05M) | 1,000,000 tokens (1M) | Astra is slightly larger on paper; retrieval quality still depends on the task. | Maximum output | 128,000 tokens | 128,000 tokens | The output ceiling is effectively the same. | Reliable knowledge cutoff | April 30, 2026 | June 2026 | Neither cutoff is a substitute for current retrieval. | Input → output | Text and images → text | Text and images → text | Neither model page lists native audio or video output for this comparison. | Reasoning control | low, medium, high, xhigh, max | Adaptive thinking, always on; default high | The controls are not interchangeable across providers. | Standard input / output | $10 / $50 per MTok | $10 / $50 per MTok | Sticker prices match before cache, batch, or gateway modifiers. | Cache read | $1 / MTok | $0.25 / MTok | Fable 5.1 has the lower official cache-hit price. | Cache writes | $12.50 / MTok | $12.50 (5-minute) or $20 (1-hour) / MTok | Compare cache lifetime as well as the write rate. | Long-context billing | Above 272K input tokens: 2× input/cache and 1.5× output for the full request | Full 1M context at standard per-token pricing | A large context window does not imply identical economics. | Provider availability | Trusted Access rollout, followed by API and paid ChatGPT plans | Active latest model for all Claude API customers and partner platforms | Provider availability is separate from Modelflare availability. | The table also shows why “same $10/$50 price” is an incomplete comparison. GPT-6 Astra offers a larger nominal context window and a rich tool surface, while Fable 5.1 makes repeated cache reads unusually inexpensive and includes its full 1M window at standard pricing. Neither provider has published a parameter figure that can be responsibly compared. What GPT-6 Astra changes End-to-end execution The central GPT-6 Astra upgrade is workflow execution. The model is documented for complex reasoning, software engineering, browsing, computer use, research, and document creation. In the Responses API it supports web search, file search, image generation, code interpreter, hosted shell, apply-patch, computer use, MCP, tool search, streaming, function calling, and structured outputs. The new API behaviors are designed for tasks that do not fit one request-and-answer turn: Async tool calling lets the model continue independent reasoning while an application runs a tool, then resume with the original call ID. Mid-turn steering lets a client send a correction while the model is working over a WebSocket continuation, preserving completed work. Dynamic effort lets an application raise or lower reasoning effort mid-conversation without rebuilding the cached prompt prefix. These are meaningful changes for agent developers because they move control from “retry the whole prompt” toward “continue the current task with a bounded update.” The application still owns tool execution, authorization, timeouts, and side effects. Computer use and professional artifacts OpenAI reports that Astra reaches 72.6% on its OSWorld 2.0 offline partial score versus 65.7% for GPT-5.6 Sol, with roughly 47% less simulated task time in that comparison. In a Codex harness, OpenAI reports 1.9× faster task completion on Mind2Web after combining the model and harness improvements. These are provider-reported evaluations, not a Modelflare latency guarantee. The intended output is not only a paragraph of text. Astra is trained to produce usable documents, spreadsheets, presentations, plots, websites, and tested code while following an existing template or style. That makes it a strong candidate when the last mile—checking a UI, updating a file, or validating a result—matters as much as the reasoning in the middle. Safety and control are part of the release GPT-6 Astra is the first OpenAI model that the company says reaches the Critical level of cybersecurity capability in its Preparedness Framework. The same release adds stronger prompt-injection resistance in browsing and workplace settings, production misalignment monitoring for tool-using inference, and stricter isolation and monitoring around deployment. The safety story has an important nuance: capability and monitorability do not move in exactly the same direction. OpenAI reports fewer high-severity misalignment flags and better scope adherence than GPT-5.6 Sol, but also reports that Astra can control its written chain of thought more effectively in adversarial monitor-evasion tests. For production systems, this means model alignment is one layer; authorization, least-privilege tools, confirmation rules, audit logs, and a kill switch remain application responsibilities. What Claude Fable 5.1 still does differently Long-horizon work Anthropic positions Fable 5.1 as the successor to Fable 5 for long-running agentic coding, multistep research, and documents, spreadsheets, and slides. Its full 1M-token context is priced at standard rates, adaptive thinking is always on, and effort can be adjusted without invalidating the prompt cache. Fable 5.1 also improves vision work over dense charts, filings, and PDF tables, plus recovery from failed computer-use steps. Fable 5.1’s clearest economic difference is cache reuse. Its cache-read price is $0.25 per MTok, while Fable 5 remains at $1 per MTok. Anthropic says the same model is also available as Claude Mythos 5.1 with a more restricted safeguard profile, but Mythos access is limited to trusted programs; it should not be treated as a generally available substitute for Fable 5.1. API behavior Fable 5.1 is not a completely transparent model-name swap for a custom Messages client. Anthropic documents three breaking boundaries: forced tool use with tool_choice types any or tool returns a 400 error; earlier Claude models cannot read Fable 5.1 thinking blocks; editing an earlier message, system prompt, or tool list can invalidate later thinking blocks. The safe migration shape is append-only history, automatic tool choice with strict schemas, and turn-scoped system messages for temporary instructions. The release also adds per-message effort, readable progress updates, and content provenance. In long loops, Fable 5.1 can issue one tool call where Fable 5 batched several, so round-trip count should be measured rather than assumed. Benchmark comparison The following selected scores are reproduced from the OpenAI GPT-6 Astra launch comparison (https://openai.com/index/gpt-6-astra/), which includes Anthropic’s reported Fable 5.1 results where available. They show the shape of the current frontier, not a universal ranking. Evaluation | GPT-6 Astra | Claude Fable 5.1 | What the result suggests | AutomationBench | 41.4% | 31.4% | Astra leads this business-workflow evaluation. | BenchCAD | 95.9% | 84.3% | Astra leads the reported CAD/professional task set. | Terminal-Bench 4.0 | 57.9% | 55.8% | Both are strong; Astra has a modest reported lead. | DeepSWE v1.1 | 74.1% | 67.4% | Astra leads this software-engineering agent evaluation. | Terminal-Bench Science 0.1 | 64.6% | 52.6% | Astra’s reported advantage is larger on this science workflow. | FrontierMath Tier 4 (v2) | 97.6% | 87.8% | Astra reports a substantial math lead in this setup. | GPQA Diamond | 96.0% | 93.7% | Both are near the top; the gap is smaller. | Humanity’s Last Exam, with tools | 57.2% | 65.0% | Fable 5.1 leads this reported tool-assisted exam score. | HealthBench Professional | 63.4% | 58.1% | Astra leads this reported professional health evaluation. | ExploitGym | 42.4% | 30.4% | Astra reports a higher cyber capability score; use safeguards. | Artificial Analysis Intelligence Index v4.1.1 | 61.2 | 65.7 | The external index shown in the launch table favors Fable 5.1. | The exceptions matter. OpenAI’s own table shows Fable 5.1 ahead on Humanity’s Last Exam with tools and on the Artificial Analysis Intelligence Index. Several evaluations use different harnesses, effort settings, tools, task edits, and safety-intervention policies; some entries are missing or not directly comparable. A provider score is evidence about a defined setup, not proof that one model wins a team’s prompts. Cost and cache economics At the provider list level, both models charge $10 per million input tokens and $50 per million output tokens. The cost divergence appears in the modifiers: GPT-6 Astra cache reads are $1 per MTok; cache writes are $12.50 per MTok. Batch and Flex are listed at 50% of standard rates, while Fast mode is 2× the applicable standard price. Requests with more than 272K input tokens use the documented 2× input/cache and 1.5× output rates for the full request. Fable 5.1 cache reads are $0.25 per MTok; 5-minute and 1-hour writes are $12.50 and $20 per MTok. Its full 1M context is priced at standard rates, and Batch is 50% off input and output. For a simple cache-read comparison, one million cached input tokens cost about $1.00 on GPT-6 Astra versus $0.25 on Fable 5.1 at official list prices. Ten such reads leave a $7.50 difference before output, tool charges, gateway discounts, or retries. This is not a claim that Fable always produces a cheaper completed task: a slower loop, extra tool turns, or more output can erase the saving. Use a complete-task equation rather than a single token rate: task cost = input tokens + cache reads + cache writes + output tokens + tool or service charges Each term must be multiplied by the applicable price, and the result should be divided by successful tasks rather than requests. Tokenizers also differ, so character counts are not a reliable cross-provider budget. Workload decision matrix This matrix is the article’s original decision aid. It combines the documented feature surface, the selected benchmark evidence, the pricing mechanics, and the current access boundary; it is not a new benchmark. Workload | Starting choice | Why | Boundary to validate | Browser or desktop tasks with several tools | GPT-6 Astra | Broad computer-use surface, async tools, steering, and strong reported OSWorld results. | Confirm permissions, confirmation policy, latency, and side-effect recovery. | Repository-scale coding and migrations | GPT-6 Astra, with Fable 5.1 as a serious alternative | Astra leads the selected coding scores; Fable remains strong for long unattended sessions and root-cause work. | Replay the same repository, tests, tool loop, and rollback process. | Research memo, spreadsheet, or slide deck | Fable 5.1 for artifact-first work; Astra for browser-to-artifact workflows | Fable is explicitly optimized for long knowledge work; Astra is strong at end-to-end computer use and document creation. | Compare citation recall, layout quality, tool turns, and human edits. | Cache-heavy repeated context | Fable 5.1 | $0.25/MTok cache reads and standard pricing across its 1M window. | Check cache hit rate, write lifetime, tokenizer usage, and gateway billing. | Safety-sensitive defensive security work | GPT-6 Astra with approved safeguards | OpenAI documents stronger cyber capability and additional monitoring. | Use authorized targets, least privilege, audit logs, and human approval. | Multilingual writing or localization | No automatic winner | Neither provider’s release establishes a decisive new language advantage. | Run the same terminology-controlled corpus in every target language. | High-throughput or strict low-latency service | A smaller model may be better | Both are frontier-priced and effort-intensive; Fable is documented as slower, while Astra latency depends on mode and harness. | Measure p50/p95, first output, cost per successful task, and fallback rate. | Model position on September 4, 2026 The current model landscape is better described as a two-axis map than a single leaderboard. GPT-6 Astra is the broad flagship. Its public API, computer-use workflow, tool orchestration, science and coding results, long context, and safety controls place it at the front of general end-to-end agent work. OpenAI’s current model guidance tells developers to start with Astra for the hardest reasoning and coding tasks. Claude Fable 5.1 is a co-leading specialist and a practical alternative. It remains near the frontier on coding and knowledge work, wins some published evaluations, and has unusually favorable cache-read economics. It is not merely a legacy model waiting for replacement. “Best” depends on the unit of work. If the unit is a browser task completed under authorization, Astra’s tool and safety stack may dominate. If the unit is ten long research turns that reread a stable 1M-token prefix, Fable’s cache economics may dominate. If the unit is a finished artifact, human correction time can be more important than either raw score. Access is part of model status. A model can be a global frontier release while still being unavailable through a particular gateway, account, region, or protocol. Availability must be checked separately from capability. Modelflare availability and integration boundary The live Modelflare catalog was checked on September 4, 2026 through https://modelflare.dev/api/pricing (https://modelflare.dev/api/pricing). At that time, claude-fable-5-1 was listed with Anthropic- and OpenAI-compatible surfaces and three public routing groups. The same response did not list gpt-6-astra. This article therefore makes no Modelflare GPT-6 access or price promise; the live pricing page (/pricing) and model list remain the product source of truth. Modelflare catalog item | Current observation | User-facing input / output | Position | claude-award + claude-fable-5-1 | Listed, ratio 0.25× | $2.50 / $12.50 per 1M tokens | Lowest current Fable route; use when retryable work can tolerate variability. | claude-stable + claude-fable-5-1 | Listed, ratio 0.315× | $3.15 / $15.75 per 1M tokens | Balanced daily and production route. | claude-premium + claude-fable-5-1 | Listed, ratio 0.5× | $5.00 / $25.00 per 1M tokens | Stability-first Fable route. | gpt-6-astra | Not listed in the checked response | No public Modelflare price at check time | Do not hard-code access before the live catalog and a real endpoint probe agree. | For Fable 5.1, the existing Modelflare Fable 5.1 deep dive (/articles/claude-fable-5-1-review-pricing) covers route selection, Messages requests, and Fable-specific migration detail. For protocol choices, see Responses API migration (/articles/responses-api-migration-checklist), Responses API vs Chat Completions (/articles/responses-api-vs-chat-completions), and reliable AI API routing (/articles/reliable-ai-api-routing). If and when GPT-6 Astra appears in the Modelflare catalog, use the exact provider ID gpt-6-astra, verify the selected group, and test the endpoint your client actually needs. OpenAI’s guidance recommends the Responses API for tool calling and says to remove unsupported parameters such as temperature and top_p when migrating. A successful model-list response alone does not prove streaming, tools, structured outputs, or computer use. Limits, caveats, and responsible use Parameter uncertainty: total and active parameter counts are undisclosed. Context length, output length, and benchmark scores are not parameter counts. Benchmark uncertainty: the scores above are provider-reported or reproduced in a provider launch table. Harnesses, prompts, effort, tools, safety filters, and task versions can change the result. Protocol uncertainty: a model ID does not guarantee identical fields or event streams across Responses, Chat Completions, Anthropic Messages, Bedrock, Google Cloud, or a gateway compatibility layer. Freshness: both models have dated knowledge cutoffs. Use retrieval for current facts and record the source date. Safety: stronger models can make higher-impact mistakes. Restrict tools, separate planning from execution, require confirmation for irreversible actions, and keep an external audit trail. Pricing and access: provider prices, cache rules, routing groups, and availability can change. Recheck the live provider and Modelflare pages immediately before a production decision. FAQ Is GPT-6 Astra better than Claude Fable 5.1? GPT-6 Astra is the stronger broad flagship for end-to-end computer use, tool orchestration, coding, and science in the published launch evidence. Fable 5.1 remains competitive and leads some evaluations; it can be the better choice for cache-heavy, long-horizon knowledge work. How many parameters does GPT-6 Astra have? OpenAI has not published a verified total or active parameter count on the official model and launch pages reviewed for this article. Do not treat an online estimate as an official specification. Which model is cheaper? Their base input/output rates are both $10/$50 per million tokens. Fable 5.1 is cheaper for cache reads, while Astra has its own Batch, Flex, and Fast modifiers. The cheaper completed task depends on token volume, cache hits, tool turns, latency, and retries. Which model is better for coding agents? Start with GPT-6 Astra when the agent must browse, edit, test, and coordinate tools. Consider Fable 5.1 when the agent runs for a long time over a stable context or already uses the Claude Messages contract. Replay the same repository and tool loop before switching. Is GPT-6 Astra available on Modelflare? It was not listed in the live Modelflare pricing response checked on September 4, 2026. Official OpenAI release status and gateway availability are separate facts; check /pricing (/pricing) and the model list for a later update. Do both models have a 1M-token context window? Yes, both provider pages document a 1M-class window: 1.05M for GPT-6 Astra and 1M for Fable 5.1. Their billing, tokenization, retrieval behavior, and protocol limits are not identical. Sources and update record OpenAI: GPT-6 Astra — A new generation of intelligence (https://openai.com/index/gpt-6-astra/) OpenAI API: GPT-6 Astra model (https://developers.openai.com/api/docs/models/gpt-6-astra) OpenAI API: GPT-6 Astra model guidance (https://developers.openai.com/api/docs/guides/latest-model?model=gpt-6-astra) OpenAI: Safety overview for GPT-6 Astra (https://openai.com/index/safety-overview-gpt-6-astra/) Anthropic: Introducing Claude Fable 5.1 and Claude Mythos 5.1 (https://www.anthropic.com/claude-fable-and-mythos-5-1) Claude Platform: Fable 5.1 overview (https://platform.claude.com/docs/en/models/fable-5-1/overview) Claude Platform: What’s new in Fable 5.1 (https://platform.claude.com/docs/en/models/fable-5-1/whats-new-fable-5-1) Claude Platform: Fable pricing (https://platform.claude.com/docs/en/about-claude/pricing) Modelflare live models and pricing (/pricing) Update record: September 4, 2026 — initial publication package. Official GPT-6 Astra and Fable 5.1 specifications, launch comparisons, pricing rules, and availability statements were rechecked. The Modelflare catalog was checked at the same date; Fable 5.1 was listed and GPT-6 Astra was not. No independent benchmark reproduction or undisclosed parameter claim is made. ### Build an AI Coding Agent Gateway Canonical URL: https://modelflare.dev/articles/ai-coding-agent-api-gateway Build an AI Coding Agent Gateway: a production guide with an explicit decision, reusable artifact, failure tests, operating signals, and source-qualified limits. Choose the wire contract before the tool Build an AI Coding Agent Gateway should be implemented as a coding-agent integration contract, not as a one-off configuration. Freeze the protocol, ownership, evidence, and rollback condition before traffic moves. The concrete control points in this guide are protocol_per_agent, scoped_identity, attempt_budget, durable_usage. Use one gateway policy plane, but preserve each coding agent’s native wire contract. Give every human, CI job, or workspace a scoped identity; select only protocol-compatible routes; bound retries and tool side effects; and reconcile usage from durable attempt records. A universal base URL without those controls is a proxy, not an operable coding-agent gateway. Separate client setup from gateway policy Separate the reader-facing task from the control-plane work behind it. Client setup owns the local file or environment variable; the gateway owns authentication, routing, limits, accounting, and attempt records; the provider owns its native protocol and volatile capability contract. A passing text prompt proves only that one path worked once. A useful integration page owns protocol selection, configuration precedence, proof cases, and team rollout. The live setup document remains the owner of copy-paste installation steps. This division follows a capability-tree model: the article explains why and how to verify; the documentation supplies the current commands. The owner record for this page is ai-coding-agent-api-gateway; its frozen controls are protocol_per_agent, scoped_identity, attempt_budget, durable_usage. Every value is reviewed at the wire or durable-state boundary and never inferred from a marketing label. Compatibility contract: Build an AI Coding Agent Gateway Use the following review record as the deployable artifact. Technical values are intentionally explicit so a reviewer can compare configuration, wire evidence, and durable state without relying on a screenshot or a successful-looking outer response. Control point | Fixed decision | Evidence to retain | agent_identity | one_scoped_key_per_owner_or_workload | key_id + owner + expiry + allowed_groups | wire_contract | responses_or_messages_or_chat_selected_explicitly | captured_endpoint + content_type + terminal_event | model_policy | aliases_resolve_only_to_compatible_routes | alias_version + selected_channel + native_probe | attempt_budget | one_retry_owner_with_deadline | logical_request_id + attempt_sequence + remaining_deadline | tool_boundary | authorize_and_deduplicate_before_side_effect | call_id + policy_decision + idempotency_record | accounting | usage_and_final_charge_reconcile | provider_usage + normalized_usage + durable_settlement | Reference configuration or flow The example uses placeholders and deterministic inputs. Replace identifiers with reviewed values, never with credentials or customer content. Preserve the exact configuration snapshot alongside the probe result. agent -> protocol adapter -> policy gateway -> compatible route -> provider | | | | +-> attempt ledger +-> native request ID +-> scoped key usage + charge terminal event release gate: positive_probe: pass negative_probe: pass tool_side_effect_replay: no_duplicate rollback: tested Prove the complete coding-agent path Run the ladder in order. A later check cannot compensate for a missing earlier boundary, and every attempt must remain attributable to one logical request. Freeze the current client, gateway policy, model alias, route set, and observable baseline. Evidence record for agent_identity: enforce one_scoped_key_per_owner_or_workload and retain key_id + owner + expiry + allowed_groups. Run one deterministic positive probe and capture the client-visible response, request ID, selected route, terminal state, and usage. Evidence record for wire_contract: enforce responses_or_messages_or_chat_selected_explicitly and retain captured_endpoint + content_type + terminal_event. Run the paired negative, limit, or disconnect case and verify that it fails in the intended layer. Evidence record for model_policy: enforce aliases_resolve_only_to_compatible_routes and retain alias_version + selected_channel + native_probe. Repeat the probe through the actual protocol surface; do not infer native support from a neighboring compatibility endpoint. Evidence record for attempt_budget: enforce one_retry_owner_with_deadline and retain logical_request_id + attempt_sequence + remaining_deadline. Roll out to a bounded cohort with an explicit owner, expiry time, stop threshold, and prepared rollback. Evidence record for tool_boundary: enforce authorize_and_deduplicate_before_side_effect and retain call_id + policy_decision + idempotency_record. Re-read durable configuration and accounting after the test, then remove temporary access or test data. Evidence record for accounting: enforce usage_and_final_charge_reconcile and retain provider_usage + normalized_usage + durable_settlement. Integration failures that a text demo hides Treat each item below as a release blocker. A 200 response, attractive dashboard, or single successful demo does not override these failure conditions. protocol_flattening — Converting every agent to one convenient schema can silently drop tool calls, typed output, streaming events, or explicit zero values. shared_human_key — A shared credential removes attribution and makes revocation, quota, and incident scope unnecessarily broad. nested_retry_multiplication — Agent, SDK, gateway, and upstream retries can multiply attempts and cost during overload. tool_replay — A reissued model tool call can repeat a write unless authorization and idempotency live outside the model. Rollout evidence and stop conditions Monitor success and harm together. The threshold is a policy input, not a universal benchmark; choose it from the workload SLO and record the denominator before the observation window begins. Signal | Decision threshold | Action | protocol_probe_pass_rate | 100%_for_required_cases | block_route_on_any_contract_failure | attempts_per_logical_request | <=_reviewed_attempt_budget | disable_lower_retry_layer | unattributed_usage_ratio | 0 | stop_rollout_and_repair_identity_mapping | duplicate_side_effect_count | 0 | revoke_tool_access_and_reconcile | Modelflare boundary and limitations Modelflare can centralize OpenAI-compatible and native-protocol routing, scoped keys, groups, usage records, and failure handling. A configured channel is not proof that every optional field, model alias, retention promise, region, or fallback is supported. Verify the selected route with its native protocol, preserve explicit zero values, and use the final durable settlement as billing truth. Do not promise that one gateway makes Codex, Claude Code, OpenCode, and every future agent interchangeable. Their configuration formats, authentication precedence, streaming events, tool contracts, and model assumptions can differ. The reusable layer is policy and evidence; the wire adapter remains explicit. Continue through the topic cluster Use the linked parent for the broader decision, the sibling for the next implementation step, and the documentation route for current client configuration. These body links are deliberate because managed CMS articles do not currently carry a separate related-slug field. Parent decision guide (/articles/what-is-ai-api-gateway) Next implementation guide (/articles/openai-compatible-api-conformance-tests) Current setup documentation (/docs) Frequently asked questions Should every coding agent use the same endpoint? Only when the agents genuinely use the same protocol. Keep Responses, Anthropic Messages, and Chat Completions as separate tested contracts. Where should retries live? Choose one owning layer per failure class and enforce one logical attempt budget across the path. How should teams be separated? Use scoped identities, groups, quotas, model policy, expiration, and privacy-safe attribution rather than one organization-wide key. Sources and verification date Sources were checked on 2026-08-07. They establish external contracts and engineering principles; they do not prove an untested route or future provider state. OpenAI: custom model providers for Codex (https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers) Anthropic: Claude Code LLM gateway configuration (https://docs.anthropic.com/en/docs/claude-code/llm-gateway) OpenCode: provider configuration (https://opencode.ai/docs/providers) ### Claude Fable 5.1 Deep Dive: Capabilities, Pricing, and Fable 5 Comparison Canonical URL: https://modelflare.dev/articles/claude-fable-5-1-review-pricing Claude Fable 5.1 is a frontier model for demanding reasoning and long-horizon agentic work. Modelflare now exposes the exact model ID claude-fable-5-1 through three public pricing routes. The important decision is not whether Fable 5.1 is “the best” model in the abstract, but which route and effort level make a complete task cheapest and safest for your workload. Short answer: choose Fable 5.1 for repository-scale coding, multi-step research, document-heavy analysis, and agents that must keep working for hours. Keep Fable 5 or a smaller model for routine, latency-sensitive work. The prices below are the current Modelflare user-facing rates checked on September 2, 2026; the live pricing page remains the source of truth. This article separates Anthropic-documented facts, the live Modelflare catalog, provider-reported benchmarks, and our interpretation. A benchmark score or a configured model row is not a guarantee that every prompt, tool, or protocol behaves identically. Executive answer Fable 5.1 is an incremental model release in name but a meaningful workflow release in practice. It keeps Fable 5's 1M-token context window, 128K maximum output, and always-on adaptive thinking, while improving long-running coding, multistep research, document and spreadsheet work, slide generation, vision, and computer use. Anthropic also lowers cache-read pricing on its API. On Modelflare, the three routes make the cost-versus-continuity choice explicit: claude-premium is the stability-first route. claude-award is the price-first route for work that can be retried. claude-stable is the everyday choice for individual developers and production workloads. The model is not a drop-in upgrade for every custom Messages client. Forced tool use is rejected, Fable 5.1 thinking blocks have stricter history binding, and the model may emit fewer parallel tool calls than Fable 5. Plan a protocol-level migration test before switching a long-running agent. Key facts and current prices The following facts come from Anthropic's Fable 5.1 model overview (https://platform.claude.com/docs/en/models/fable-5-1/overview) and What's new documentation (https://platform.claude.com/docs/en/models/fable-5-1/whats-new-fable-5-1). The Modelflare fields were checked against the live /api/pricing response on September 2, 2026. Property | Fable 5.1 | Model ID | claude-fable-5-1 | Context window | 1M tokens | Maximum output | 128K tokens | Thinking | Adaptive, always on | Default effort | high | Comparative latency | Slower | Knowledge cutoff | June 2026 | Official API input/output | $10 / $50 per 1M tokens | Official cache read | $0.25 per 1M tokens | Modelflare route prices These are the current user-facing Modelflare prices, not Anthropic's direct list prices. Amounts are USD per 1M tokens and apply to input and output separately. Route | Multiplier | Input | Output | Positioning | claude-premium | 0.5x | $5.00 | $25.00 | Stability and request success first | claude-award | 0.25x | $2.50 | $12.50 | Lowest cost; retryable development and tests | claude-stable | 0.315x | $3.15 | $15.75 | Everyday development and production | The model row currently reports a cache-read ratio of 0.025; cache and any other usage dimensions are reconciled from the live pricing page and the completed usage record. Do not copy this table into a cost forecast without checking the current route, cache state, and billing mode. Fable 5.1 versus Fable 5 Anthropic positions Fable 5.1 as a successor rather than a different model family. The practical delta is concentrated in long tasks and in the economics of rereading a stable prompt prefix. Dimension | Fable 5.1 | Fable 5 | What changes for a builder | Context and output | 1M / 128K | 1M / 128K | No capacity migration is needed | Adaptive thinking | Always on | Always on | Keep thinking blocks and use effort controls | Input and output price | $10 / $50 per 1M | $10 / $50 per 1M | Base rates are unchanged | Cache reads | $0.25 / 1M official | $1 / 1M official | Long agent loops can reread prefixes more cheaply | Coding | Stronger multi-file work, refactors, debugging, and verification | Strong long-horizon coding | Re-run repository-level evaluations | Research and artifacts | Better multistep research, documents, spreadsheets, and slides | Strong, but less consistent on long chains | Give the model explicit checkpoints and sources | Tools and history | Forced tool use rejected; stricter thinking-block binding | Fewer of these constraints | Update custom Messages clients before migration | Multilingual behavior | Comparable to Fable 5 | Baseline | Do not assume a new language advantage | The change is not simply “more intelligence for the same invoice.” A long task can become cheaper when cache reads dominate, but a slower model or extra tool turn can still increase wall-clock cost. Measure the complete task, including retries and human review. Capability analysis Long-horizon coding Fable 5.1 is most differentiated when a change crosses files, services, tests, and documentation. Anthropic describes multi-day autonomous sessions, code review, performance work, design implementation, and visual checking of the result. The useful behavior is root-cause repair: the model is more likely to trace a failing test or rare crash to the underlying defect instead of applying a local patch that merely turns the symptom green. That does not make unattended code execution safe by default. Give the agent a bounded workspace, explicit approval points, a test budget, and a rollback path. Ask it to state the files and checks it changed, and record the final diff and test output outside the model transcript. Research and knowledge work The model is designed to take a question through search, evidence gathering, analysis, and a finished artifact. It is a good candidate for a research memo, a spreadsheet with formulas, a slide outline, or a long financial or operational review where later conclusions depend on early findings. The main risk is evidence drift. At low effort the model may search less often and rely more on memory. Require source links, dates, and a short “what would change this conclusion” section. Treat provider examples and customer quotes as qualitative evidence, not as an independent benchmark. Vision and computer use Fable 5.1 can read dense charts, tables nested in PDFs, and document layouts, and it can use vision to compare a generated interface with the original goal. This is valuable for design review, financial statements, architecture diagrams, and browser workflows. Vision is not proof of pixel-perfect understanding. Crop important regions, preserve the original file bytes across turns, and test small labels and coordinates separately. Computer-use actions should remain behind application-level authorization and a human stop point for irreversible operations. Multilingual output Anthropic reports multilingual performance on par with Fable 5. Modelflare exposes localized article and pricing surfaces, but model quality still depends on the prompt, source language, terminology, and output constraints. For a multilingual product, freeze names, units, code identifiers, and legal terms in a glossary, then evaluate each target language on the same task set. Benchmark reading The following table reproduces Anthropic's public comparison. These scores are provider-reported, use the production safeguards described in the announcement, and should be treated as directional evidence rather than a Modelflare guarantee. Benchmark | Fable 5.1 | Fable 5 | Reading | Terminal-Bench-Science 0.1 | 52.6% | 24.7% | Large reported gain on agentic science tasks | Terminal-Bench 4.0 | 55.8% | 42.0% | Better terminal coding under the published harness | GDPval-AA v2 | 1853 | 1723 | Higher knowledge-work score | OSWorld 2.0, partial | 77.9% | 72.9% | Computer-use gain; safeguards affected tasks | OSWorld 2.0, strict | 41.7% | 36.1% | Smaller absolute score, same direction | Humanity's Last Exam, no tools | 60.9% | 57.8% | Moderate reasoning gain | Humanity's Last Exam, with tools | 65.0% | 63.8% | Tools add value, but not a universal win | AutomationBench | 31.4% | 17.1% | Stronger published business-workflow result | CursorBench 3.2.0 | 73.4% | 70.5% | Better coding-agent result | Anthropic notes that safety interventions can reduce scores, including zeros on some OSWorld and AutomationBench tasks. The evaluation harness, effort level, tool permissions, timeout, and retry policy all change the result. A fair internal comparison should replay the same privacy-safe tasks and report successful-task cost, latency, retries, and reviewer correction—not only the highest benchmark number. API and migration notes The live Modelflare catalog currently marks claude-fable-5-1 for its Anthropic- and OpenAI-compatible surfaces. Endpoint support is not feature parity: verify the exact streaming events, tool schema, structured output behavior, and vision input your client needs. export MODELFLARE_API_KEY='YOUR_MODELFLARE_API_KEY' curl -sS https://modelflare.dev/v1/messages \ -H "x-api-key: ${MODELFLARE_API_KEY}" \ -H "anthropic-version: 2023-06-01" \ -H "Content-Type: application/json" \ -d '{ "model": "claude-fable-5-1", "max_tokens": 512, "messages": [{"role": "user", "content": "List three risks in this migration plan."}] }' When migrating a custom Messages integration from Fable 5: remove tool_choice values of any and tool; use automatic tool choice with strict schemas or structured outputs; treat the conversation as append-only when thinking blocks are preserved; do not rebuild the system prompt or tool list between turns unless the API's mid-conversation mechanism is used; use the per-message effort beta only after measuring cache and latency effects; expect more variable parallel tool calling and fewer progress updates in long loops; handle a refusal as a successful HTTP response with stop_reason: "refusal", then apply a bounded fallback policy. The raw chain of thought is not returned. If your UI needs progress, request user-facing updates explicitly and keep them separate from hidden reasoning. Choosing a Modelflare route The route choice should follow failure cost, not a universal quality ranking. For a simple worked example with 100K input tokens and 10K output tokens, excluding cache, retries, and other fees: Route | Calculation | Approximate task charge | claude-premium | 0.1 × $5 + 0.01 × $25 | $0.75 | claude-award | 0.1 × $2.50 + 0.01 × $12.50 | $0.375 | claude-stable | 0.1 × $3.15 + 0.01 × $15.75 | $0.4725 | Use claude-premium when a failed attempt is expensive or continuity matters. Use claude-award for development, batch exploration, and tasks that can be retried safely. Use claude-stable as the balanced default for an individual developer, a long-running project, or a production integration. Recheck the price page before high-volume traffic. Safety, limits, and data handling Fable 5.1 includes safety classifiers for cybersecurity and biology. Some flagged requests may be refused or routed to a less capable fallback; Anthropic documents that a refusal can return HTTP 200 with stop_reason: "refusal". Your billing and retry code must distinguish a refusal from a transport error. Anthropic classifies Fable 5.1 as a Covered Model. Thirty-day retention is the default on the Claude API unless an approved enterprise arrangement applies. Do not send secrets or regulated data merely because the context window is large. Apply least-privilege tools, redact logs, cap execution time, and retain an auditable usage record. The model is slower than smaller Claude options, and a 1M-token ceiling does not mean every position is equally retrievable or cheap. The tokenizer introduced in the recent Claude generation can produce roughly 30% more tokens for the same text than older models. Budget by the usage record, not by character count or context limit alone. FAQ Is Fable 5.1 always better than Fable 5? No. It is a stronger choice for long-horizon tasks, but Fable 5 can be faster or sufficient for routine work. Evaluate the complete task and its retry pattern. Which Modelflare route is cheapest? claude-award has the lowest current multiplier at 0.25x. Its lower price is intended for work that can tolerate retries or variable availability. Which route should I use in production? Start with claude-stable for a balanced default, or choose claude-premium when continuity and request success have higher business value than price. Can I reuse my Fable 5 tool loop unchanged? Not safely. Forced tool use is rejected, thinking-block history is more strictly bound, and parallel tool calls can be emitted differently. Run a real multi-turn test. Does a successful text request prove vision or tool support? No. Test each modality and protocol separately, including streaming, refusal, cancellation, and malformed-input behavior. Sources and update record Anthropic: Introducing Claude Fable 5.1 and Claude Mythos 5.1 (https://www.anthropic.com/claude-fable-and-mythos-5-1) Claude Fable 5.1 model overview (https://platform.claude.com/docs/en/models/fable-5-1/overview) What's new in Claude Fable 5.1 (https://platform.claude.com/docs/en/models/fable-5-1/whats-new-fable-5-1) Migrating to Claude Fable 5.1 (https://platform.claude.com/docs/en/models/fable-5-1/migration-guide) Modelflare live pricing (https://modelflare.dev/pricing) Modelflare reliable routing guide (/articles/reliable-ai-api-routing) Modelflare token cost tracking guide (/articles/ai-api-cost-tracking) Update record: September 2, 2026 — initial draft; Modelflare model availability, route multipliers, and user-facing input/output prices were checked against the live catalog. Anthropic specifications and benchmark values were checked against the linked first-party pages. No independent benchmark claim is made here. ### Migrate from Chat Completions to Responses API Canonical URL: https://modelflare.dev/articles/responses-api-migration-checklist Production guide: Migrate from Chat Completions to Responses API. It includes a deterministic artifact, failure boundaries, rollout checks, and source-qualified limitations. Decision first Migrate from Chat Completions to Responses API is an explicit production contract, not an isolated code change. Define the success signal, terminal failure state, and rollback condition before moving traffic. The artifact below separates evidence from assumption and keeps every attempt attributable. Start with endpoint, prove request_body under a deterministic case, and make rollback a release gate rather than a follow-up task. The reusable artifact Use this table as the review record. A row passes only when its evidence comes from the same request, test window, or configuration snapshot. Checkpoint | Evidence to capture | Pass condition | endpoint | /v1/chat/completions->/v1/responses | The value is exact, preserved, and compared at the wire boundary. | request_body | messages[]->input;response_format->text.format | The value is exact, preserved, and compared at the wire boundary. | tool_result | tool_call_id->call_id;role:_tool->function_call_output | The record joins to one logical request and one concrete attempt. | zero_values | temperature:_0,stream:_false,empty_arrays | The value is exact, preserved, and compared at the wire boundary. | state | previous_response_id_and_repeated_top-level_instructions | The owner, source, checked date, and known limitation are recorded. | rollback | old_endpoint_remains_selectable_during_bounded_rollout | The limit is explicit and the request fails closed when it is exceeded. | Worked example The example is synthetic and deterministic. Replace identifiers and thresholds with reviewed values from your own workload; never paste production secrets or customer prompts into a test fixture. # Chat Completions curl -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"'"$MODEL_ID"'","messages":[{"role":"user","content":"Return OK"}],"stream":false}' # Responses curl -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"'"$MODEL_ID"'","input":"Return OK","stream":false}' Implementation procedure Freeze the current request, response, configuration, and observable baseline before changing behavior. Run one deterministic positive case and retain the complete client-visible result. Run the paired negative or limit case so failure behavior is proved rather than inferred. Join every attempt to one logical request ID, then record timing, terminal state, and usage without sensitive content. Roll out through a bounded cohort with explicit stop conditions; do not widen traffic merely because the happy path passed. Re-read the durable state and public behavior after the change, and execute the prepared rollback if any invariant fails. Failure modes The following failures invalidate the result even when the outer HTTP request appears successful: A zero or false value disappears during parsing or re-serialization, changing the caller's explicit intent. The next request assumes hidden provider state that the selected route does not store or cannot retrieve. The integration reads one convenience field and silently drops typed output items, tool calls, refusals, or partial results. A single text response is treated as proof of protocol compatibility across streaming, tools, schema, usage, and errors. Modelflare boundary Modelflare can centralize OpenAI-compatible routing, keys, groups, usage records, and failure handling, but a configured route is not proof that every upstream supports every optional field. Verify the selected model and channel with the native protocol path. Keep provider prices in the current pricing source, preserve explicit zero values, and treat the final durable settlement—not an estimate—as billing truth. Use the parent guide (/articles/responses-api-vs-chat-completions) for the broader decision boundary and the setup documents (/docs) for current client configuration. Publication checklist The main question is answered before background material. Every request field, status, metric, and formula has one named owner. Examples use synthetic identifiers and contain no credential or customer data. All supported locales preserve the same headings, tables, code, limits, and caveats. Volatile API contracts, model support, and prices are rechecked at T-1 day; an invalidated claim moves the publication date. Before the scheduled time, the public API, localized routes, and sitemap still exclude the article. Sources and verification date Sources were checked on 2026-08-07. They establish the external contract or operating principle; they do not prove support on an untested provider route. OpenAI: Migrate to the Responses API (https://developers.openai.com/api/docs/guides/migrate-to-responses) OpenAI API reference overview (https://developers.openai.com/api/reference/overview) ### MiniMax H3 Deep Dive: Open Weights, Pricing, Quality, and Disruption Canonical URL: https://modelflare.dev/articles/minimax-h3-open-weights-review-pricing MiniMax H3 (https://www.minimax.io/blog/minimax-h3) is one of the few 2026 video releases that changes both the quality frontier and who can modify the stack. It accepts text, images, video, and audio in one context, generates native stereo audio with 4–15 seconds of video, exposes 768P and 2K workflows, and ships downloadable H3-Base checkpoints. The important conclusion is more precise than “H3 is open source and beats closed models.” H3 is a frontier-class open-weight release with unusually aggressive API pricing, but it is not a complete OSI-style Open Source AI system, and current blind preference data does not establish a universal win over Seedance 2.5. Seedance 2.5 remains the more capable contract for 30-second one-pass stories and very large reference boards. H3 changes the market by making top-tier quality, modification rights, and low hosted prices available at the same time. Official specifications, license terms, API prices, and Modelflare availability were rechecked on August 30, 2026. Arena snapshots are dated August 25–26, 2026 and will move as votes accumulate. Executive verdict For openness: H3 publishes two task-specific BF16 base checkpoints, model components, and recipes for SGLang, vLLM, diffusers, and ComfyUI. This is a material release, not a paper-only promise. For price: MiniMax lists 768P output at $0.08 per generated second and 2K at $0.13 per second. Open weights do not make inference free; they move the bill from an API to GPUs, operations, storage, and engineering. For effects: Arena places H3 and Seedance 2.5 in overlapping top groups. H3 is first on the image-to-video snapshot, while Seedance 2.5 is slightly ahead on text-to-video and video editing; their confidence intervals overlap in all three comparisons. For disruption: H3 sharply narrows the historical quality gap between downloadable and closed video models. The faster H3 Max derivative appearing within weeks is early evidence that downstream teams can change the model, not merely wrap its API. For the caveat: the official 2K-quality path still uses hosted H3-Context-IR and H3-Regenerate-2K components, and the community license carries territory, use, disclosure, attribution, and large-commercial-product conditions. What MiniMax H3 actually ships Item | Verified H3 contract | Official API model ID | MiniMax-H3 | Output duration | 4–15 seconds, integer values | Output | 24 FPS video with 32 kHz stereo audio | Resolution | 768P base output; 2K through the official generation or regeneration workflow | First/last-frame mode | Zero, one, or two images | Reference mode | Up to 9 images, 3 video clips, and 3 audio clips; maximum 12 files total | Reference duration | Each video or audio clip 2–15 seconds; each modality capped at 15 seconds total | Dialogue | Officially lists stable support for 11 languages | Open checkpoints | FL2VA for text/first/last-frame tasks; Ref2VA for multimodal reference tasks | Core generator | Dense 33B H3-Omni-Transformer plus Qwen3-VL-32B encoder and visual/audio VAEs | Recommended runtimes | SGLang, vLLM, diffusers, and ComfyUI | The API presents one model name, but the open package is two specialized base checkpoints. That distinction matters operationally: a team serving both first/last-frame generation and arbitrary multimodal reference generation must plan storage and capacity for both paths. An openness audit: four layers, not one label Layer | What is open | What remains limited | Practical reading | Weights and inference components | Full H3-Base checkpoint families, processor, tokenizer, encoder, Omni Transformer, visual VAE, and audio VAE | The repository is large and production inference still needs substantial GPU capacity | Real local execution and fine-tuning are possible | Runtime ecosystem | Official recipes for four major runtimes; downloadable examples and reproducible 768P cases | Initial release uses full attention; sparse-attention inference is promised later | The engineering surface is useful but not yet the most efficient form | Complete product pipeline | Local H3-Base can generate 768P audio-video | H3-Context-IR is hosted, and the official 2K workflow calls hosted Context-IR and regeneration APIs | The highest-quality official workflow is hybrid, not fully local | License and reproducibility | Modification, derivatives, redistribution, and many commercial uses are allowed inside the applicable territory | Custom license, excluded territories, use restrictions, disclosure duties, $20M revenue authorization threshold, and no complete training-data/training-code package | “Open weights under a community license” is more accurate than unconditional “fully open source” | MiniMax calls the release open source. Under the Open Source AI Definition (https://opensource.org/ai/open-source-ai-definition), however, Open Source AI requires freedom to use, study, modify, and share for any purpose, together with the preferred form for modification, including sufficient data information and training code. H3's territorial and field-of-use restrictions and incomplete training package do not meet that bar. This is not a semantic dismissal. Publishing a frontier video base with modification and redistribution rights is a major contribution. The accurate description protects developers from assuming that downloadable weights automatically grant universal deployment rights. How H3 is built and why it matters H3 uses one packed multimodal sequence. Text goes through the H3 encoder, visual inputs through both the encoder and VisualVAE, and audio through AudioVAE. A single-stream Omni Transformer jointly predicts video and audio latents before separate decoders reconstruct the final picture and stereo sound. That design targets a recurring production failure: a pipeline of separate subject-reference, motion-transfer, lip-sync, sound-effects, and upscaling models can lose identity or timing at every handoff. H3 instead tries to learn text, visual identity, motion, editing rhythm, voice, and soundscape inside one context. The system still has three layers: H3-Context-IR interprets complex free-form multimodal material and produces a structured intermediate representation. H3-Base generates 768P audio-video from that representation. H3-Regenerate-2K sends the base result and original context back through a second generation pass for higher detail. This architecture explains both the quality claim and the openness caveat. The released base is substantial, but the official system's reasoning and high-resolution finish are partly delivered as services. H3 vs Seedance 2.5: opposite bets Decision dimension | MiniMax H3 | Seedance 2.5 | Production consequence | Maximum native story length | 15 seconds | 30 seconds, plus extensions | Seedance carries a longer narrative arc before stitching | Reference board | 12 files total: up to 9 images, 3 video, 3 audio | Up to 30 images, 10 video, 10 audio | Seedance fits complex campaigns with more characters, products, voices, and motion references | Output positioning | 768P base and 2K workflow; native stereo | Hosted joint audio-video generation; cited APIs expose 480p/720p tiers | H3 emphasizes resolution and portability; Seedance emphasizes duration and control volume | Editing | Multimodal reference and natural-language video editing | Timestamp-level edits, green screen, camera perspective, and reference editing | Seedance documents more production-specific edit controls; H3 offers a more modifiable foundation | Weights | Downloadable H3-Base checkpoints | Closed | H3 can be self-hosted, fine-tuned, quantized, or post-trained where the license allows | License | MiniMax H3 Community License | Hosted-service terms | H3 reduces technical lock-in but does not remove legal review | Hosted headline price | $0.08/s at 768P; $0.13/s at 2K on MiniMax | About $0.473/s at 720p on fal; $0.36/s at 720p on Modelflare | H3 has a large list-price advantage, but routes and resolution are not identical | The strategic split is clear. Seedance 2.5 tries to replace more of a professional timeline in one hosted model call. H3 tries to make a smaller 15-second frontier core cheap, downloadable, and extensible. A studio may use both: Seedance for a long reference-heavy master shot, H3 for short product variations, controlled edits, or private fine-tuning. What blind human preference data actually says Arena ranks models from anonymous head-to-head user preferences. It is broader than a vendor demo, but it is still a moving preference signal rather than a controlled production benchmark. Arena snapshot | MiniMax H3 | Seedance 2.5 720p | Gemini Omni 1.1 Flash | Careful interpretation | Text-to-video, Aug 25 | 1460±10, rank spread 4–7 | 1476±14, spread 3–7 | 1515±16, spread 1–3 | H3 and Seedance overlap; Gemini leads this snapshot | Image-to-video, Aug 25 | 1494±6, spread 1–3 | 1483±12, spread 1–4 | 1488±11, spread 1–4 | All three occupy the same top statistical group | Video editing, Aug 26 | 1392±19, spread 1–5 | 1410±26, spread 1–3 | 1367±15, spread 3–5 | Seedance is numerically higher, but intervals overlap materially | These numbers justify calling H3 frontier-class. They do not justify “H3 beats Seedance 2.5.” The boards do not disclose your exact prompts, reference rights, brand constraints, failure budget, or downstream acceptance criteria. Seedance also has far fewer votes than older models in these snapshots, so its estimate can still move. The most useful effect conclusion is task-specific: H3 is especially credible for image-conditioned generation and editing, while Seedance's spec advantage remains long-form, reference-heavy production. Gemini Omni shows that a low-priced closed model can still lead text-to-video preference. H3 against the downloadable field Arena's license labels put H3 beside downloadable models, even though their licenses are not equivalent. The quality gap in the same snapshots is unusually large. Downloadable model | License label shown by Arena | Text-to-video | Image-to-video | MiniMax H3 | MiniMax H3 Community License | 1460±10 | 1494±6 | Hunyuan Video 1.5 | Tencent community license | 1169±16 | 1197±16 | Kandinsky 5.0 T2V Pro | MIT | 1172±20 | — | Wan 2.2 A14B | Apache 2.0 | 1132±15 | 1170±10 | LTX-2 19B | LTX community license | 1152±8 | 1155±5 | The point is not that one Elo number settles every shot. It is that downloadable video had usually required a visible quality compromise; H3 arrives in the same preference cluster as the best closed systems. That changes procurement, research, and fallback strategy even before a company self-hosts it. Price: the headline is real, the comparison needs a denominator Route checked on Aug 30 | Resolution | Price per generated second | 15-second output | Important exclusions | MiniMax official H3 API | 768P | $0.08 | $1.20 | Reference-video input and extra images can add cost | MiniMax official H3 API | 2K | $0.13 | $1.95 | Context-IR has separate token pricing | Modelflare Seedance 2.5 | 480p | $0.18 | $2.70 | Requires seedance-stable; maximum 30 seconds | Modelflare Seedance 2.5 | 720p | $0.36 | $5.40 | Different provider route and lower resolution than H3 2K | fal Seedance 2.5, 16:9 estimate | 720p | about $0.473 | about $7.10 | Token formula depends on frame area; reference video changes billing | Google Gemini Omni Flash | 720p | about $0.10 | $1.50 | Closed model; input tokens and higher-resolution behavior are separate | For H3, audio references are free, the first five images are free, each additional image is $0.04, and input video is billed by its duration at the selected output-resolution rate. Regenerating a 768P result to 2K is $0.05 per output second. A reference-heavy job can therefore cost materially more than the headline output rate. For Seedance 2.5, Modelflare's live effective prices remain $0.18/s at 480p and $0.36/s at 720p after the seedance-stable group multiplier. That is cheaper than fal's cited route but still higher than H3's official hosted rate. This is a budgeting comparison, not proof that two differently configured clips deliver equal usable quality. The correct production metric is cost per accepted second: all successful and failed generations, paid references, retries, human editing time, and final usable duration divided by accepted output seconds. A model that costs four times more per generated second can still win if it avoids enough retries or replaces expensive editing. Why self-hosted does not mean free H3's open package changes control, not physics. The core is a dense 33B generator with a full Qwen3-VL-32B encoder and multiple VAEs. The official SGLang examples use four GPUs for each serving variant. The initial release runs full attention. The sparse-attention implementation used in development is scheduled for a later release. Serving both checkpoint families adds storage, warm capacity, model-loading, observability, moderation, and upgrade work. Local H3-Base produces 768P. Reproducing the official 2K result still requires hosted services or a separately validated community pipeline. GPU utilization determines economics. A bursty team can pay more for idle self-hosted capacity than for the API, even if its marginal GPU-second looks cheap. Self-hosting is compelling when raw assets must stay inside a controlled environment, a repeatable style merits LoRA work, traffic is steady enough to use GPUs efficiently, or the team needs to modify inference. API access remains rational for experiments, burst traffic, and the official 2K finish. Where H3 is genuinely disruptive It turns openness into a quality decision, not only an ideology. Teams can now evaluate a downloadable model without automatically accepting a second-tier output class. It creates price pressure at the frontier. An official 2K rate of $0.13/s forces closed systems to justify a premium through longer duration, more controls, reliability, or workflow integration. It lets downstream engineering change the model layer. Quantization, LoRA, post-training, new schedulers, and specialized inference are possible rather than waiting for one vendor roadmap. It enables hybrid architecture. Sensitive or repetitive base generation can stay local, while Context-IR, regeneration, or burst capacity can be purchased selectively. It shortens the derivative cycle. fal introduced H3 Max, a post-trained and inference-optimized derivative, within the same month. fal reports a 5-second 768P generation in under 3 seconds; Artificial Analysis places it in the current top text-to-video group. The speed claim is vendor-reported, but the existence of the derivative is independently visible. That last point is the deepest disruption. A closed API can lower price or add parameters, but only a modifiable base lets another team attack quality, latency, and cost together at the model and systems layers. Where the disruption stops The community license is not an OSI-approved license and excludes the EU, UK, South Korea, and United States unless a separate license is obtained. Commercial products above $20 million in annual revenue need prior written authorization; commercial interfaces must display “MiniMax H3.” Public generated content has disclosure requirements, and downstream services must implement safeguards and enforce use restrictions. H3-Context-IR and the official 2K regeneration implementation are not part of the open release. Training data information and complete training code sufficient to reproduce the system are not published. Open weights do not solve consent, likeness, voice, copyright, brand safety, or output-review obligations. H3 therefore disrupts the technical and economic stack more than it dissolves vendor dependence. MiniMax has opened the valuable base while retaining hosted quality modules, a commercial API, and license control. Which model fits which production constraint Primary constraint | Better starting point | Why | 16–30 second continuous narrative | Seedance 2.5 | Twice H3's single-generation duration and documented extensions | More than 12 mixed references | Seedance 2.5 | Up to 50 files versus H3's 12-file cap | Self-hosting, fine-tuning, or custom inference | H3 | Downloadable base checkpoints and multi-runtime support | Low-cost 2K short-form output | H3 | $0.13/s official 2K rate and strong preference data | Image-to-video preference signal | H3, then verify | First in the current Arena snapshot, but the top confidence intervals overlap | Timestamp and green-screen workflow | Seedance 2.5 | More explicit professional editing contract | Conversational closed-API editing | Gemini Omni 1.1 Flash | Native conversational editing, current text-to-video lead, and about $0.10/s at 720p | Multi-shot and element-specific commercial control | Kling 3 family | Stronger explicit storyboard, element, and motion-control surface | Existing Google or OpenAI production estate | Gemini/Veo or Sora | Identity, safety, billing, and tooling integration may outweigh model portability | “Better starting point” is not a final winner. Run the exact deliverable through a controlled test before changing production routing. A better evaluation protocol Select 12–20 authorized briefs across dialogue, product text, human motion, multi-character continuity, camera motion, image conditioning, reference video, and precise edits. Fix output duration, aspect ratio, resolution class, prompt, reference files, and audio requirement as closely as each contract permits. Record unsupported parameters instead of silently giving one model an easier task. Keep every paid attempt, moderation failure, timeout, and unusable output. Do not score only the best seed. Have reviewers grade identity, instruction following, motion/physics, text, audio synchronization, temporal continuity, edit locality, and final usability while model names are hidden. Measure queue time, generation time, retry count, human correction minutes, input charges, output charges, and storage/egress. Report first-pass usable rate and cost per accepted second with confidence intervals; do not collapse everything into one opaque “quality score.” Re-run a smaller sentinel set after any model, provider, prompt-expander, safety, or pricing change. For self-hosting, add identical energy, reserved-GPU, utilization, deployment-labor, and failure-recovery assumptions. Comparing an API invoice with only the electricity line of a local cluster is not a valid TCO analysis. Modelflare availability boundary At the August 30 production-catalog check, Modelflare did not expose an exact MiniMax-H3 or hailuo-03 model ID. This article does not imply that H3 can currently be called through a Modelflare API key. Modelflare does expose seedance-2.5 through the asynchronous OpenAI-compatible Video API in seedance-stable, at effective prices of $0.18/s for 480p and $0.36/s for 720p. Read the localized Seedance 2.5 API guide (/articles/seedance-2-5-api-guide) and check the live Seedance 2.5 pricing page (/pricing/seedance-2.5) before generating. If H3 is added later, the article must be updated with the exact model ID, endpoint, exposed reference fields, resolutions, groups, effective price, and a real billed task. A configured upstream or third-party alias would not be enough proof. Frequently asked questions Is MiniMax H3 really open source? MiniMax describes it as open source, and the release provides real weights, components, inference recipes, modification, and redistribution rights. Under the stricter OSI Open Source AI definition, the custom territorial/use restrictions and missing complete training package mean “open-weight under the MiniMax H3 Community License” is the safer description. Does H3 beat Seedance 2.5? Not universally. H3 is numerically ahead in the current image-to-video Arena snapshot; Seedance 2.5 is numerically ahead in text-to-video and editing. Their confidence intervals and rank spreads overlap, while Seedance has a stronger 30-second and 50-reference contract. Can H3 generate 2K fully offline? The released H3-Base produces 768P. MiniMax's official full 2K recipe combines local base inference with hosted H3-Context-IR and H3-Regenerate-2K APIs. A fully local community alternative must be evaluated separately and should not be described as the official reproduced path without evidence. Is local H3 cheaper than the API? It depends on utilization, hardware, runtime optimization, both checkpoint families, operations, and quality targets. The API is usually simpler for low or bursty volume; sustained workloads, privacy requirements, or custom fine-tuning can justify self-hosting. Conclusion MiniMax H3 is disruptive because it combines three things that normally arrive separately: frontier human-preference results, an official hosted price as low as $0.08 per generated second, and downloadable weights that downstream teams can modify. It makes an open-weight option part of the primary model shortlist rather than a fallback chosen only for philosophy or privacy. The honest limits are just as important. H3 is a hybrid open-core system with a restrictive community license, substantial hardware demands, and hosted components in the official 2K-quality workflow. Seedance 2.5 remains the more ambitious hosted production contract for 30-second stories and 50-reference boards; Gemini Omni remains a strong low-price closed competitor. Choose by bottleneck. Use H3 when portability, modification, short-form economics, or image-conditioned quality matters most. Use Seedance when long continuity and a very large reference surface are worth the premium. Judge both by accepted output, not launch reels. Official sources and update log Primary and measurement sources: MiniMax: H3 launch (https://www.minimax.io/blog/minimax-h3) MiniMax: H3 open-weight release and architecture (https://www.minimax.io/news/minimax-h3-open-source) MiniMax H3 official model repository (https://huggingface.co/MiniMaxAI/MiniMax-H3) MiniMax H3 Community License (https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE) MiniMax API: video generation contract (https://platform.minimax.io/docs/guides/video-generation) MiniMax API pricing (https://platform.minimax.io/subscribe/token-plan?tab=api-enterprise) ByteDance Seed: Seedance 2.5 launch (https://seed.bytedance.com/en/blog/one-take-creation-flexible-referencing-introducing-seedance-2-5) Google: Gemini Omni Flash model contract (https://ai.google.dev/gemini-api/docs/models/gemini-omni-flash) Google: Gemini API pricing (https://ai.google.dev/gemini-api/docs/pricing) Arena text-to-video leaderboard (https://arena.ai/leaderboard/text-to-video) Arena image-to-video leaderboard (https://arena.ai/leaderboard/image-to-video) Arena video-edit leaderboard (https://arena.ai/leaderboard/video-edit) Open Source Initiative: Open Source AI Definition 1.0 (https://opensource.org/ai/open-source-ai-definition) fal: H3 Max derivative announcement (https://blog.fal.ai/introducing-h3-max-by-fal/) Update log: August 30, 2026: Initial publication. Rechecked official H3 architecture, open package, license, API prices, Seedance 2.5 contract, live Modelflare prices and availability, and dated Arena snapshots. The cover is a synthetic editorial illustration, not H3 output, a benchmark result, or a real product interface. ### GLM-5.3, Kimi K3, and Qwen3.8-Max: Capabilities, Pricing, and API Setup Canonical URL: https://modelflare.dev/articles/glm-5-3-kimi-k3-qwen3-8-max-api-guide Modelflare now exposes glm-5.3, kimi-k3, and qwen3.8-max through one compatible gateway. This launch guide explains what each model is good at, the current USD price snapshot, the three supported request contracts, and a safe path from API key creation to a production probe. The model descriptions below separate first-party documentation from Modelflare catalog facts. A provider claim is not automatically a gateway feature: always check the live model row and test the exact endpoint your client uses. Volatile model availability, group ratios, and prices in this page were checked on August 29, 2026. The Modelflare pricing page (https://origin.modelflare.dev/pricing) remains the authority after that date. Three models at a glance Model ID | Good starting point | First-party capability notes | Context and reasoning | glm-5.3 | Complex coding and long-horizon agent work | Z.ai positions GLM-5.3 for difficult coding and longer agent tasks; its API keeps thinking enabled | Thinking is always on; documented effort levels are low, high, and max, with max as the default | kimi-k3 | Long-context coding and knowledge work | Kimi documents native visual understanding and end-to-end, long-horizon work | Up to 1M context; thinking is always on; effort levels are low, high, and max, default max | qwen3.8-max | Professional workflows and long-running agents | Qwen documents a 2.4T-total / 95B-active MoE model with coding, visual input, and tool-oriented workflows | 1M context; text, image, and video input; text output; some tools depend on region and endpoint | Read the GLM-5.3 announcement (https://z.ai/blog/glm-5.3), Kimi model selection guide (https://www.kimi.ai/help/kimi-api/api-model-selection), and Qwen3.8-Max reference (https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max) for the vendors' current wording. Their benchmark numbers are provider-reported and are not a Modelflare guarantee. Choose by workload Choose glm-5.3 when the task needs sustained planning, difficult code changes, or a longer agent loop and you can budget for always-on reasoning. Choose kimi-k3 when a very large context, visual understanding, or an end-to-end knowledge-work loop is central to the request. Choose qwen3.8-max when a 1M-token window, multimodal input, structured tool workflows, or professional automation matters. These are starting points, not a universal ranking. Replay a privacy-safe sample of your own tasks and measure successful-task cost, retries, latency, and review time. A large context limit is a ceiling, not a promise that every position will be equally useful or fast. Modelflare price snapshot The following are USD per 1 million tokens for the current public Modelflare groups. The group multiplier is 0.8x, matching the launch announcement's “official price × 0.8” wording. Model | Group | Input | Output | Cached input | glm-5.3 | glm-stable | $1.12 | $3.52 | about $0.21 | kimi-k3 | kimi-stable | $2.40 | $12.00 | $0.24 | qwen3.8-max | qwen-stable | $1.60 | $4.80 | $0.20 | Values are calculated from the live catalog's input price, completion ratio, cache ratio, and group multiplier, then rounded for readability. If the pricing page shows a separate cache-write field, use that field for cache creation rather than inferring it from the read price. The three groups currently show rpm: 0, which means no fixed platform-side group RPM cap is configured in the catalog. It does not remove upstream, account, network, or abuse controls. Prices and availability can change; use the live pricing page (https://origin.modelflare.dev/pricing) and the usage record for reconciliation. One gateway, three API contracts As of this snapshot, the Modelflare catalog marks all three model IDs for these public formats: Contract | Endpoint | Authentication | Best fit | OpenAI Chat Completions | POST /v1/chat/completions | Authorization: Bearer | Existing chat clients and broad SDK compatibility | OpenAI Responses | POST /v1/responses | Authorization: Bearer | Clients that consume Responses items and events | Anthropic Messages-compatible | POST /v1/messages | x-api-key or Bearer plus anthropic-version | Anthropic-shaped clients and message streams | The shared OpenAI-compatible base URL is https://modelflare.dev/v1 (https://docs.modelflare.dev/guides/endpoints/). Anthropic SDKs normally use https://modelflare.dev (https://modelflare.dev) as the base and append /v1/messages. Endpoint availability is not feature parity: verify streaming events, tools, structured output, multimodal input, and reasoning controls separately. Connect through Modelflare In API Keys, create or update a key and grant it the group that contains the desired model: glm-stable, kimi-stable, or qwen-stable. Keep the key in an environment variable. Do not paste it into source control, browser storage, or a support ticket. Confirm the authenticated /v1/models response, then send a small non-streaming request using the exact model ID. Test streaming as a separate contract before moving an agent or user-facing workload. Chat Completions export MODELFLARE_API_KEY='YOUR_MODELFLARE_API_KEY' curl -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "kimi-k3", "messages": [{"role": "user", "content": "Summarize the migration risks in three bullets."}], "stream": false }' Responses API curl -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3.8-max", "input": "Return a short checklist for a safe database migration.", "stream": false }' Anthropic Messages curl -sS https://modelflare.dev/v1/messages \ -H "x-api-key: $MODELFLARE_API_KEY" \ -H "anthropic-version: 2023-06-01" \ -H "Content-Type: application/json" \ -d '{ "model": "glm-5.3", "max_tokens": 256, "messages": [{"role": "user", "content": "List the first three rollout checks."}] }' The examples intentionally use plain text and stream: false. Add streaming only after the client handles that protocol's event format. Preserve explicit zero and false values when they carry meaning, and do not copy a provider-specific field from one model into another without checking its contract. Verify before production List models with the same key and confirm the exact ID and group are visible. Run one non-streaming and one streaming probe for the endpoint the application will actually call. If the workload uses tools, structured output, images, video, or reasoning controls, test each feature on the selected model; a successful text response is not proof. Record request status, selected group, input/output/cache tokens, latency, retries, and final charge from the usage record. Keep application retries bounded and distinguish a temporary capacity response from a terminal model or policy error. Recheck the live pricing page before a high-volume launch; the page and usage record outrank a copied table. FAQ Are these prices permanent? No. This is a dated catalog snapshot. The live pricing page is the source of truth. Does rpm: 0 mean unlimited traffic? No. It means the group has no configured platform-side RPM ceiling. Provider, account, network, and safety limits can still apply. Can one payload be sent to all three protocols? No. Keep the model ID but adapt the envelope, authentication, response parser, and streaming handler to the selected contract. Which models have a documented 1M context window? Kimi K3 and Qwen3.8-Max do in the linked first-party references. This page makes no 1M claim for GLM-5.3. Sources and freshness Z.ai: GLM-5.3 (https://z.ai/blog/glm-5.3) Kimi API model selection (https://www.kimi.ai/help/kimi-api/api-model-selection) Alibaba Cloud Model Studio: Qwen3.8-Max (https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max) Modelflare live pricing (https://origin.modelflare.dev/pricing) Qwen3.8-Max deep guide on Modelflare (/articles/qwen3-8-max-guide) OpenAI-compatible API guide (/articles/openai-compatible-api-guide) Responses API vs Chat Completions (/articles/responses-api-vs-chat-completions) Update record: August 29, 2026 — initial launch guide; first-party capability pages, Modelflare groups, endpoint flags, and prices were checked on this date. ### How to Evaluate an AI API Gateway: A Production Checklist Canonical URL: https://modelflare.dev/articles/evaluate-ai-api-gateway Evaluate an AI API gateway by running your real protocol contract through it, forcing the failure modes you care about, and checking the request-level evidence it produces. A feature list or successful “hello world” request cannot prove streaming correctness, tool compatibility, fallback safety, cost accuracy, security boundaries, or a usable exit path. The most reliable process uses two kinds of criteria: non-negotiable gates that cause a candidate to fail, and scored operational qualities that distinguish candidates after every gate passes. Define the workload contract first Do not begin with a vendor comparison table. Begin with one representative workload and write down its invariant contract: exact endpoint: Responses, Chat Completions, embeddings, images, or another API; exact model identifiers and whether aliases are permitted; streaming and non-streaming modes in use; Structured Outputs, Function Calling, hosted tools, reasoning, or other required fields; typical and high-percentile input and output lengths; concurrency, request rate, region, and user-facing deadline; required usage, cache, cost, and request-correlation fields; allowed fallback routes and whether model substitution is forbidden; data-retention, access, residency, and deletion requirements; application operations that can create side effects. One gateway can pass for a text-only internal assistant and fail for a streaming coding agent. “OpenAI-compatible” is not a sufficient workload definition because compatibility can differ by endpoint, event type, tool, schema keyword, and provider route. If the team is still deciding whether it needs a proxy or a model-aware control plane, start with LLM Proxy vs AI Gateway (/articles/llm-proxy-vs-ai-gateway). This checklist assumes the gateway category is already justified and tests whether a specific implementation can own the required responsibilities. Apply fast disqualifiers before a long trial The first review should remove candidates that cannot satisfy a mandatory boundary. Ask for demonstrable behavior, not a roadmap statement. Gate | Immediate failure condition | Evidence to request | Protocol | A required request field, output item, or stream event is dropped or rewritten incorrectly | Captured redacted request/response pair and parser result | Model identity | The gateway silently changes the requested model | Attempt record with requested and actual model | Streaming | It buffers the full response, loses cancellation, or corrupts Tool Argument fragments | Timestamped event sequence and cancellation trace | Authentication | Browser or workload clients receive provider credentials | Credential-flow diagram and a live key-rotation test | Tenant isolation | One project can use or inspect another project’s keys, usage, or logs | Authorization tests using real isolated accounts | Cost evidence | A final charge cannot be connected to model, route, price basis, and usage | One reconciled request ledger | Failure safety | A partial stream is transparently replayed or a cancellation starts another attempt | Forced partial-stream and cancel traces | Export and exit | Configuration and request contract cannot be recovered without rewriting the application | Export sample and provider-native rollback drill | A candidate that fails a mandatory gate should not receive a high aggregate score to compensate. Security isolation cannot be offset by a good dashboard, and an incorrect Tool Contract cannot be offset by a lower price. Build a small protocol conformance corpus Use deterministic, non-sensitive inputs and keep the expected wire behavior under version control. The corpus should call the real gateway endpoint; it should not mock the provider or reimplement the gateway’s conversion logic as the test oracle. Case | Request | Required observation | Basic non-streaming text | Pinned model and fixed prompt | Correct status, model identity, text location, usage, request ID | Streaming text | Same prompt with streaming enabled | Ordered events, first effective output, final event, cancellation behavior | Structured output | Strict schema with required fields and additionalProperties: false | Valid output or an explicit unsupported error; no silent downgrade | Function Calling | One read-only function and one returned result | Function name, JSON arguments, Call ID correlation, final answer | No-tool path | Same tools declared but no tool required | Normal text without a fabricated Tool Call | Invalid field | Deliberately unsupported or malformed request | Stable client error; no provider fallback that hides the defect | Long input boundary | Input just below and above the approved limit | Documented acceptance or explicit rejection, not truncation | Usage detail | Request that exercises cached or reasoning usage when supported | Fields survive the route and reconcile with the billing record | Cancellation | Client cancels after connection and after first output | Upstream work stops and no new fallback attempt starts | Partial stream | Connection fails after effective output | One explicit partial failure; no invisible second answer | Run every required case against every route that may serve the workload. A primary route passing does not qualify its fallback. The Structured Outputs guide (/articles/structured-outputs-openai-compatible-api) and Function Calling comparison (/articles/function-calling-responses-vs-chat-completions) provide field-level test cases for those two capabilities. Record the gateway version, route configuration version, model ID, provider, region, timestamp, and sanitized result hash. Volatile model and provider behavior should be rechecked before rollout and after a material route change. Test routing and failure behavior, not only success Reliability claims matter only when the failure policy is visible. Force each condition before production: primary route unavailable before headers; provider rate limit with and without Retry-After; upstream authentication or account failure; slow headers and slow first effective output; malformed provider response; client cancellation while the upstream is pending; connection loss after visible output begins; all eligible routes exhausted. For each case, capture the attempt order, selected route, status, timing, whether output had started, terminal reason, and any usage or cost. Confirm that the gateway preserves the requested model and protocol unless a separate model-substitution policy is explicitly enabled. Measure attempt amplification across the SDK, application, gateway, and provider. One layer should own immediate same-contract fallback, while the application owns whether the entire user action can be repeated. The AI API Fallback Strategy (/articles/ai-api-fallback-matrix) provides a phase-aware failure matrix and retry-budget model. Latency needs the same precision. Compare time to upstream headers, first SSE event, first effective output, first visible text, completion, and visible output speed at realistic concurrency. Do not accept one unexplained “latency” average. See AI API Latency Metrics (/articles/ai-api-latency-metrics) for definitions and comparison controls. Reconcile usage and cost from one request Choose several completed requests and follow each one through the full accounting chain: application request ID → gateway attempt sequence → selected model and route → provider or normalized usage → applicable price basis → final recorded charge The evaluation should answer: Are input, output, cached, reasoning, and tool-related units represented when applicable? Which values come from the provider and which are estimated? When is the model price selected, and is it frozen for the request? Does a group, service tier, discount, or surcharge change the user charge? Which failed attempts can create provider cost, and how are they recorded? Can a final successful fallback hide earlier billable attempts? Are currency conversion and rounding rules explicit? Can finance reproduce a daily total from immutable request records? Test a normal completion, a same-contract fallback, a cancelled request, and an upstream error. A dashboard total is not enough; the gateway must produce a defensible per-request record. AI API Cost Tracking (/articles/ai-api-cost-tracking) separates provider usage, platform pricing, customer charge, and supplier cost. Do not compare vendor savings unless the same model, workload, cache behavior, output length, failure rate, and provider price basis are held constant. A lower apparent cost can be caused by missing usage or silent model substitution. Verify the security and data boundary Draw the actual data flow from the client to the gateway and each provider. For every hop, identify who can read credentials, request content, response content, metadata, and administrative configuration. At minimum, verify: provider credentials are stored server-side, encrypted at rest, and never returned to ordinary clients; application keys can be scoped by project or workload and revoked independently; authorization is enforced server-side on every management and log endpoint; logs avoid full API keys and have explicit prompt/response retention controls; support access is attributable and bounded; configuration changes have actor, time, before/after state, and rollback evidence; exported traces redact secrets and personal or proprietary content; deletion and retention behavior can be demonstrated, not only described; region and subprocessor claims match the route actually used; abuse limits run before expensive upstream work when possible. Ask what happens during key rotation, an operator departure, a compromised application key, and a provider-key leak. Run a rotation and revocation exercise with real isolated test credentials. Do not copy a production secret into an evaluation environment. The gateway cannot make unsafe application tools safe. Tool authorization, transactionality, approval, and idempotency remain application responsibilities even when model access is centralized. Use AI API Key Security and Cost Controls (/articles/api-key-security-cost-controls) to separate credentials and workload limits. Evaluate the operational control plane The data plane may work while the control plane creates operational risk. Review how operators make and recover from changes: Area | Questions to answer | Versioning | Is each route, price, policy, and key change versioned or attributable? | Validation | Can an invalid route or incompatible model be rejected before activation? | Rollout | Can a change target a small workload or percentage first? | Rollback | Can operators restore the last known-good configuration quickly? | Availability | What happens to existing and new requests if the control plane is unavailable? | Health | Is channel health based on current evidence, and can automatic disablement be inspected? | Incidents | Can one request be reconstructed without searching several unrelated systems? | Limits | Are rate and quota decisions atomic enough to remain correct under concurrency? | Change ownership | Are emergency edits separated from normal product configuration? | Run one configuration rollback and one unhealthy-route removal. Measure the operator steps and verify the resulting data-plane behavior. A screenshot of a rollback button is not the same as a completed rollback drill. Test the exit path before signing Gateway adoption can create dependencies on model aliases, custom headers, proprietary route names, log APIs, normalized error shapes, or hosted Prompt and Tool configuration. List each dependency and decide whether it is an intentional benefit or accidental lock-in. A practical exit drill should: export route, key-policy, price, and audit configuration in a documented format; switch one workload to a provider-native test endpoint; replace gateway-only headers or aliases with explicit application configuration; preserve request correlation and usage reconciliation during the switch; document functionality that cannot move without redesign; estimate exit engineering from the observed work, not a sales claim. An exit path does not require the gateway to be interchangeable with every provider. It requires the team to know what it owns, what the gateway owns, and how to recover the underlying protocol contract. Score only after every mandatory gate passes Use a worksheet with pass/fail for hard boundaries and a small evidence score for operational quality: Score | Meaning | 0 | Unsupported or contradicted by the test | 1 | Claimed or manually demonstrated once, with weak evidence | 2 | Repeatably demonstrated with request-level evidence | 3 | Repeatably demonstrated, monitored, and recoverable through a tested control | Suggested scored areas are protocol coverage, route reliability, attempt evidence, latency diagnostics, usage accuracy, cost reconciliation, key isolation, auditability, configuration rollback, supportability, and exit effort. Weight them according to the workload, but keep the raw evidence beside every score. Avoid a false-precision total such as 87.4/100 when several rows are subjective. Record the decision as: mandatory gates and outcomes; score by area with evidence links; accepted gaps and owner; remediation deadline; cost and contract assumptions; selected candidate and rejected alternatives; review date after the first production month. Compare build versus buy by ownership The right question is not whether an internal gateway has a license fee. It is which responsibilities the team can own continuously. Responsibility | If built internally | If purchased or managed | Protocol updates | Track provider schemas and regressions | Verify vendor updates and route compatibility | Routing and retry | Design state machine and failure evidence | Configure policy and audit actual attempts | Usage and billing | Normalize usage and maintain pricing logic | Reconcile vendor records with internal finance truth | Security | Store secrets, enforce tenancy, audit access | Validate vendor boundary and configure least privilege | Reliability | Operate data plane, control plane, and on-call | Monitor vendor plus your integration and retain an exit path | Product support | Diagnose every application/provider interaction | Triage gateway versus provider versus application failures | Do not insert generic salary or “engineering time saved” figures. Estimate from your own on-call load, protocol-change history, incident frequency, finance requirements, and compliance work. A managed product still requires an accountable internal owner. Apply the checklist to Modelflare accurately For Modelflare, the current evaluation boundary should be explicit. It provides workload API Keys, requested-model routing through eligible groups and channels, ordered group fallback for regular keys, strategy-based group selection for Smart API Keys, pre-upstream group RPM admission, and request-level usage, cost, status, and timing records. GPT, Codex, and OpenAI traffic is the fully adapted compatibility target. Other OpenAI-compatible model families should be evaluated as raw Chat Completions pass-through unless a capability has been separately verified. A shared base URL is not proof that Responses, hosted tools, Structured Outputs, or Function Calling behave identically on every route. Modelflare’s fallback should search for an eligible path for the requested model rather than silently selecting another model. Channel failover stops after downstream output starts. Those claims can be evaluated with the protocol corpus and failure drills above instead of accepted as marketing statements. Use Models & Pricing (/pricing) to identify the current model and group surface and Modelflare Docs (/docs) to configure an isolated test key. Keep the evaluation requests non-sensitive, pin the exact model, and retain the Request IDs needed to inspect each attempt. The final decision should be reproducible: another engineer should be able to run the same corpus, inspect the same categories of evidence, and understand why the candidate passed. That process is slower than reading a comparison page, but it is much faster than discovering an incompatible Tool Contract, untraceable bill, or unsafe fallback after the gateway owns production traffic. ### Why AI Agent Cache Hit Rates Collapse: GPT, Claude, and Auditable Gateways Canonical URL: https://modelflare.dev/articles/ai-agent-cache-hit-rate AI agent caching is not a checkbox. It is a protocol, routing, and accounting contract. This article explains why an opaque third-party agent can show a very low cache hit rate even when the upstream model supports prompt caching, how GPT and Claude measure reuse differently, and what an auditable gateway should guarantee. The decision in one sentence Do not put a production, cache-sensitive agent behind a third-party agent or gateway that cannot expose native cache usage, preserve a stable prefix, keep model and route affinity, and reconcile the result against a durable ledger. “Supports caching” is a feature claim; a measured, eligible hit rate is an operating fact. This is not a claim that every third-party agent fails. Public documentation cannot establish a vendor-wide percentage without a fixed corpus, a fixed model, a known observation window, and request-level usage evidence. The reliable conclusion is narrower: an opaque translation layer creates several independent ways to turn a valid provider cache into an almost-always-cold path. Treat that lack of evidence as a reason not to use the path for cache-dependent production work. Define the denominator before discussing a percentage The provider usually reports three different buckets. Let R be tokens read from a cache, W be tokens written to a cache, and U be input tokens processed without reuse. For cache-eligible prefixes, the comparable hit rate is: eligible_hit_rate = R / (R + W) For all input sent to the model, the reuse share is: input_reuse_share = R / (R + W + U) Those numbers answer different questions. A dashboard that divides by all input can look low when the agent sends many short, ineligible requests. A dashboard that divides only eligible traffic can look high while a large dynamic suffix still dominates spend. Report both, plus the eligibility rule and the time window. The provider fields are the evidence, not a green badge in an agent UI: Signal | OpenAI field | Claude field | What it proves | Reused prefix | input_tokens_details.cached_tokens | cache_read_input_tokens | Tokens actually served from a matching entry | New cache entry | input_tokens_details.cache_write_tokens | cache_creation_input_tokens | Tokens charged to create or extend an entry | Unreused input | input_tokens - input_tokens_details.cached_tokens - input_tokens_details.cache_write_tokens (derive carefully) | input_tokens after the breakpoint | Input outside the cached prefix; semantics differ | Correlation | request ID plus model and route | request ID plus model and route | Which provider attempt produced the usage | Worked example, using synthetic numbers rather than Modelflare production telemetry: R=720,000, W=80,000, and U=200,000 give an eligible hit rate of 720,000 / 800,000 = 90%, but an all-input reuse share of 720,000 / 1,000,000 = 72%. Publishing only one of those values hides the denominator. input_tokens_details.cache_write_tokens=0 is not proof of “no cache.” A request can be a cache read and therefore create no new entry. Conversely, a missing usage field in a third-party response is not a zero; it is an observability failure that must be recorded as not auditable. Why an agent gateway loses hits even when the model supports caching The failure is usually in the path between the application and the provider. These are the high-frequency breakpoints: Dynamic bytes arrive too early. Timestamps, request IDs, user names, experiment flags, or a changing “current date” in the system prompt alter the prefix before the reusable instructions. Tools are reserialized. Adding, removing, reordering, or nondeterministically serializing a tool schema changes the exact prefix. A harmless-looking JSON key order can become a cache miss. Fallback changes the cache identity. A load balancer that moves between model aliases, regions, organizations, or provider credentials does not share one universal cache entry. The adapter drops native controls. Removing cache_control, prompt_cache_key, retention options, or usage details turns a provider feature into an invisible best effort. The prompt is below the provider threshold. Short agent turns can be valid requests but still be ineligible for caching. The TTL window is crossed. A five-minute Claude entry or a model-specific OpenAI retention window can expire during a human-in-the-loop pause. Parallel warm-up races. Concurrent first requests may all arrive before the first response makes a cache entry available. History is rewritten. Summarization, compaction, truncation, or a different serialization of prior messages changes the prefix instead of appending to it. None of these failures requires a dishonest vendor. They are predictable consequences of a gateway that treats an agent request as free-form text instead of preserving a provider’s cache contract. The practical warning is stronger than a marketing comparison: if the agent cannot show which breakpoint failed, you cannot price or debug cache-dependent traffic. GPT and Claude use related ideas, not interchangeable semantics Both providers require an exact reusable prefix, but the control surface and usage accounting are different. The table below summarizes the official guides checked on 2026-08-25; model names, minimums, and retention policies can change, so recheck the linked pages before changing a production policy. Dimension | OpenAI prompt caching | Claude prompt caching | Reusable unit | The full rendered context prefix, including instructions, tools, history, and multimodal parts | The ordered prefix through a cache_control breakpoint: tools, system, then messages | Minimum length | The current guide lists 1,024 visible tokens for GPT-5.6+ and generally 2,048 for older models | Model-specific minimums currently span roughly 512–4,096 tokens; shorter prompts run without caching | Controls | Implicit caching; explicit breakpoints and a stable prompt_cache_key on supported models | Top-level automatic caching or explicit block breakpoints; up to four breakpoints and a 20-block lookback | Retention | GPT-5.6+ supports a 30-minute TTL; earlier models expose retention modes with provider-defined typical windows | Five-minute default TTL, refreshed on use; optional one-hour TTL at a higher write price | Price shape | On the current GPT-5.6+ guide, writes are 1.25× base input and reads are 0.1× | Five-minute writes are 1.25× and one-hour writes are 2×; reads are 0.1× | Usage evidence | input_tokens_details.cached_tokens and, when applicable, input_tokens_details.cache_write_tokens | cache_read_input_tokens, cache_creation_input_tokens, and post-breakpoint input_tokens | Frequent invalidator | Model/tool/settings changes, machine overflow, or a prefix change before the breakpoint | Model/system/tools/message changes, a missing previous message, or a concurrent warm-up miss | OpenAI notes that cache entries live on individual machines and that a prompt_cache_key helps group and route requests but does not pin a machine or guarantee a hit. Claude documents exact matching, workspace-level isolation, and explicit diagnostics such as tools_changed and messages_changed. A gateway that hides those distinctions cannot honestly convert provider documentation into a single “cache rate” promise. The price multipliers explain why this matters. If one write is followed by one read, the cached input portion costs about 1.25 + 0.10 = 1.35× the uncached input unit on the cited OpenAI model, rather than two uncached units. Ten fully reused requests cost about 1.25 + 9×0.10 = 2.15× instead of 10×. These are official illustrative calculations, not a Modelflare invoice or a promise for every model. The adapter tax is measurable Use a failure matrix instead of a screenshot. Keep the application prompt, model, credentials, and request rate fixed; change one variable at a time and retain the raw usage object. Controlled change | Expected provider signal | What an opaque agent may hide | Release interpretation | Append only the user turn | R rises after the first W | Rewritten history or a new route | Prefix preservation works | Add a timestamp to the system prompt | R falls to zero or a new W appears | Which bytes changed | Dynamic prefix is a cache breaker | Reorder one tool property | tools_changed or a new write | Serializer behavior | Tool schema must be deterministic | Split traffic across aliases or machines | Lower and more variable R | Selected route and cache key | Affinity policy is missing | Sleep past the documented TTL | New write after expiry | Expiry time and retention mode | Measure human-pause workload separately | Fire two identical first requests in parallel | One or both may write | Warm-up race and attempt order | Do not infer capacity from a cold burst | The test must retain the request ID, stream mode, first event time, exact model, selected route, cache fields, and UTC timestamp. Redact prompts, keys, and customer content. If a gateway returns only a normalized “input tokens” total, label the run not auditable; do not turn missing dimensions into a fabricated zero or a favorable estimate. A reproducible audit record The following is a small, synthetic record format. It is deliberately provider-neutral at the envelope and preserves native fields inside usage. It is not a benchmark result. {"request_id":"demo-001","timestamp":"2026-08-25T02:00:00Z","model":"MODEL_ID","route":"route-a","stream":true,"usage":{"input_tokens":1000000,"input_tokens_details":{"cached_tokens":0,"cache_write_tokens":1000000}}} {"request_id":"demo-002","timestamp":"2026-08-25T02:00:03Z","model":"MODEL_ID","route":"route-a","stream":true,"usage":{"input_tokens":1000000,"input_tokens_details":{"cached_tokens":720000,"cache_write_tokens":80000}}} Run at least five phases: a serial warm-up, an append-only conversation, a dynamic-system mutation, a tool-order mutation, and a TTL-gap replay. For Claude, map the native read/write fields into the same report without replacing them; for OpenAI, retain both cached and write fields when returned. Compare R/(R+W) and R/(R+W+U) per phase, route, model, and UTC day. A single blended number is not enough to decide whether an agent is safe for production. What Modelflare guarantees—and what it does not Modelflare’s public status page currently describes an OpenAI Cache Hit Rate Guarantee. For eligible requests that satisfy OpenAI’s caching requirements and form a valid cache, the daily rate is calculated by UTC calendar day. If the measured rate falls below the applicable tier, the shortfall is compensated and shown in Dashboard → Token Usage Analysis. The public tiers currently shown are 65%, 75%, and 85% for the published group levels. The boundary is intentional: The guarantee applies to eligible, valid OpenAI cache traffic, not to a short or constantly changing prompt that never qualifies. It does not turn a changing model, tool schema, route, or TTL into a reusable prefix. It is calculated per UTC calendar day and credited through the documented account surface; it is not an assertion that every individual request will hit. The native usage evidence and eligibility calculation remain visible requirements. A compensation rule is useful precisely because the denominator is explicit. That is the difference between an auditable service promise and a third-party agent slogan: the former names the eligible population, observation window, threshold, and credit destination. Price advantage and operating legitimacy are separate proofs The current public pricing page also displays a 0.015 first-top-up offer ratio for the applicable campaign/group, with the page’s stated minimum single top-up and eligibility terms. This is a price incentive; it must not be conflated with the cache guarantee. Check the live pricing page for the exact model scope, campaign status, and settlement rules before making a purchase decision. Modelflare’s public trust and legal materials identify Havenbyte LLC as the operator and describe the operation as United States based. They list Stripe as the payment processor for payment handling, billing, invoices, refunds, and fraud controls. These are useful custody and accountability signals, not a substitute for reading the terms or testing the API. The article makes no claim about a particular state registration, certification, or provider affiliation beyond those public materials. Recommendation for production agents Choose a third-party agent only when it can pass this checklist: It forwards the provider’s native cache controls and returns native read/write usage. It keeps the stable instruction and tool prefix byte-identical, then appends dynamic state. It exposes the selected model, route, organization/region boundary, TTL mode, and request ID. It records serial and parallel warm-up behavior instead of averaging them away. It defines eligible traffic, denominator, UTC window, threshold, and compensation in writing. It lets you export request-level evidence without exposing secrets or customer content. If any of those answers is “no,” do not use the path for a workload whose economics depend on cache reuse. Use a direct provider path for the experiment, or choose an auditable gateway such as Modelflare and verify the exact route with the same corpus. For adjacent controls, see reliable AI API routing (/articles/reliable-ai-api-routing), AI API cost tracking (/articles/ai-api-cost-tracking), pricing (/pricing), and trust (/trust). Sources and verification date The following primary sources were checked on 2026-08-25. They describe provider contracts and Modelflare’s current public policy; they do not establish a universal hit rate for an unnamed third-party agent. OpenAI Prompt Caching guide (https://developers.openai.com/api/docs/guides/prompt-caching) OpenAI API pricing (https://developers.openai.com/api/docs/pricing) Anthropic Prompt Caching guide (https://platform.claude.com/docs/en/build-with-claude/prompt-caching) Anthropic cache diagnostics (https://platform.claude.com/docs/en/build-with-claude/cache-diagnostics) Modelflare live status and cache guarantee (https://modelflare.dev/api/status) Modelflare pricing and current offer terms (https://modelflare.dev/pricing) Modelflare trust and operating entity (https://modelflare.dev/trust) FAQ: Is a low third-party hit rate always fraud? No. It may be a valid result of short prompts, a changing prefix, route fan-out, TTL expiry, or a missing usage field. The correct finding is “not reproducible” or “not auditable” until the controlled test shows the cause. FAQ: Can I compare Claude and GPT with one number? Only after normalizing the denominator and preserving each provider’s native fields. Compare the same corpus and workload phase, not a blended dashboard total. FAQ: Does compensation remove the need to design a stable prefix? No. Compensation limits the financial downside for eligible OpenAI cache traffic; it cannot make an ineligible request reusable or repair an agent that rewrites the prefix. ### DeepSeek V4 Flash Vision Exp: Pricing, Limits, and Model Comparison Canonical URL: https://modelflare.dev/articles/deepseek-v4-flash-vision-exp-review-pricing DeepSeek released DeepSeek V4 Flash Vision Exp (https://api-docs.deepseek.com/updates/) on August 21, 2026. The exact API model ID is deepseek-v4-flash-vision-exp. It accepts text and images, returns text, and keeps the same token rates as V4 Flash. That last detail is the real story. DeepSeek resizes each image before inference and caps it at 384 input tokens. This makes screenshot-heavy agent loops unusually inexpensive. It also creates the model's clearest trade-off: a large image is reduced to roughly an 800×800-pixel budget, so low cost should not be confused with full-resolution visual fidelity. This is an experimental model, not a silent production replacement for every vision workload. The analysis below separates DeepSeek's official specifications and provider-published benchmark claims from worked cost calculations and current Modelflare availability. Volatile facts were checked on August 22, 2026. DeepSeek V4 Flash Vision Exp at a glance Property | Official value | Release date | August 21, 2026 | Status | Experimental API model | Model ID | deepseek-v4-flash-vision-exp | Input and output | Text and image input; text output | Context window | 1M tokens | Maximum output | 384K tokens | Thinking | Thinking and non-thinking modes; thinking is the documented default | Image formats | JPEG, PNG, GIF, WebP | API formats | Chat Completions, Responses API, Anthropic-compatible Messages | Image token ceiling | 384 input tokens per image after automatic resizing | Concurrency limit | 2,500 | FIM completion | Not supported | DeepSeek has not disclosed parameter count, active parameters, expert count, training corpus, or architecture changes for this release. Any precise number presented without a first-party source is speculation. What actually changed from V4 Flash DeepSeek says the vision model is on par with the official V4 Flash for pure-text agents, reasoning, and world knowledge. The change is native visual understanding: screenshots, document images, charts, and other pictures can now enter the same agent loop without a separate image-captioning proxy. The launch table reports the following results for V4 Flash Vision Exp: Evaluation | DeepSeek-published score | Terminal Bench 2.1 | 83.9 | NL2Repo | 57.7 | DeepSWE | 59.3 | DSBench-Hard | 63.6 | AutomationBench, public | 25.7 | ApexBench, Pass@1 | 36.5 | Agents' Last Exam | 27.3 | Chartography | 64.3 | ZeroBench, Pass@5 | 35.0 | DeepSeek also says multimodal-agent performance is close to Claude Opus 4.8. This is a provider statement, not an independent Modelflare test. The rows mix text-agent and vision-dependent workloads, and the launch note specifies DeepSeek Harness minimal mode, max effort, top_p=0.95, and temperature=1.0 for its public code-agent tests. Use the table to choose workloads to replay, not as a universal ranking. Official pricing and the real image cost DeepSeek's pricing page (https://api-docs.deepseek.com/quick_start/pricing/) lists the following US-dollar rates per one million tokens. Peak hours are 01:00–04:00 and 06:00–10:00 UTC; all other hours are off-peak. Meter | Off-peak | Peak | Cached input | $0.007 | $0.014 | Uncached input | $0.22 | $0.44 | Output | $0.66 | $1.32 | Maximum input charge for one image at 384 tokens | $0.00008448 | $0.00016896 | The per-image row is a calculation, not a separate product price: 384 × input rate / 1,000,000. Smaller images may use fewer tokens. Text input and generated output are billed in addition to image tokens. Automatic resizing explains both the low price and the limit. Images below roughly 384×384 pixels are scaled up while preserving aspect ratio. Larger images are scaled down toward a total pixel count near 800×800. A 2000×2000 image and a 5000×5000 image therefore reach the same 384-token ceiling. You save money, but very small text, fine chart labels, and precise screen coordinates may disappear during resizing. Price comparison with Gemini 3.7 Flash and Claude Opus 4.8 The three models solve different versions of the multimodal problem. The table uses each vendor's direct standard API rates and excludes tools, storage, retries, and gateway discounts. Model | Release status | Input scope | Context / max output | Uncached input | Output | Main trade-off | DeepSeek V4 Flash Vision Exp | Experimental | Text, image | 1M / 384K | $0.22 off-peak; $0.44 peak | $0.66 off-peak; $1.32 peak | Lowest rates and 384-token image cap; aggressive image resizing | Gemini 3.7 Flash | GA | Text, image, video, audio, PDF | 1M / 64K | $0.75 through Dec 31, 2026 | $3.75 through Dec 31, 2026 | Broader modalities and built-in tools; higher price and later price increase | Claude Opus 4.8 | GA | Text, image, PDF | 1M / 128K | $5.00 | $25.00 | High-resolution vision and mature computer-use work; premium cost | For a text-only workload of 100K uncached input tokens plus 10K output tokens: DeepSeek costs about $0.0286 off-peak or $0.0572 peak. Gemini 3.7 Flash costs about $0.1125 at its introductory rate. Claude Opus 4.8 costs about $0.75. At peak rates, the DeepSeek example is about 49% cheaper than Gemini; off-peak it is about 75% cheaper. It is about 92% cheaper than Claude at peak and 96% cheaper off-peak. These percentages compare invoices, not answer quality. Image economics widen the gap, but resolution changes too. Anthropic documents 1,296 visual tokens for a 1000×1000 image on Opus 4.8, or about $0.00648 per image. DeepSeek caps any image at $0.00016896 peak and $0.00008448 off-peak. That is roughly 38× to 77× less input cost, but DeepSeek reduces the image toward an 800×800 pixel budget while Opus can retain more detail. As a concrete batch, 100 one-megapixel images plus 100K text input and 10K output cost at most about $0.0370 off-peak or $0.0741 peak on DeepSeek. Using Anthropic's documented 1,296 tokens per one-megapixel image, the equivalent token bill is about $1.398 on Opus 4.8. This is not a quality-equivalent test; it shows why cheap visual triage and high-resolution inspection should be treated as different workloads. Which model fits which workload Choose DeepSeek V4 Flash Vision Exp for a controlled test when the workload consists of many screenshots, ordinary documents, charts, OCR-like extraction, or visual observations inside a tool loop, and price per successful task matters more than retaining every source pixel. Choose Gemini 3.7 Flash for broader multimodality when one workflow must handle image, video, audio, and PDF inputs, or depends on Google's search grounding, code execution, file search, URL context, or preview Computer Use. It is GA, while the DeepSeek vision endpoint is explicitly experimental. Choose Claude Opus 4.8 for high-resolution visual agent work when dense documents, fine UI elements, computer use, and long-running browser tasks justify a premium. Anthropic supports up to 2,576 pixels on the long edge and 4,784 visual tokens for high-resolution images; that fidelity is part of why a per-image price comparison is not apples to apples. The practical architecture can use both tiers: a cheap first pass to classify, extract, and route images, then a high-resolution model only for ambiguous or detail-sensitive cases. Measure the extra handoff, latency, and error rate; a two-model pipeline is not automatically cheaper if most images escalate. Image input methods and limits DeepSeek's vision guide (https://api-docs.deepseek.com/guides/vision/) supports three input paths: inline base64 for a local image; a public HTTP or HTTPS URL; a reusable file_id from the Files API. Boundary | Limit | Inline request body | 48 MiB | One base64 or URL image | 32 MiB | One Files API image | 64 MiB | Images in one request | 600 | Total images without file_id | 64 MiB | Total images including file_id | 200 MiB | Maximum image edge | 8,192 px; 4,096 px with 15 or more images | External URL | 8,192 characters and a 60-second download limit | detail: "low" forces a 512×512 preprocessing step and is useful for fast, low-detail OCR or classification. high, original, and currently auto preserve the original before DeepSeek's normal resizing rule. In Chat Completions, images are allowed only in user messages. The Responses API also permits them in developer messages and tool-output items, but rejects image parts in system or assistant messages. Minimal Responses API example through Modelflare Modelflare now lists the exact model ID and the Responses API route. Use an API key assigned to the deepseek-stable group: import os from openai import OpenAI client = OpenAI( api_key=os.environ["MODELFLARE_API_KEY"], base_url="https://modelflare.dev/v1", ) response = client.responses.create( model="deepseek-v4-flash-vision-exp", input=[ { "role": "user", "content": [ { "type": "input_text", "text": "Read this dashboard. Return the three anomalies and the visible evidence for each.", }, { "type": "input_image", "image_url": "https://example.com/dashboard.png", "detail": "low", }, ], } ], ) print(response.output_text) For a private image, use base64 or the Files API instead of a public URL. Do not place credentials, private object URLs, or personal data in logs. The Responses implementation is stateless: previous_response_id, conversations, and store are not supported. Function calling and web search are supported, while built-in computer use, code interpreter, file search, and MCP are ignored by DeepSeek's Responses-compatible endpoint. Production checklist for an experimental vision model Pin the exact deepseek-v4-flash-vision-exp ID; do not assume deepseek-v4-flash is a vision alias. Build a privacy-safe evaluation set with small text, dense tables, charts, screenshots, rotated images, and adversarial content. Compare detail: "low" with the default on task success, latency, and billed input—not visual aesthetics. Test Chat Completions, Responses, and Messages separately if your clients use more than one protocol. Verify structured output, tool-call correlation, streaming, cancellation, refusals, and malformed image handling. Use Files API IDs for repeated private images and define retention and deletion procedures. Record image count, dimensions, input tokens, cached tokens, output tokens, retries, latency, route, and final charge. Keep a GA fallback for detail-sensitive or business-critical work until the experimental model passes a real traffic threshold. An HTTP 200 only proves that one request completed. It does not prove that a screenshot was read correctly, every tool call was authorized, or the low-cost path reduced cost per successful task. DeepSeek V4 Flash Vision Exp pricing on Modelflare Yes. In the later production check on August 22, 2026, Modelflare's public Models & Pricing catalog listed the exact deepseek-v4-flash-vision-exp ID in the deepseek-stable group, with OpenAI-compatible Chat Completions and Responses API routes. The public catalog configures the official peak reference rates—$0.44 input, $1.32 output, and $0.014 cached input per million tokens—and applies a 0.9x deepseek-stable group multiplier: Modelflare group | Input | Cached input | Output | Maximum uncached input cost for one 384-token image | deepseek-stable | $0.396 / 1M | $0.0126 / 1M | $1.188 / 1M | $0.000152064 | At these rates, 100K uncached text-input tokens plus 10K output tokens cost about $0.05148. A workload with 100 images at the 384-token ceiling, 100K text input, and 10K output costs at most about $0.06669, before retries or other requests. Do not treat the 0.9x group price as DeepSeek's direct off-peak schedule. Modelflare applies the group multiplier to its configured reference prices; it does not automatically switch between DeepSeek's vendor peak and off-peak windows. Group access, pricing, and routes can change, so use the live pricing page and the completed request record as the current source of truth. Bottom line DeepSeek V4 Flash Vision Exp is compelling because it changes the economics of visual agents: image input is capped at 384 tokens and priced at the same rate as V4 Flash. For screenshot triage, document extraction, chart reading, and high-volume visual tool loops, that can make a meaningful cost difference. The same cap is also the warning label. Aggressive resizing can erase the exact detail that computer-use and dense-document tasks need, the endpoint is experimental, and DeepSeek's launch benchmarks are provider-published. Treat it as a low-cost visual worker to evaluate—not proof that one model now replaces every multimodal stack. Official sources and update log Primary sources: DeepSeek API change log: V4 Flash Vision Exp release (https://api-docs.deepseek.com/updates/) DeepSeek Vision guide (https://api-docs.deepseek.com/guides/vision/) DeepSeek models and pricing (https://api-docs.deepseek.com/quick_start/pricing/) DeepSeek Responses API guide (https://api-docs.deepseek.com/guides/responses_api/) Google: Gemini 3.7 Flash model page (https://ai.google.dev/gemini-api/docs/models/gemini-3.7-flash) Google: Gemini API pricing (https://ai.google.dev/gemini-api/docs/pricing) Anthropic: Claude model overview (https://platform.claude.com/docs/en/about-claude/models/overview) Anthropic: vision tokenization and image limits (https://platform.claude.com/docs/en/build-with-claude/vision) Update log: August 22, 2026: Initial publication. Official release status, specifications, image processing, limits, protocol support, peak/off-peak pricing, competitor prices, worked calculations, and Modelflare catalog availability checked. Later on August 22, 2026: Updated after the exact model became available on Modelflare. Added the deepseek-stable 0.9x prices, worked Modelflare costs, supported routes, and a Modelflare Responses API example. ### AI API Fallback Strategy: Build a Provider Failure Matrix Canonical URL: https://modelflare.dev/articles/ai-api-fallback-matrix An AI API retry, a route fallback, and a model substitution are three different actions. A retry repeats an attempt under the same contract. A route fallback sends the requested model and protocol to another eligible channel or group. A model substitution deliberately changes the requested model and may change quality, price, latency, tool behavior, context limits, and output format. A reliable policy decides which action is allowed before an incident. The decision must consider the failure class, whether any response has already reached the caller, whether replay is idempotent, and how many attempts fit within the user’s deadline. Separate retry, fallback, and model substitution Use distinct names in configuration, logs, and runbooks. Action | What changes | Appropriate purpose | Main risk | Same-route retry | Time and attempt number | Recover from a short transient failure on the same route | Repeating load against an unhealthy dependency | Same-contract fallback | Upstream channel, account, or explicitly ordered group | Preserve the requested model and protocol when one path fails | Hidden incompatibility between supposedly equivalent routes | Model substitution | Model identifier or a named model policy | A product-approved quality, cost, or availability tradeoff | Silent behavior and billing changes | Do not label all three as “retry.” Operators need to know whether a request was replayed, moved to another path, or answered by a different model. Users need model substitution to be an explicit product contract, not an invisible recovery shortcut. Some gateways support ordered model or provider steps. For example, Cloudflare’s fallback documentation (https://developers.cloudflare.com/ai-gateway/configuration/fallbacks/) exposes which step produced the successful response. The important design lesson is not to copy a vendor’s exact policy; it is to preserve attempt-level evidence whenever a route changes. Apply four gates before replaying a request A status code alone is not a retry policy. Evaluate four gates: Failure class: Is the failure transient, permanent, caller-caused, or ambiguous? Response phase: Did it happen before headers, before effective output, or after output was delivered? Idempotency: Can the complete application operation be repeated without a duplicate side effect? Attempt budget: Is there enough remaining wall-clock time and an unused attempt? The Google Cloud retry strategy (https://docs.cloud.google.com/storage/docs/retry-strategy) makes the same two foundational distinctions for general APIs: the response indicates whether retry may be useful, and idempotency determines whether replay is safe. It lists 408, 429, 5xx, socket timeouts, and disconnects as commonly transient, while warning that non-idempotent operations need stronger conditions. For an AI workflow, idempotency extends beyond the model HTTP request. A replayed prompt can propose the same email, refund, deployment, or database write again. Tool execution therefore needs its own stable idempotency key and persisted result, even if the inference call itself is read-only. Start from a failure matrix The matrix below is a conservative application policy. A gateway may perform an internal same-contract channel failover before the application sees the terminal result, so coordinate the two layers rather than duplicating them. Failure or phase | Same-route retry | Same-contract fallback | Stop or investigate | Reason | Client validation error, unsupported field, or malformed request | No | No | Fix request | Replaying the same invalid contract cannot succeed | Gateway authentication, authorization, quota, or policy denial | No | No | Stop and correct account or policy | Another provider route must not bypass the gateway decision | Upstream credential or account failure before output | No on the failed route | Yes, if another verified channel is eligible | Quarantine and investigate failed channel | The user contract can remain the same while an unhealthy credential is removed | Network failure or 408 before response output | At most one bounded attempt if idempotent | Yes | Stop at deadline | The result is transient, but completion may be ambiguous after a disconnect | 429 before output | Honor Retry-After for delayed retry | Yes when another same-contract route has capacity | Stop when budget is exhausted | Immediate repeated retries amplify rate limiting | Upstream 500, 502, 503, or 504 before output | Bounded retry with backoff | Yes | Investigate repeated route failures | Usually transient, but not evidence that every route is safe | Schema-invalid or malformed provider response before downstream output | Usually no | Only to a route verified for the same schema | Quarantine or investigate compatibility | Repeating the same incompatible implementation is unlikely to help | Model refusal or policy-safe completion | No | No | Return the modeled outcome | A valid refusal is not an infrastructure outage to route around | Caller cancellation or downstream 499 | No | No | Stop immediately | The caller no longer wants work; retries waste capacity and cost | Partial stream after visible content or tool arguments reached the caller | No transparent replay | No transparent fallback | Mark partial and let the application decide | A second stream can duplicate or contradict already delivered output | Tool side effect has unknown completion state | No until reconciled | No until reconciled | Query the idempotency record or downstream system | Re-inference can propose the same side effect again | This matrix is deliberately phase-aware. A 503 received before any downstream output is different from a connection that closes after 400 text tokens have already been rendered. Treat a started stream as a commit boundary Before downstream output starts, a gateway can often discard a failed attempt and try another eligible route without exposing two answers. After the first byte of meaningful output reaches the caller, transparent replay becomes unsafe. A restarted stream may: repeat the beginning of an answer; produce a different continuation; issue a duplicate function call with a new Call ID; change usage and cost without a clear boundary; leave the client unable to tell which events belong together. When a stream breaks after output has started, return a terminal partial or transport error with the original request identity. The application can offer an explicit “try again” action, continue from a safe application checkpoint, or discard the partial output. It should not splice a new model stream onto the old stream as if nothing happened. For Function Calling, the safest boundary is stricter: persist accepted Tool Call identities and side-effect results before any retry can recreate them. The Function Calling comparison (/articles/function-calling-responses-vs-chat-completions) shows how Call IDs and application idempotency fit together. Bound backoff by attempts and wall-clock time Exponential backoff spreads repeated attempts over time. Jitter prevents many clients from retrying in lockstep after a shared outage. delay_cap = min(max_delay, base_delay * 2^retry_index) sleep_for = random_between(0, delay_cap) Honor a valid Retry-After signal when it fits the user’s deadline. Backoff is not permission to retry: the failure and idempotency gates must pass first. Define a total budget, not only a retry count: maximum attempts for one user action; maximum elapsed time including queueing and backoff; maximum attempts before and after selecting a fallback group; minimum time left to produce a useful answer; cancellation propagation from caller to every active attempt. For an interactive request with a 15-second deadline, three 10-second attempts are not a real policy. The later attempts cannot finish within the user contract. Batch workloads can use a longer budget, but should still have a terminal deadline and a durable job identity. Prevent retry amplification across layers Assume an SDK makes one request plus two retries: three client attempts. The gateway makes one primary attempt plus two channel fallbacks for each client attempt: three gateway attempts. An upstream proxy makes one retry per attempt: two provider calls. 3 client attempts × 3 gateway attempts × 2 upstream attempts = 18 provider calls One user action has now become 18 provider calls. During an outage, this increases queueing, rate limiting, cost, and recovery time. Assign retry ownership deliberately: the gateway owns immediate same-contract channel failover; the application owns whether a complete user action may be repeated; an SDK’s automatic retries are disabled or bounded when the gateway already retries; asynchronous jobs use one durable job ID and an attempt ledger; no layer starts a new attempt after caller cancellation. Log both “attempt number at this layer” and a stable end-to-end request identity. Otherwise each layer appears to have made only two or three attempts while the combined amplification remains hidden. Verify that a fallback really preserves the contract The same public model name does not prove two routes behave identically. Before adding a channel to a same-contract fallback set, test: Contract area | Evidence required | Model identity | Requested model remains available without silent mapping | Endpoint | Responses or Chat Completions request is accepted as configured | Streaming | Event types, termination, usage, and cancellation are handled | Structured output | Required JSON Schema subset and strict behavior work | Function calling | Tool definitions, Call IDs, argument streaming, and results round-trip | Limits | Context, output, rate, and concurrency limits fit the workload | Errors | Status and error bodies remain classifiable without exposing secrets | Usage and cost | Token accounting, cache fields, service tier, and price policy are understood | Safety and region | Required policy, data path, and residency constraints remain satisfied | If a candidate route fails one required area, it is not a transparent fallback for that workload. It may still be usable under a separate, explicit product policy. Changing to another model is always such a policy decision. Define the permitted model, quality floor, price ceiling, Tool Contract, and user-visible disclosure. Do not use a cheaper or weaker model merely because a route returned an error. Understand Modelflare’s current fallback semantics Modelflare searches eligible paths for the model requested by the API client. A regular API Key has one primary group and can have an ordered list of fallback groups. A Smart API Key evaluates groups available to the account using its configured routing strategy. Neither mechanism should silently replace the requested model with another model. Group-level RPM admission occurs before billing and before an upstream request. If the selected group is full, an ordered fallback group or Smart Routing candidate can be evaluated; if no eligible group remains, the request returns 429. Within a selected group, channel priority defines account failover order. After an upstream error, the failed channel is excluded and selection continues through remaining eligible channels. Current channel failover is independent of RetryTimes and AutomaticRetryStatusCodes. It stops on success, eligible-route exhaustion, caller cancellation, or after a downstream response has started and can no longer be transparently replayed. These are internal same-model path decisions, not permission for a client to add another unlimited retry loop. Use Reliable AI API Routing (/articles/reliable-ai-api-routing) for group and channel design, and AI API Error Troubleshooting (/articles/ai-api-error-troubleshooting) to distinguish gateway policy errors from upstream failures. Preserve evidence for every attempt A final 200 does not prove that the first route succeeded. A final channel ID does not describe failed attempts. Retain enough metadata to reconstruct the path: stable request ID and caller-visible correlation ID; attempt sequence and selected group/channel reference; model and endpoint contract used for each attempt; failure status, error class, and stream phase; whether downstream output had started; timing milestones and cancellation state; input, output, and cached-token usage when available; cost attribution by completed or billable attempt; final terminal reason: success, exhausted, cancelled, partial, or policy stop. Do not retain API keys, raw prompts, raw responses, or provider credentials merely to diagnose fallback. Redacted error classes and timing metadata are usually sufficient, with restricted short-lived request archives reserved for explicitly configured failure investigation. Drill the policy before production traffic depends on it Run a staging exercise using the real protocol boundary and safe deterministic inputs: make the primary channel unavailable before headers and verify the next eligible same-model path; return a rate limit and verify attempt limits and any Retry-After handling; cancel the caller and prove that no later attempt starts; break a stream after output and prove that no transparent replay occurs; send an invalid request and prove that fallback does not hide it; repeat a tool workflow and prove that one side effect is recorded; exhaust every route and verify one clear terminal error; inspect the attempt ledger and reconcile usage and cost. Roll out the policy to a small workload, monitor attempt count and success-after-fallback separately from raw success rate, and keep a fast way to remove an unhealthy route. The goal is not to maximize fallback frequency. It is to recover only when the original model contract can be preserved safely, within a bounded deadline, with evidence for every attempt. ### AI API Latency Metrics: TTFT, First Response, and Output Speed Canonical URL: https://modelflare.dev/articles/ai-api-latency-metrics AI API latency is not one number. For a streaming request, record when the upstream headers arrive, when the first non-empty SSE event arrives, when the first meaningful content or action appears, when the first visible text appears, and when the response completes. These milestones answer different questions and should not all be labeled “TTFT.” For a text-only chat, time to first visible text is usually the user-facing latency. For a reasoning or tool-using workflow, the first effective output can occur earlier as reasoning metadata or function-call arguments. Generation speed is a separate measurement taken after output starts. Build one request timeline before naming a metric Use a monotonic clock and one origin for all client- or gateway-level timestamps. A streaming request can be represented as follows: t0 gateway receives request t1 gateway starts upstream request t2 upstream response headers arrive t3 first non-empty SSE event arrives t4 first effective content or action arrives t5 first visible text delta arrives t6 upstream body completes or closes t7 gateway handler finishes Not every request has every milestone. A non-streaming response has no useful SSE timeline. A function-call-only response may never produce visible text. A cancelled stream may have a start but no normal completion. Store absent values as absent rather than as zero, because zero would falsely imply that the event happened immediately. Modelflare records request-level metadata under other.timing. The fields map to the timeline like this: Field | What it measures | Important boundary | auth_ms | Token and user authentication duration | Local phase duration, not upstream time | distribution_ms | Channel selection and routing-policy duration | Local phase duration | body_read_ms | Time spent reading the downstream request body | Can expose a slow upload before routing | upstream_headers_ms | Upstream request start to response headers | Its origin is upstream start, not initial gateway receipt | first_sse_event_ms | Gateway receipt to first non-empty upstream SSE event | An event can contain metadata rather than useful output | first_response_ms | Gateway receipt to first effective content or action delta | Includes visible text, reasoning summary, function arguments, or custom tool input | first_text_delta_ms | Gateway receipt to first visible text delta | Exists only when visible streaming text appears | upstream_done_ms | Gateway receipt to upstream body completion or close | A close is not automatically a successful completion | total_handler_ms | Gateway receipt to the latest known relay completion point | Closest gateway-side end-to-end duration | visible_output_tps | Output tokens divided by time from first visible text to upstream completion | A visible-output diagnostic, not system-wide TPS | This mapping prevents a common error: subtracting timestamps that do not share an origin. For example, upstream_headers_ms is a duration beginning when the upstream request starts, while first_response_ms begins when the gateway receives the request. TTFT has more than one practical meaning The conventional definition of time to first token is the duration from query submission to the first received output token. The NVIDIA NIM benchmarking guide (https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html) notes that this generally includes network latency, queueing, and prompt prefill, and that empty initial responses should not count. That definition works for a text-generation benchmark, but modern API streams can emit several things before visible text: a response-created or metadata event; a reasoning summary delta; function-call argument fragments; custom tool input; an empty heartbeat or provider-specific envelope. Use explicit names in dashboards and incident reports: Name | Recommended meaning | Best use | Time to headers | Request start to upstream headers | Network, proxy, and upstream admission diagnosis | Time to first event | Request start to first non-empty SSE event | Transport liveness only | Time to first effective response | Request start to useful content or action | Agent and reasoning workflow responsiveness | Time to first visible text | Request start to text a user can render | Chat and user-perceived responsiveness | If a chart says only “TTFT,” document which row it means. Otherwise two teams can report different values for the same request and both appear correct. Separate responsiveness from generation speed End-to-end latency measures the whole request from submission until the full response is received. It includes queueing, prefill, decoding, network transfer, tool pauses represented within the upstream stream, and gateway work visible inside the chosen boundary. For a simple completed text stream: end_to_end_latency = final_response_time - request_start_time visible_generation_window = final_response_time - first_visible_text_time visible_output_tps = output_tokens / visible_generation_window_seconds Inter-token latency, also called time per output token, is normally the average spacing between output tokens after the first token. NVIDIA’s AIPerf definition excludes TTFT and divides the remaining duration by output_tokens - 1. Its reciprocal approximates per-user decode throughput for sufficiently long outputs. Do not calculate token-level ITL by counting SSE events or text Delta objects. One event can contain zero, one, or several tokens, and token boundaries depend on the model tokenizer. Modelflare’s visible_output_tps uses the recorded output-token count and the visible generation window; it does not claim to be a per-token event-spacing measurement or total system throughput. These metrics answer separate questions: low first-text latency with slow output speed feels responsive at first and then sluggish; high first-text latency with fast output speed feels stalled and then finishes quickly; low per-user output speed does not prove low system throughput under concurrency; high aggregate TPS does not guarantee good latency for one user. Reasoning and tools can precede visible text In a text-only flow, first_response_ms and first_text_delta_ms may be nearly identical. In a reasoning or function-calling flow, the gap can be meaningful. Suppose a model emits function-call arguments at 1.8 seconds, the application or provider-side tool flow continues, and visible text begins at 6.4 seconds. The system has produced an actionable result at 1.8 seconds, but the user has not seen prose until 6.4 seconds. Calling both values TTFT hides whether the delay occurred before planning, during a tool step, or before answer rendering. Interpret the fields according to the product experience: for a terminal agent, first effective action may be the best responsiveness signal; for a chat UI that displays only prose, first visible text is the relevant user signal; for a tool-call API consumed by code, visible text may be absent and should not be required; for a reasoning model, an early SSE envelope proves connection progress but not useful progress. The AI API Streaming Guide (/articles/ai-api-streaming-guide) covers SSE parsing, cancellation, and idle-timeout behavior. Latency metrics should be computed only after the parser distinguishes envelopes from effective output. Diagnose the slow stage in a fixed order Start at the upstream boundary, then move toward local phases. This avoids blaming authentication or routing when the request spent most of its time waiting for the model. Symptom | Primary metric | Likely layer to inspect | Next check | Slow before headers | upstream_headers_ms | Network path, upstream admission, provider queue, proxy route | Compare channel, region, status, and concurrency | Headers are fast, useful output is late | first_response_ms minus early phases | Model queueing, prompt prefill, reasoning, upstream scheduling | Compare input tokens, model, route, and inflight count | First event is fast, effective output is late | Gap from first_sse_event_ms to first_response_ms | Metadata-only events, heartbeats, reasoning startup | Inspect safe event types, not raw content | Effective action is fast, visible text is late | Gap from first_response_ms to first_text_delta_ms | Tool/reasoning phase or response composition | Check requested tools and output mode | Text begins quickly, then crawls | visible_output_tps and upstream_done_ms | Decode throughput, contention, long context, network backpressure | Compare output length and channel inflight count | All upstream metrics are normal | auth_ms, distribution_ms, body_read_ms | Local auth, policy selection, client upload | Inspect only the phase that is elevated | No single metric proves a root cause. High upstream_headers_ms, for example, groups several possibilities that require route, region, provider, and concurrent-load evidence to separate. Read three synthetic traces The following values are illustrative diagnostics, not Modelflare production averages or provider benchmarks. Trace A: waiting before upstream headers Metric | Value | upstream_headers_ms | 6,100 ms | first_sse_event_ms | 6,300 ms | first_response_ms | 6,350 ms | first_text_delta_ms | 6,400 ms | total_handler_ms | 9,200 ms | visible_output_tps | 42 | Most of the wait happens before headers. Generation after visible text is comparatively healthy. Check the selected upstream route, provider admission, network path, region, and concurrent load before optimizing the client renderer. Trace B: useful action before visible prose Metric | Value | upstream_headers_ms | 240 ms | first_sse_event_ms | 310 ms | first_response_ms | 2,900 ms | first_text_delta_ms | 8,700 ms | total_handler_ms | 10,200 ms | visible_output_tps | 55 | Transport is alive early and an effective action appears at 2.9 seconds, but visible text waits another 5.8 seconds. If the request uses reasoning or tools, inspect that phase. Increasing a header timeout would not address this pattern. Trace C: fast start, slow generation Metric | Value | upstream_headers_ms | 260 ms | first_sse_event_ms | 330 ms | first_response_ms | 420 ms | first_text_delta_ms | 430 ms | total_handler_ms | 20,430 ms | visible_output_tps | 9.8 | The request becomes visible quickly but spends about 20 seconds generating. Compare output length, context length, selected channel, inflight count, and provider behavior. A first-output timeout would pass and therefore cannot detect this failure mode. Compare latency only under controlled conditions A fair comparison holds the workload contract stable. At minimum, record: exact model identifier and route or group; input-token and output-token distributions, not just averages; streaming versus non-streaming mode; enabled reasoning effort and tools; region and client-to-gateway network path; concurrency or arrival rate; sampling settings and maximum output length; warmup policy, retry policy, and excluded failures; sample size, time window, and percentile method. Compare p50, p95, and p99 rather than presenting one average. Keep failures visible: removing timeouts and errors can make a route look faster precisely when it is less reliable. Never compare two providers with different prompts, output lengths, concurrency, or endpoint behavior and present the result as a model-speed ranking. Retain diagnostic metadata without retaining content Latency diagnosis does not require storing prompts, responses, API keys, or plaintext client identities. A useful request record can contain: a generated request ID and timestamp; model, group, and selected channel reference; status and terminal outcome; input, output, and cached-token counts when available; the timing fields above; stream event and text-delta counts; channel inflight count; coarse region and a privacy-reviewed network trace identifier; a coarse classifier such as upstream_headers_slow or generation_slow_tps. Apply retention limits and access controls even to metadata. Request IDs, route choices, timing patterns, and token counts can still reveal operational behavior when combined. Begin an investigation with the request timeline, not a single TTFT label. Then use Reliable AI API Routing (/articles/reliable-ai-api-routing) to compare the exact route selected for each attempt and AI API Error Troubleshooting (/articles/ai-api-error-troubleshooting) to pair timing with the terminal status. This produces evidence that can separate admission delay, model startup, reasoning or tool work, visible generation, and local gateway overhead. ### Grok 4.6 vs GPT-5.6 Sol: Coding, Agents, Context, and API Cost Canonical URL: https://modelflare.dev/articles/grok-4-6-vs-gpt-5-6-sol xAI released Grok 4.6 (https://x.ai/news/grok-4-6) on August 12, 2026, putting it directly into the same coding-agent conversation as OpenAI's GPT-5.6 Sol. The useful question is not which launch chart has the taller bar. It is which model completes your workload with fewer retries, less context growth, and an acceptable bill. The short answer: Grok 4.6 is the stronger price-first default for iterative coding and agent loops; GPT-5.6 Sol is the stronger capability-first choice for the hardest tool-heavy work, very large contexts, and workflows that use OpenAI's newer agent controls. Neither is a universal winner. All specifications and prices below were checked against official vendor documentation on August 15, 2026. Benchmark figures are vendor-reported, not independent Modelflare test results. Quick answer Pick Grok 4.6 when output cost matters, your working context stays below 500K, and you want a capable coding or research agent without paying flagship output rates. Pick GPT-5.6 Sol when the task benefits from a 1.05M context window, max reasoning, Pro mode, persisted reasoning, Programmatic Tool Calling, or OpenAI's multi-agent beta. Do not choose from token price alone. A cheap run that retries twice can cost more than an expensive run that lands the change once. For production routing, test both on the same repositories, tools, timeouts, and acceptance criteria. For the xAI-specific API contract, pricing boundary, and migration checklist, read the Grok 4.6 API guide (/articles/grok-4-6-api-guide). Grok 4.6 vs GPT-5.6 Sol specifications Property | Grok 4.6 | GPT-5.6 Sol | Release | August 12, 2026 | July 9, 2026 | Exact API model ID | grok-4.6 | gpt-5.6-sol (gpt-5.6 alias) | Context window | 500,000 tokens | 1,050,000 tokens | Maximum output | No fixed limit listed by xAI; request and context limits still apply | 128,000 tokens | Modalities | Text and image input; text output | Text and image input; text output | Reasoning effort | low, medium, high, xhigh | none, low, medium, high, xhigh, max | API formats | Responses, Chat Completions | Responses, Chat Completions | Documented agent features | Function calling, structured outputs, web/X search, code execution | Function calling, structured outputs, hosted tools, persisted reasoning, Programmatic Tool Calling, multi-agent beta, Pro mode | Parameter counts are absent because neither vendor publishes an authoritative parameter count for these production models. Treat third-party numbers as estimates, not API specifications. Official API pricing and long-context cost The headline rates per one million tokens are: Price component | Grok 4.6 | GPT-5.6 Sol | Input | $2.00 | $5.00 | Cached input | $0.50 | $0.50 | Output | $6.00 | $30.00 | Long-context boundary | At or above 200K prompt tokens | Above 272K input tokens | Long-context rule | $4 input, $1 cached input, $12 output | 2x input and 1.5x output for the full request | Worked examples using official token rates: Request shape | Grok 4.6 | GPT-5.6 Sol | 100K input + 5K output | $0.23 | $0.65 | 220K input + 10K output | $1.00 | $1.40 | 300K input + 10K output | $1.32 | $3.45 | These examples exclude built-in tool fees, cache writes, fast or priority service tiers, retries, and gateway pricing. Grok crosses its higher rate at 200K prompt tokens; Sol crosses at more than 272K input tokens. Both rules apply to the entire qualifying request, so context compaction can change the bill materially. The cached-input headline is equal at $0.50 per million before long-context adjustments. The large gap is output: Sol's standard output rate is five times Grok 4.6's. That matters for agents that produce long patches, tool arguments, test logs, and recovery turns. What the launch benchmarks actually show xAI's Grok 4.6 announcement publishes a direct table against GPT-5.6 Sol. It compares Grok 4.6 High with GPT-5.6 Sol Max and says competitor figures come from published system cards or benchmark leaderboards. Evaluation | Grok 4.6 High | GPT-5.6 Sol Max | Artificial Analysis Intelligence Index | 61 | 61 | GDPVal-AA v2 | 1753 | 1728 | CursorBench v3.2 | 69.9% | 67.2% | DeepSWE v1.1 | 65.9% | 73.0% | FrontierCode v1.1 Extended | 61.3% | 60.6% | APEX-Agents | 57.5% | 56.7% | Terminal-Bench v3.0 | 26.0% | 34.6% | AA-Briefcase | 1577 | 1502 | Read this as a workload map, not a final ranking. On xAI's table, Grok leads several knowledge-work and coding-agent measures, while Sol leads DeepSWE and Terminal-Bench. The reasoning settings differ, the data comes from a vendor launch post, and benchmark harnesses do not reproduce your repository, tools, permissions, or definition of done. Where each model is meaningfully different Grok 4.6's practical advantage is simpler economics for long agent trajectories. Its standard input rate is lower, its output rate is much lower, and xAI explicitly trained the model for multi-step research, codebase work, web development, and self-verification. It also offers an xhigh reasoning level and a fast service variant. GPT-5.6 Sol has the broader execution surface. Its context window is more than twice as large, max adds another reasoning level, and Pro mode spends more model work on one answer. The Responses API can preserve reasoning across turns, execute eligible tool calls programmatically, and coordinate subagents through a beta multi-agent feature. Those are real differences when the harness can use them; they do not help if your client only sends ordinary Chat Completions requests. Which model should you choose? Workload | Better starting point | Why | Frequent coding iterations with controlled context | Grok 4.6 | Lower input and output rates; strong launch results on several coding-agent evals | Repository or document context above 500K | GPT-5.6 Sol | 1.05M context window | Output-heavy agents | Grok 4.6 | $6/M standard output versus $30/M | Deep terminal and software-engineering tasks | GPT-5.6 Sol | Leads DeepSWE and Terminal-Bench in xAI's published comparison | Programmatic tool orchestration or multi-agent execution | GPT-5.6 Sol | Native Responses features documented by OpenAI | Cost-sensitive fallback behind a premium model | Grok 4.6 | Lower retry and output exposure while retaining frontier-class capability | High-stakes production changes | Test both | Task success, review findings, latency, and rollback safety matter more than one benchmark | A sensible router does not need one permanent winner. Use the cheaper model for routine work, escalate tasks that fail a clear acceptance gate, and keep the final model, attempts, usage, latency, and charge visible. Call both models through one Modelflare endpoint At the production catalog check on August 15, 2026, Modelflare listed both exact model IDs with OpenAI-compatible Responses and Chat Completions support. grok-4.6 was available in the grok-award and grok-stable groups; gpt-5.6-sol was available in the corresponding OpenAI groups and additional eligible groups. The request stays the same; only MODEL_ID changes: export MODELFLARE_API_KEY="" export MODEL_ID="grok-4.6" # or gpt-5.6-sol curl -sS https://origin.modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d "{ \"model\": \"$MODEL_ID\", \"input\": \"Review this patch for correctness, race conditions, and billing regressions. Return only actionable findings.\" }" Check the live Grok 4.6 pricing page (/pricing/grok-4.6) and GPT-5.6 Sol pricing page (/pricing/gpt-5.6-sol) before budgeting. Group access, multipliers, routes, and upstream prices can change. A catalog entry also does not prove that every vendor-native tool or beta field is translated by an OpenAI-compatible adapter. Run a useful comparison on your own workload Use the same evaluation contract for both models: Select 10–30 representative tasks, including at least a few failures from production. Freeze the repository state, prompt, tool schema, timeout, and permission boundary. Run the same reasoning class where possible and record any unavoidable mismatch. Measure accepted tasks, review findings, wall time, input, cached input, output, tool calls, retries, and total charge. Separate first-pass success from success after recovery; retries are part of model cost. Inspect failures by type: wrong assumption, missed edge case, invalid tool call, incomplete patch, or excessive context growth. Choose the route by cost per accepted task, not cost per token. For long-running agents, also test immediately below and above each model's long-context boundary. A workload that slowly grows past 200K or 272K can change the comparison without any code change. Frequently asked questions Is Grok 4.6 better than GPT-5.6 Sol for coding? Not across every coding task. xAI's launch table shows Grok 4.6 ahead on CursorBench and slightly ahead on FrontierCode, while GPT-5.6 Sol leads DeepSWE and Terminal-Bench. Use Grok as a cost-efficient starting point and test Sol on the hardest repository and terminal tasks. Is GPT-5.6 Sol worth the higher API price? It can be when the larger context, deeper reasoning controls, or Responses-only agent features improve first-pass success. If your workflow does not use those capabilities, the five-times-higher standard output rate is harder to justify. Which one is cheaper for long conversations? Usually Grok 4.6 at official list price, especially when output is large. However, Grok's long-context pricing starts earlier at 200K prompt tokens. Measure context growth, cache hits, retries, and output together. Can one API key switch between them? Yes on Modelflare when the key has access to eligible groups for both models. Keep the exact model IDs, verify the supported endpoint, and run a non-sensitive request before changing production traffic. Official sources and update log Primary sources: xAI: Introducing Grok 4.6 (https://x.ai/news/grok-4-6) xAI developer guide: Grok 4.6 (https://docs.x.ai/developers/grok-4-6) xAI API pricing (https://docs.x.ai/developers/pricing) OpenAI: GPT-5.6 Sol model page (https://developers.openai.com/api/docs/models/gpt-5.6-sol) OpenAI: GPT-5.6 model guidance (https://developers.openai.com/api/docs/guides/latest-model) OpenAI: GPT-5.6 launch (https://openai.com/index/gpt-5-6/) Update log: August 15, 2026: Initial comparison. Official specifications, prices, long-context rules, launch benchmark table, and Modelflare catalog availability checked. ### Seedance 2.5 API Guide: Modelflare Setup, Pricing, and Model Comparison Canonical URL: https://modelflare.dev/articles/seedance-2-5-api-guide ByteDance Seedance 2.5 (https://seed.bytedance.com/en/seedance2_5) is a joint audio-video generation model built around a practical change: a single generation can tell a story for up to 30 seconds. ByteDance also highlights more precise reference-video understanding, broader editing control, and production-oriented tools such as white-model control and green-screen editing. That does not make every Seedance 2.5 integration identical. A model can have capabilities that a particular API does not expose. This guide separates ByteDance's official model claims from the fields available through Modelflare, then compares Seedance 2.5 with Seedance 2.0, Veo 3.1, Kling 3.0 Pro, Runway Gen-4.5, and Sora 2 using current documented contracts rather than promotional demo rankings. All volatile availability and Modelflare pricing facts below were checked on August 15, 2026. Seedance 2.5 at a glance Property | What is currently documented | Model family | ByteDance Seedance | Core architecture | Joint audio-video generation | Maximum native clip | Up to 30 seconds in one generation | Continuation | The official page says a result can be extended twice | Reference control | More precise interpretation of reference-video intent, framing, and cinematic language | Editing | Broader audio and visual editing requests, with more reliable instruction response | Production controls | White-model control, green-screen editing, camera movement, and performance blocking | Modelflare model ID | seedance-2.5 | Modelflare public surface | Asynchronous Video API for text-to-video and single-image-to-video | The official page does not publish a parameter count, training-compute figure, or a neutral cross-provider benchmark. It also does not substantiate several numbers repeated by third-party launch articles. This guide therefore does not treat claims such as a particular reference-asset count as official Seedance 2.5 API limits. What changed from Seedance 2.0 Seedance 2.0 (https://seed.bytedance.com/en/blog/seedance-2-0-official-launch) already combined text, image, video, and audio input in one audio-video model. ByteDance documents up to 9 images, 3 video clips, and 3 audio clips for its full reference workflow, along with 15-second multi-shot output, stereo audio, video editing, and extension. Decision point | Seedance 2.0 | Seedance 2.5 | Practical effect | Native duration | Up to 15 seconds | Up to 30 seconds | 2.5 can hold a longer beat or scene without a stitch-first workflow | Reference workflow | Explicit mixed-reference limits are documented | Reference-video interpretation is emphasized, but no numeric limit appears on the official page | 2.0 has the clearer published input contract; 2.5 targets finer interpretation | Editing | Prompt-directed editing and continuation | Broader, more reliable audio-visual editing | 2.5 is the better candidate when revision control matters more than first-pass cost | Production control | Camera planning and multimodal direction | Adds emphasis on white models, green screens, camera movement, and blocking | 2.5 is aimed more directly at controlled production pipelines | Current Modelflare price | Lower | 50% higher at every listed resolution | 2.0 remains the economical baseline when 15 seconds and simpler control are enough | The upgrade is therefore not simply “twice the duration.” Its intended value is fewer continuity breaks, stronger interpretation of reference footage, and more controllable revision. Those advantages need to be tested on your own shots; a longer clip is more expensive to discard when identity, hands, dialogue, or motion fails late in the generation. Horizontal comparison with other video models The table compares current API contracts, not image quality scores. Provider demos use different prompts, safety policies, resolutions, durations, sampling, and post-processing, so they cannot support a fair universal ranking. Model | Documented single-generation duration | Audio and control surface | Best fit | Seedance 2.5 | Up to 30 seconds | Joint audio-video; stronger reference interpretation and editing; professional white-model and green-screen controls | Longer directed scenes and production workflows that value continuity | Seedance 2.0 | Up to 15 seconds | Native stereo audio; up to 9 images, 3 videos, and 3 audio references in the full native workflow | Lower-cost Seedance work and clearly specified multimodal reference jobs | Veo 3.1 | 4, 6, or 8 seconds | Native audio always on; first/last frames, up to 3 reference images, and Veo-only extension; 720p, 1080p, or 4K under documented constraints | Short high-resolution shots and Google-native workflows | Kling 3.0 Pro | 3–15 seconds on the referenced API host | Optional native audio, multi-prompt shots, custom elements, and motion control | Multi-shot planning, element consistency, and motion-driven clips | Runway Gen-4.5 | 2–10 seconds | Text-to-video and image-to-video; the core generation endpoint does not expose native-audio generation | Teams already using Runway's creation and editing environment | Sora 2 | 4, 8, or 12 seconds | Synced audio and optional image reference; current OpenAI API catalog marks the model Legacy | Existing Sora integrations that have already evaluated the legacy lifecycle | Seedance 2.5 has the clearest native-duration advantage in this set. Veo 3.1 has a tightly documented resolution and extension contract, Kling 3.0 Pro exposes direct multi-shot and element controls, and Runway offers a broader production environment around shorter generations. Sora 2 remains useful context, but its current Legacy label makes it a weak default for a new long-lived integration. No specification table can tell you which model will preserve your actor, product, camera path, or dialogue best. Run the same licensed inputs and shot brief through the candidates, then compare usable-shot rate, revision count, wall time, and total cost—not only the first output. Which Seedance version should you choose? Choose seedance-2.5 when a continuous 16–30-second scene, reference interpretation, or later editing control is worth the price premium. Choose seedance-2.0 when the shot fits within 15 seconds, budget matters, or you want ByteDance's explicitly documented mixed-reference limits. Split a story into shorter shots when one late artifact would force an expensive long clip to be regenerated. Keep both model IDs behind the same application abstraction. Their Modelflare submission, polling, and download flow is the same, so controlled A/B routing does not require a second job system. Seedance pricing on Modelflare At the production check, both exact model IDs were available only in the public seedance-stable group through the openai-video protocol. The live effective prices below are in US dollars per generated second. Resolution tier | Seedance 2.0 | Seedance 2.5 | 2.5 premium | 480p | $0.12/s | $0.18/s | 50% | 720p | $0.24/s | $0.36/s | 50% | 1080p | $0.60/s | $0.90/s | 50% | 4K | $1.20/s | $1.80/s | 50% | A 10-second 720p request is therefore $2.40 on 2.0 or $3.60 on 2.5 before any future pricing change. A 30-second 720p 2.5 request is $10.80. The current catalog has a price row for all four tiers, but a price row is not a promise that every duration, resolution, input mode, or account combination will be accepted upstream. Use the live Seedance 2.5 pricing page (/pricing/seedance-2.5) and Seedance 2.0 pricing page (/pricing/seedance-2.0) before a batch. Modelflare's initial integration verification completed a 4-second 480p Seedance 2.5 image-to-video task and a matching Seedance 2.0 text-to-video task; both produced an approximately 4.064-second MP4 with video and two-channel audio. That proves those exact paths at the verification time, not a permanent service-level guarantee for every combination. Call Seedance 2.5 through Modelflare Seedance uses an asynchronous Video API. Submit a job, save the public task ID, poll until a terminal state, and then download the content. Do not send this request to /v1/chat/completions. Submit a text-to-video task export MODELFLARE_API_KEY="" curl https://modelflare.dev/v1/videos \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "seedance-2.5", "prompt": "A white cat runs through a neon-lit street in the rain, cinematic low-angle tracking shot, natural foot contact, synchronized city ambience", "duration": 4, "resolution": "720p" }' The response includes id and task_id for the same public task. Persist one of them with your application job record. Submit an image-to-video task Add one publicly reachable HTTPS image URL. This switches the current Modelflare surface to image-to-video. curl https://modelflare.dev/v1/videos \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "seedance-2.5", "prompt": "The subject turns slowly toward the camera while the camera moves forward; preserve identity, clothing, composition, and natural lighting", "image": "https://example.com/licensed-reference.png", "duration": 4, "resolution": "720p" }' The current public contract accepts one image field. Do not infer support here for ByteDance's full native multi-image, video-reference, audio-reference, white-model, green-screen, or editing controls until those fields are explicitly documented by Modelflare. Poll and download the result Poll every few seconds with bounded backoff. Continue through queued, pending, processing, or in_progress; stop at completed or failed. export TASK_ID="" curl "https://modelflare.dev/v1/videos/$TASK_ID" \ -H "Authorization: Bearer $MODELFLARE_API_KEY" After completion, download through the authenticated content endpoint instead of depending on an upstream storage URL: curl -L "https://modelflare.dev/v1/videos/$TASK_ID/content" \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -o seedance-2.5.mp4 The complete request and status contract is also maintained in the Seedance API setup document (/docs/seedance). Production checklist and limitations Use an API key eligible for seedance-stable and pin the exact model ID. Start with a 4-second 480p or 720p request before spending on a long, high-resolution clip. Validate that every source image is licensed, publicly reachable over HTTPS, and free of expiring query parameters during generation. Store your own job ID, Modelflare task ID, model, prompt version, requested duration, resolution, timestamps, final state, and recorded charge. Make submission retries idempotent in your application; repeating a timed-out POST can create and bill a second task. Poll with backoff and a deadline. Treat 429 as temporary capacity pressure, and do not retry invalid 400 parameters unchanged. Download and archive successful output according to your retention policy instead of assuming provider storage is permanent. Compare models on usable-shot cost. Include failed generations, discarded takes, editing time, and regeneration—not just dollars per generated second. AI video can still produce identity drift, malformed hands or text, inconsistent object permanence, physically implausible motion, and audio artifacts. Human review remains necessary before advertising, publishing, or using generated footage in a customer workflow. Bottom line Seedance 2.5 is the stronger Seedance candidate when a 30-second native scene and more deliberate reference or editing control reduce stitching and revision work. Seedance 2.0 remains compelling when 15 seconds is enough: on Modelflare it currently costs one-third less than 2.5, uses the same job protocol, and has a more explicit published mixed-reference contract at the model level. Against other video models, 2.5's differentiator is long native storytelling plus production-oriented control—not a proven win on every visual-quality dimension. Keep the integration asynchronous, test short and cheap first, and choose by usable outputs per dollar on your own licensed workload. Official sources and update log Primary and API sources: ByteDance Seed: Seedance 2.5 (https://seed.bytedance.com/en/seedance2_5) ByteDance Seed: Seedance 2.0 official launch (https://seed.bytedance.com/en/blog/seedance-2-0-official-launch) Google AI: Generate videos with Veo 3.1 (https://ai.google.dev/gemini-api/docs/video) OpenAI API: Sora 2 model (https://developers.openai.com/api/docs/models/sora-2) Runway API: available models and Gen-4.5 contract (https://docs.dev.runwayml.com/guides/models/) fal API: Kling 3.0 Pro contract (https://fal.ai/docs/model-api-reference/video-generation-api/kling-video-v3-pro) Update log: August 15, 2026: Initial publication. Official model pages, competitor API contracts, Modelflare model availability, effective prices, public fields, and the earlier real-task verification record were checked. ### Gemini 3.7 Flash Released: Specs, Benchmarks, Pricing, and Model Comparison Canonical URL: https://modelflare.dev/articles/gemini-3-7-flash-review-pricing Google released Gemini 3.7 Flash (https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-3-7-flash/) on August 13, 2026, only three weeks after Gemini 3.6 Flash. It is generally available under the stable API model ID gemini-3.7-flash and is positioned as Google's most capable Flash model for coding, agents, multimodal reasoning, and reliable multi-step execution. The headline is not a larger context window or a disclosed parameter count. Gemini 3.7 Flash keeps the 1M-input and 64K-output class of 3.6 Flash, while Google reports substantial gains in coding, tool use, knowledge work, document understanding, and computer use. Its introductory token price is also unchanged from 3.6 Flash. This analysis separates official specifications, Google-published benchmark results, temporary launch pricing, and current Modelflare availability. Volatile facts were checked on August 15, 2026. Gemini 3.7 Flash at a glance Property | Official value | Release date | August 13, 2026 | API status | Generally available; stable model ID | Model ID | gemini-3.7-flash | Foundation | Based on Gemini 3.6 Flash with algorithmic improvements | Input modalities | Text, image, video, audio, and PDF | Output modality | Text | Maximum input | 1,048,576 tokens | Maximum output | 65,536 tokens | Thinking levels | low, medium, high; medium is the documented default | Knowledge cutoff | March 2026, with some domains potentially limited to January 2025 | Parameter count | Not disclosed | Latest documented update | August 2026 | Supported capabilities include context caching, code execution, Computer Use in preview, file search, function calling, Google Maps grounding, Google Search grounding, structured outputs, thinking, and URL context. Batch, Flex, and Priority consumption modes are available. Audio generation, image generation, and the Live API are not supported by this model. What actually changed from Gemini 3.6 Flash Gemini 3.7 Flash is an iteration on 3.6 Flash rather than a new context or modality class. Google attributes the release to algorithmic improvements in the reasoning foundation and feedback from production developers. The practical changes are concentrated in: higher first-pass accuracy for code generation and software engineering; stronger design adherence when turning screenshots, images, or design systems into web interfaces; better multi-step planning, tool calls, recovery from roadblocks, and intent clarification; better reasoning over complex documents in finance, law, and biosciences; improved long-context retrieval and agentic computer use; the same introductory Standard price as Gemini 3.6 Flash through December 31, 2026. The last point matters. At launch, 3.7 is not a higher-priced replacement for 3.6. The migration decision is mainly about behavior, latency, and compatibility, not a token-price premium. Official API pricing The following paid-tier prices are in US dollars. Token prices are per 1 million tokens; cache storage is per 1 million tokens per hour. Consumption mode | Meter | Through Dec 31, 2026 | From Jan 1, 2027 | Standard | Input | $0.75 | $1.50 | Standard | Output, including thinking tokens | $3.75 | $7.50 | Standard | Cached input | $0.075 | $0.15 | Standard | Cache storage per hour | $0.50 | $1.00 | Batch | Input | $0.375 | $0.75 | Batch | Output, including thinking tokens | $1.875 | $3.75 | Batch | Cached input | $0.0375 | $0.075 | Flex | Input | $0.375 | $0.75 | Flex | Output, including thinking tokens | $1.875 | $3.75 | Flex | Cached input | $0.0375 | $0.075 | Priority | Input | $1.35 | $2.70 | Priority | Output, including thinking tokens | $6.75 | $13.50 | Priority | Cached input | $0.135 | $0.27 | Google Search grounding includes 5,000 free search requests per month shared across Gemini 3.x models, then costs $14 per 1,000 requests. Google Maps grounding uses a shared allowance of 5,000 prompts and then the same $14 per 1,000 search queries. At the introductory Standard rate, 100K input tokens plus 10K output tokens cost about $0.1125 before caching, tools, storage, or retries. One million cached input tokens plus 100K output tokens cost about $0.45, excluding cache storage. Both examples double after the introductory period. Full horizontal benchmark comparison Google's August 2026 model card compares Gemini 3.7 Flash with Gemini 3.6 Flash, Claude Sonnet 5, GPT-5.6 Terra, and Muse Spark 1.2. Higher is better for every row below. A dash means the model card did not publish a result. Benchmark | Gemini 3.7 Flash | Gemini 3.6 Flash | Claude Sonnet 5 | GPT-5.6 Terra | Muse Spark 1.2 | Artificial Analysis Intelligence Index | 56 | 52 | 55 | 57 | 57 | FrontierCode 1.1 Main | 43.6% | 34.4% | 42.7% | 41.3% | — | DeepSWE v1.1 | 65.3% | 48.6% | 53.8% | 69.6% | 54.9% | Code Arena, WebDev Elo | 1588 | 1538 | 1541 | 1523 | 1535 | Terminal-bench 2.1 | 85.8% | 78.0% | 80.4% | 87.4% | 82.9% | Terminal-bench 3.0 | 14.9% | 5.4% | 14.6% | 20.8% | — | AutomationBench, private set | 30.4% | 17.0% | 10.7% | 23.6% | — | GDPVal-AA v2, Elo | 1525 | 1422 | 1598 | 1578 | 1628 | Harvey LAB-AA | 90.7% | 85.1% | 90.1% | 85.2% | — | GDP.pdf | 34.0% | 22.0% | 28.0% | 24.7% | 16.0% | CharXiv Reasoning, no tools | 84.5% | 85.2% | 77.0% | 85.9% | — | CharXiv Reasoning, with tools | 88.7% | 89.4% | 88.3% | — | — | LVBench | 85.4% | 84.2% | 68.5% | 78.9% | — | GDM-MRCR v2, 128K average | 97.0% | 91.8% | 81.5% | 93.5% | — | OSWorld 2.0 | 47.9% | 33.8% | — | 50.2% | — | Agent's Last Exam | 26.3% | 24.2% | 33.3% | 28.0% | — | HLE-Verified | 53.6% | 51.2% | 31.0% | 51.1% | — | BioMysteryBench, human-solvable | 87.1% | 80.6% | 87.5% | 83.8% | — | BioMysteryBench, human-difficult | 43.5% | 41.2% | 34.1% | 49.4% | — | LABBench2 | 82.1% | 76.1% | 80.1% | 81.2% | — | These numbers are useful, but they are not one clean independent tournament. Google says Gemini scores are generally pass@1 with default sampling. Non-Gemini numbers are provider-reported unless the methodology says otherwise, and maximum available reasoning settings are preferred. Several evaluations were self-computed; harnesses, tool access, run aggregation, and even video frame limits vary by model. Capability and performance analysis Against Gemini 3.6 Flash, the upgrade is broad rather than cosmetic. FrontierCode rises 9.2 percentage points, DeepSWE 16.7 points, AutomationBench 13.4 points, GDP.pdf 12 points, OSWorld 14.1 points, and Code Arena gains 50 Elo. The long-context GDM-MRCR score rises from 91.8% to 97.0%. The comparison with other model families is more nuanced: Gemini 3.7 Flash leads the published table on FrontierCode, Code Arena, AutomationBench, Harvey LAB-AA, GDP.pdf, LVBench, GDM-MRCR, HLE-Verified, and LABBench2. GPT-5.6 Terra remains ahead on DeepSWE, both Terminal-bench rows, no-tool CharXiv, OSWorld, and the human-difficult BioMysteryBench split. Claude Sonnet 5 leads Agent's Last Exam and narrowly leads the human-solvable BioMysteryBench split. Muse Spark 1.2 leads GDPVal-AA and ties GPT-5.6 Terra on the Artificial Analysis Intelligence Index, but many rows are unreported. Gemini 3.7 is not uniformly better than 3.6: CharXiv is 0.7 points lower both without and with tools. The practical reading is that 3.7 Flash is unusually balanced for its launch price. It is a strong default candidate for coding agents, browser or desktop workflows, long documents, multimodal analysis, and high-volume knowledge work. GPT-5.6 Terra still deserves a direct test for long-horizon software engineering and terminal agents; Claude Sonnet 5 remains competitive for desktop-agent tasks; Muse Spark's knowledge-work result is strong but its comparison surface is incomplete. Parameters, context, and multimodality Google has not disclosed the total parameter count, active parameter count, expert count, architecture details, training corpus size, or compute budget. The official model card only states that Gemini 3.7 Flash is based on Gemini 3.6 Flash with algorithmic improvements. Any exact parameter estimate presented as fact would be speculation. The parameters developers can actually depend on are the API contract: 1,048,576 input tokens and 65,536 output tokens; text, image, video, audio, and PDF input with text output; low, medium, and high thinking levels; minimal returns an error; function calling, structured output, search, Maps, code execution, file search, URL context, caching, and preview Computer Use; no Live API, audio generation, or image generation on this model. A 1M context window is capacity, not a target. Long prompts still increase latency and cost, and retrieval quality does not eliminate the need to preserve authoritative state, compact repeated history, and verify tool side effects. Call Gemini 3.7 Flash Google's launch guide uses the Interactions API and documents medium as the default thinking level: export GEMINI_API_KEY="" curl "https://generativelanguage.googleapis.com/v1beta/interactions" \ -H "x-goog-api-key: $GEMINI_API_KEY" \ -H "Content-Type: application/json" \ -X POST \ -d '{ "model": "gemini-3.7-flash", "input": "Review this retrying payment workflow for race conditions and propose a safe transaction boundary.", "generation_config": { "thinking_level": "medium" } }' Modelflare currently exposes the exact model through its OpenAI-compatible Chat Completions route: export MODELFLARE_API_KEY="" curl "https://origin.modelflare.dev/v1/chat/completions" \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gemini-3.7-flash", "messages": [ { "role": "user", "content": "Review this retrying payment workflow for race conditions and propose a safe transaction boundary." } ] }' The two examples use different wire formats. Do not assume every Google-native built-in tool or Interactions field is available through an OpenAI-compatible adapter. Migration and production notes Google's 3.7 migration guide asks developers to remove temperature, top_p, and top_k, replace thinking_budget with the string enum thinking_level, and remove unsupported candidate_count. Multi-turn Interactions workflows should use server-side previous_interaction_id. Before replacing an existing model: pin the exact gemini-3.7-flash ID and reject silent alias substitution; replay a fixed, privacy-safe task set against the current model and 3.7; compare low, medium, and high thinking by successful-task cost and wall time; test streaming, structured output, function calling, multimodal payloads, cancellation, and refusal paths separately; preserve thought signatures and tool-call correlation fields required by the chosen API; verify cache usage, retries, timeout behavior, and duplicate-side-effect protection; record the final model, route, input, cached input, output, tools, latency, and charge; keep a rollback route until real traffic meets the acceptance threshold. An HTTP 200 proves one request completed. It does not prove protocol parity, stable latency, correct tool behavior, or lower cost per successful task. Gemini 3.7 Flash on Modelflare At the production check on August 15, 2026, Modelflare's public Models & Pricing (/pricing/gemini-3.7-flash) catalog listed the exact gemini-3.7-flash model ID in the gemini-stable group. The catalog reports support for the Gemini-native generateContent route and OpenAI-compatible Chat Completions. It does not list a Responses route for this model. The displayed group multiplier was 0.15x at that check. Google's introductory price, Modelflare's configured reference price, group access, and the final recorded request charge are separate facts and can change independently. Use the live pricing page and a completed non-sensitive request as the current source of truth. For production, create or update a key with an eligible group, use the exact model ID, and validate the protocol features your application actually uses before shifting traffic. Bottom line Gemini 3.7 Flash is a meaningful upgrade over 3.6 Flash at the same temporary launch price. Its strongest case is breadth: production coding, web development, tool-driven agents, complex documents, multimodal inputs, long context, and computer use all improve without moving into a premium price tier. It is not the winner on every benchmark, its parameter count is unknown, and the launch discount expires at the end of 2026. The sensible default is to benchmark 3.7 Flash against the model already handling your real tasks, then choose by completion rate, latency, retries, and total cost—not by one composite score. Official sources and update log Primary sources: Google: Introducing Gemini 3.7 Flash (https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-3-7-flash/) Google AI: Gemini 3.7 Flash model page (https://ai.google.dev/gemini-api/docs/models/gemini-3.7-flash) Google AI: latest model and migration guide (https://ai.google.dev/gemini-api/docs/latest-model) Google AI: Gemini API pricing (https://ai.google.dev/gemini-api/docs/pricing) Google DeepMind: Gemini 3.7 Flash model card (https://deepmind.google/models/model-cards/gemini-3-7-flash/) Google DeepMind: evaluation methodology (https://deepmind.com/models/evals-methodology/gemini-3-7-flash) Update log: August 15, 2026: Initial publication. Specifications, capabilities, benchmark methodology, official prices, migration guidance, and Modelflare catalog availability checked. ### DeepSeek V4 Pro GA Review: Benchmarks, API Pricing, and Cost Analysis Canonical URL: https://modelflare.dev/articles/deepseek-v4-pro-review-pricing DeepSeek released DeepSeek-V4-Pro GA (https://api-docs.deepseek.com/news/news260813) on August 13, 2026. The hosted API now identifies the version as DeepSeek-V4-Pro-0813, while the stable API model ID remains deepseek-v4-pro. It brings a 1-million-token context window, up to 384,000 output tokens, native Responses API support, and a new price schedule that starts shortly after launch. Our verdict: V4 Pro deserves a production evaluation for difficult coding, repository-scale work, and long-running agents. DeepSeek reports consistent gains over V4 Flash on hard agent benchmarks, but the Pro model is not the economical default for every request. Its official input and output prices are roughly three times Flash after the new schedule begins, and its account concurrency limit is one fifth as large. This review separates current API facts, DeepSeek-published benchmark results, our analysis, and current Modelflare availability. We did not independently reproduce DeepSeek's benchmark scores. Volatile specifications and prices were checked on August 14, 2026. DeepSeek V4 Pro GA at a glance Item | Current official information | GA date | August 13, 2026 | Stable API model ID | deepseek-v4-pro | Hosted model version | DeepSeek-V4-Pro-0813 | Context window | 1,000,000 tokens | Maximum output | 384,000 tokens | Thinking mode | Supported and enabled by default | Reasoning effort | GA announcement: low, high, max; see the compatibility note below | Official API formats | Chat Completions, Responses API, Anthropic Messages-compatible API | Documented features | JSON output, tool calls, automatic context caching, prefix completion, FIM in non-thinking mode | Official account concurrency | 500 concurrent requests | The million-token limit is a capacity ceiling, not a quality guarantee. Retrieval accuracy, latency, cache reuse, output length, client timeouts, and the amount of irrelevant context still determine whether a long request is useful. What changed from the V4 preview The April preview introduced the V4 family and its open-weight architecture. The August GA update is a hosted product update with three practical changes: DeepSeek says agent performance improved substantially, especially in production-style coding and automation tasks; the official API now supports the OpenAI Responses format natively and includes Codex-oriented compatibility; DeepSeek announced peak and off-peak API prices effective at 16:00 UTC on August 16, 2026. The stable request model remains deepseek-v4-pro, so existing clients do not need a model-name migration. The pricing page exposes the current hosted version as DeepSeek-V4-Pro-0813. Do not automatically treat the downloadable preview checkpoint as bit-for-bit identical to the hosted GA version. DeepSeek's public V4 model card still describes the preview family and does not label the checkpoint 0813. Self-hosted and official-API deployments therefore need separate evaluations. Benchmark review: where Pro gains over Flash DeepSeek published the following GA V4 Pro results. The comparison column uses DeepSeek's July 31 V4 Flash update values, and the delta is an absolute score difference. Benchmark | V4 Pro GA | V4 Flash 0731 | Pro delta | Terminal Bench 2.1 | 87.9 | 82.7 | +5.2 | NL2Repo | 61.5 | 54.2 | +7.3 | Cybergym | 83.3 | 76.7 | +6.6 | DeepSWE | 62.7 | 54.4 | +8.3 | Toolathlon-Verified | 74.1 | 70.3 | +3.8 | Agents' Last Exam | 25.7 | 25.2 | +0.5 | AutomationBench (Public) | 31.8 | 25.1 | +6.7 | DSBench-FullStack | 71.1 | 68.7 | +2.4 | DSBench-Hard | 67.2 | 59.6 | +7.6 | DeepSeek also reports HLE scores of 42.7 without tools and 60.0 with tools for V4 Pro GA. The pattern matters more than one headline score. The largest absolute gains appear in repository work, difficult software-engineering tasks, and automation. The gap is much smaller on Agents' Last Exam and moderate on broad tool use. That supports a routed strategy: reserve Pro for tasks where a better chance of completion is worth more than raw token efficiency, and keep Flash for high-volume classification, summarization, triage, and simpler subagents. These are provider-reported results, not an independent Modelflare benchmark. Harnesses, tool permissions, reasoning effort, sampling settings, time limits, and scoring rules can materially change agent results. A production decision should replay the same privacy-safe tasks on both models and measure successful-task cost, not just tokens or benchmark rank. Architecture and the 1M context claim DeepSeek's public V4 model card describes the Pro family as a Mixture-of-Experts model with 1.6 trillion total parameters and 49 billion activated parameters. It documents a hybrid attention design combining Compressed Sparse Attention and Heavily Compressed Attention, Manifold-Constrained Hyper-Connections, the Muon optimizer, and pretraining on more than 32 trillion tokens. DeepSeek says that at a 1M-token context, V4 Pro uses 27% of the single-token inference FLOPs and 10% of the KV cache required by V3.2. Those are architecture-level claims from the model publisher; they do not mean a one-million-token request will be fast, cheap, or equally accurate at every position. For real long-context work: put stable instructions and reusable repository material at the beginning so prefix caching can help; track prompt_cache_hit_tokens and prompt_cache_miss_tokens instead of assuming a hit; compact stale tool traces and duplicate files before they consume the context budget; test retrieval at the actual document depth and token position your application uses; reserve room for reasoning, tool results, and the final answer rather than filling the entire window with input. DeepSeek's disk cache is automatic and best-effort. It matches reusable prefixes, may take seconds to construct, and is usually cleared after hours to days of inactivity. A repeated request with changed early content can lose most of the economic benefit. API compatibility: native does not mean identical Surface | Official DeepSeek support | Important boundary | Chat Completions | Supported | Thinking tool loops must preserve reasoning_content | Responses API | Supported | Stateless; several OpenAI fields are ignored or unsupported | Anthropic-compatible API | Supported | Some fields are ignored; image and document content are not supported | FIM completion | Beta | Non-thinking mode only and uses the beta endpoint | The official Responses layer supports text, function calls, server-side web search, and the apply_patch custom tool used for Codex compatibility. It does not support previous_response_id, conversations, stored responses, background mode, service_tier, or OpenAI-managed prompt-cache keys. Unsupported fields are often silently ignored, so an HTTP 200 does not prove semantic parity. Responses clients must carry their own history and inspect event types rather than wait for a Chat Completions-style [DONE] marker. There is another important tool-loop boundary in Chat Completions: when thinking mode and tools are used together, the assistant's reasoning_content must be passed back in subsequent requests. DeepSeek documents a 400 error when this field is lost. Gateways and SDK adapters should be tested with a complete multi-turn tool loop, not a single text prompt. The GA announcement says low, high, and max reasoning effort are available. However, the current detailed thinking guide and API reference say high and max are the effective values, with low and medium mapped to high and xhigh mapped to max. Until DeepSeek aligns those documents or a live test proves distinct low behavior, do not budget around an assumed low-effort cost reduction. DeepSeek official API pricing: current and upcoming All prices below are USD per 1 million tokens. The current fixed prices remain listed until the new schedule takes effect at 16:00 UTC on August 16, 2026, which is 00:00 on August 17 in Beijing time. Price period | Cache-hit input | Cache-miss input | Output | Current price checked Aug 14 | $0.003625 | $0.435 | $0.87 | New off-peak price | $0.022 | $0.66 | $1.98 | New peak price | $0.044 | $1.32 | $3.96 | Under the upcoming schedule, peak hours are 01:00–04:00 and 06:00–10:00 UTC. All other hours are off-peak, and off-peak prices are half the peak rates. The transition is not a simple 2× time-of-day split from today's prices. Compared with the current fixed rate, the new off-peak price is about 6.07× higher for cached input, 1.52× for uncached input, and 2.28× for output. Peak pricing doubles those new off-peak values. Cache reuse remains valuable, but the exceptionally low launch-era cached-input price changes the most. Worked cost examples The token-only formula is: cost = cache_hit_tokens × cache_hit_rate + cache_miss_tokens × cache_miss_rate + output_tokens × output_rate The following examples exclude any separate network, subscription, tool-service, or payment costs. A tool-using agent may generate several model calls, so calculate the whole task rather than only the first request. Workload | DeepSeek current | New off-peak | New peak | Modelflare Aug 14 snapshot | 100K uncached input + 10K output | $0.0522 | $0.0858 | $0.1716 | $0.0540 | 90K cached + 10K uncached input + 10K output | $0.0134 | $0.0284 | $0.0568 | $0.0138 | 900K cached + 100K uncached input + 20K output | $0.0642 | $0.1254 | $0.2508 | $0.0663 | The second and third rows show why prompt structure matters. A large stable prefix can cut cost dramatically, but only after a real cache hit. The API usage fields are the source of truth for reconciliation. DeepSeek V4 Pro vs V4 Flash: price-performance choice At the current official rates, Pro costs about 3.11× Flash for cache-miss input and output, but only 1.29× for cache-hit input. After the new schedule starts, Pro is 3× Flash for uncached input and output and about 3.14× for cached input. Official account concurrency is 500 for Pro versus 2,500 for Flash. That creates a clear operating rule: choose V4 Pro for difficult repository changes, long-horizon agents, security analysis, complex tool coordination, and tasks where one failed attempt is expensive; choose V4 Flash for simpler agents, parallel workers, preprocessing, extraction, summarization, and throughput-sensitive traffic; route by observed task difficulty and successful-task cost rather than making Pro the universal default; reevaluate after the price transition because the economics of cached Pro workloads change materially. A 3× token price can still be cheaper if Pro avoids retries or human repair. Conversely, a five-point benchmark advantage does not justify sending deterministic or trivial work to the larger model. Measure completion rate, wall time, total tokens across retries, tool failures, and reviewer correction time. DeepSeek V4 Pro price and availability on Modelflare The live Modelflare pricing catalog (https://modelflare.dev/pricing/deepseek-v4-pro) was checked on August 14, 2026. It lists deepseek-v4-pro and the pinned alias deepseek-v4-pro-0813 in the deepseek-stable group with the following base prices per 1 million tokens: Modelflare price snapshot | Price | Cache-hit input | $0.0037 | Cache-miss input | $0.45 | Output | $0.90 | Those snapshot rates are about 2.07% above DeepSeek's current cache-hit rate and 3.45% above its current cache-miss and output rates. Modelflare has not published a promise in this article that these prices will remain unchanged after DeepSeek's peak/off-peak transition. Always use the live pricing page and the request usage record for a production decision. Current public catalog metadata marks the Modelflare route as OpenAI-compatible Chat Completions. Although DeepSeek's own endpoint now supports Responses and Anthropic formats, provider support does not automatically make every gateway route compatible. Use the exact protocol shown on the live price page. export MODELFLARE_API_KEY="your-api-key" curl https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer ${MODELFLARE_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-v4-pro", "messages": [ {"role": "user", "content": "Review this migration plan and identify rollback risks."} ], "thinking": {"type": "enabled"}, "reasoning_effort": "high" }' For general client setup, see the OpenAI-compatible API guide (/articles/openai-compatible-api-guide). For accounting and cache-token interpretation, see AI API cost tracking (/articles/ai-api-cost-tracking). Production evaluation checklist Pin deepseek-v4-pro-0813 during evaluation if the route offers it, then decide whether the stable alias is acceptable for change control. Replay a fixed, privacy-safe task set on Pro and Flash with the same tools, permissions, timeout, and stopping rules. Test both high and max; treat low as equivalent to high until current API behavior proves otherwise. Complete a multi-turn tool loop and verify that reasoning_content, tool IDs, arguments, and results survive every adapter. For Responses clients, explicitly test every required field because unsupported parameters can be silently ignored. Record cache-hit input, cache-miss input, output, reasoning tokens, latency, retries, and successful-task cost. Test long prompts at realistic depths; do not infer one-million-token retrieval quality from a short request. Load-test below the account concurrency limit and handle 429 responses with bounded backoff. Model both the current price and the August 16/17 peak-off-peak schedule before approving a budget. Keep a Flash or alternative-model fallback for availability and cost control; see reliable AI API routing (/articles/reliable-ai-api-routing). Final verdict DeepSeek V4 Pro GA is a meaningful agent-focused release, not merely a renamed preview. The official score pattern is strongest on repository reasoning, difficult coding, and automation, while the new Responses surface makes it more practical for modern coding agents. Its 1M context and automatic caching are useful, but only when applications preserve stable prefixes and verify actual cache hits. The main reservation is economic: Pro's post-launch price is roughly three times Flash for most token categories, with lower concurrency. It should be the escalation tier for hard, high-value work rather than the default for every prompt. The best production architecture will usually route simple work to Flash and promote only difficult or failed tasks to Pro. Modelflare's August 14 price snapshot is close to DeepSeek's current fixed price and currently exposes Chat Completions for V4 Pro. Recheck the live page after DeepSeek's new schedule begins; neither an old article nor a cached pricing screenshot should be treated as billing truth. Sources and verification date DeepSeek: V4 Pro GA release (https://api-docs.deepseek.com/news/news260813) DeepSeek API models and pricing (https://api-docs.deepseek.com/quick_start/pricing) DeepSeek API change log (https://api-docs.deepseek.com/updates) DeepSeek thinking mode (https://api-docs.deepseek.com/guides/thinking_mode) DeepSeek Responses API compatibility (https://api-docs.deepseek.com/guides/responses_api) DeepSeek Anthropic API compatibility (https://api-docs.deepseek.com/guides/anthropic_api) DeepSeek context caching (https://api-docs.deepseek.com/guides/kv_cache) DeepSeek rate limits and isolation (https://api-docs.deepseek.com/quick_start/rate_limit) DeepSeek V4 Pro model card and open weights (https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) Official specifications, price schedules, Modelflare availability, and live prices were checked on August 14, 2026. ### Grok 4.6 Released: API, Pricing, 500K Context, and Developer Guide Canonical URL: https://modelflare.dev/articles/grok-4-6-api-guide xAI released Grok 4.6 (https://x.ai/news/grok-4-6) on August 12, 2026. The official developer guide (https://docs.x.ai/developers/grok-4-6) lists the API model ID as grok-4.6, with a 500,000-token context window, text and image input, text output, and four reasoning-effort levels: low, medium, high (default), and xhigh. For developers, the most important details are not just the launch scorecards. Grok 4.6 works with both the Responses API and Chat Completions, has a long-context pricing boundary at 200,000 prompt tokens, and requires deliberate cache affinity and context management for long-running agents. This guide separates official xAI specifications from provider-reported benchmark claims and from current Modelflare availability. Prices and availability can change; all volatile facts below were checked on August 13, 2026. Grok 4.6 at a glance Property | Official value | Release date | August 12, 2026 | API model ID | grok-4.6 | Context window | 500,000 tokens | Knowledge cutoff | February 1, 2026 | Modalities | Text and image input; text output | Text output limit | xAI lists no fixed text output limit | Reasoning effort | low, medium, high (default), xhigh | API formats | Responses API and Chat Completions | Documented capabilities | Function calling, structured outputs, web search, X search, code execution | “No text output limit” is the model-page description, not a promise that every SDK, gateway, account, or request can return unlimited text. Client timeouts, account limits, route policy, and available context still apply. What changed from Grok 4.5 xAI positions Grok 4.6 as an incremental frontier-model release focused on long-running agents and more ambitious interactive and visual work. Its release post describes longer supplemental training, regenerated supervised fine-tuning trajectories, and agentic reinforcement learning across knowledge work, coding, web development, kernel optimization, and CAD. The practical changes to evaluate are: xhigh joins low, medium, and high reasoning effort; xAI reports better continuity across multi-step research, codebase work, and iterative application building; the model accepts image input as well as text and returns text; the same 500K context class and standard $2 input / $6 output headline rates remain, but Grok 4.6 cached-input pricing is $0.50 per million tokens rather than Grok 4.5’s currently listed $0.30; xAI recommends conversation-level cache affinity and context compaction for sustained agent loops. These are reasons to run a controlled migration, not reasons to replace Grok 4.5 blindly. Compare successful task completion, latency, total tokens, tool calls, and cost on your own workload. Grok 4.6 API pricing xAI’s pricing page (https://docs.x.ai/developers/pricing) lists the following standard US-dollar prices per one million tokens: Prompt size | Input | Cached input | Output | Below 200K prompt tokens | $2.00 | $0.50 | $6.00 | At or above 200K prompt tokens | $4.00 | $1.00 | $12.00 | Once the prompt reaches 200K tokens, xAI says the long-context rates apply to all tokens in that request. Three quick calculations make the boundary concrete: 100K input plus 5K output at standard short-context rates costs about $0.23. If all 100K input tokens are served from cache, the same output costs about $0.08. 220K input plus 10K output crosses the threshold and costs about $1.00 at the listed long-context rates. These examples exclude server-side tools. xAI currently lists web search, X search, and code execution at $5 per 1,000 calls in addition to token usage. Its launch post also mentions a fast variant priced at twice the standard rate. Confirm the exact model, service tier, regional availability, and live price in the xAI console before budgeting. For a broader cost method, see AI API Cost Tracking (/articles/ai-api-cost-tracking). Call Grok 4.6 with Responses or Chat Completions The following requests use xAI’s official API endpoint, because this article is a release guide—not a claim that the model is already enabled on every gateway. Responses API: export XAI_API_KEY="" curl -sS https://api.x.ai/v1/responses \ -H "Authorization: Bearer $XAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "grok-4.6", "input": "Review this migration plan, identify the three highest-risk assumptions, and propose verification steps.", "prompt_cache_key": "grok-46-migration-review" }' Chat Completions: curl -sS https://api.x.ai/v1/chat/completions \ -H "Authorization: Bearer $XAI_API_KEY" \ -H "Content-Type: application/json" \ -H "x-grok-conv-id: grok-46-migration-review" \ -d '{ "model": "grok-4.6", "messages": [ { "role": "user", "content": "Review this migration plan, identify the three highest-risk assumptions, and propose verification steps." } ] }' Use prompt_cache_key with Responses or the x-grok-conv-id header with Chat Completions to improve cache affinity for a conversation. Treat those identifiers as operational metadata: keep them stable for one conversation, avoid secrets or personal data, and do not reuse one value across unrelated users. If you are deciding between the two wire formats, read Responses API vs Chat Completions (/articles/responses-api-vs-chat-completions). Prompt caching and long agent loops A 500K context window is capacity, not a target. Sending the full history on every turn can increase latency and can push the entire request into the long-context price tier. For sustained workflows: keep stable instructions and reusable context at the beginning of the prompt; use a stable cache-affinity key per conversation; measure cached and uncached input separately; compact or summarize old turns before the prompt approaches 200K tokens; retain tool results that are still authoritative and discard duplicated transport noise; set an explicit reasoning level and compare task-level success rather than assuming xhigh is always better; record status, latency, input, cached input, output, tool calls, and total cost per successful task. Context compaction changes the information the model sees. Validate summaries against the original state and preserve authoritative IDs, constraints, decisions, and unresolved errors. Read launch benchmarks with the right caveat xAI published the following Grok 4.6 High results in its launch announcement (https://x.ai/news/grok-4-6): Evaluation | xAI-reported Grok 4.6 score | Artificial Analysis Intelligence Index | 61 | GDPVal-AA v2 | 1753 | CursorBench v3.2 | 69.9% | DeepSWE v1.1 | 65.9% | FrontierCode v1.1 Extended | 61.3% | APEX-Agents | 57.5% | These are provider-published launch results, not independent Modelflare tests. Benchmark harnesses, reasoning settings, tool access, time limits, and scoring versions affect comparability. Use the table to identify workloads worth testing; do not convert it into a universal quality ranking or a guaranteed production outcome. Production migration checklist Before routing production traffic to Grok 4.6: use the exact grok-4.6 model ID and reject silent alias substitution; confirm the model is enabled for the intended account, region, API format, and service tier; replay a fixed, privacy-safe evaluation set on Grok 4.5 and Grok 4.6; test text input, image input, function calling, structured outputs, streaming, refusal paths, and cancellation separately where your application uses them; compare low, medium, high, and xhigh on successful-task cost and latency; test requests immediately below and above the 200K prompt threshold; verify cache hits, context compaction, tool-call counts, retry behavior, and duplicate-side-effect protection; keep a rollback route and log the final model, route, usage, latency, and charge for each request. An HTTP 200 proves that one request completed. It does not prove feature parity, stable latency, correct tool behavior, or acceptable cost. For route-level testing, use the OpenAI-Compatible API Conformance Tests (/articles/openai-compatible-api-conformance-tests) and Reliable AI API Routing (/articles/reliable-ai-api-routing). Is Grok 4.6 available on Modelflare? Yes. At the production check on August 13, 2026, Modelflare’s public Models & Pricing (/pricing) catalog listed the exact grok-4.6 model ID in the grok-award and grok-stable groups. The catalog reports support for OpenAI-compatible Chat Completions and Responses API routes. The group multipliers displayed at that check were 0.03x for grok-award and 0.06x for grok-stable. Group access, routes, prices, and availability can change. Because xAI also applies a separate long-context price tier, do not extend one headline multiplier or short-context example to every request. Treat the live pricing page and the completed request record as the current source of truth. Create or update an API key with an eligible group, use the exact grok-4.6 ID, and send one real, non-sensitive request before moving production traffic. A catalog listing confirms the configured public surface; it is not an independent quality benchmark or a guarantee for every API key. Official sources and update log Primary sources: xAI: Introducing Grok 4.6 (https://x.ai/news/grok-4-6) xAI developer guide: Grok 4.6 (https://docs.x.ai/developers/grok-4-6) xAI model page: grok-4.6 (https://docs.x.ai/developers/models/grok-4.6) xAI API pricing (https://docs.x.ai/developers/pricing) xAI release notes (https://docs.x.ai/developers/release-notes) Update log: August 13, 2026: Initial publication. Official specifications, pricing, API formats, launch benchmarks, and Modelflare catalog availability checked. ### Function Calling: Responses API vs Chat Completions Canonical URL: https://modelflare.dev/articles/function-calling-responses-vs-chat-completions Responses and Chat Completions can both ask an application to run a function, but they represent the tool loop differently. Chat Completions puts tool definitions under tools[].function, returns message.tool_calls, and accepts results as role: "tool" messages. Responses uses flat function definitions, returns typed function_call output items, and accepts function_call_output items linked by call_id. The model does not execute your application function. Your code must validate the proposed arguments, authorize the operation, execute it exactly as intended, return a result to the model, and prevent duplicate side effects when a request is retried. The tool loop has four steps The endpoint-specific JSON differs, but the application state machine is the same: 1. Declare an allowed function and its argument schema 2. Receive one or more model-proposed function calls 3. Validate, authorize, and execute each call in application code 4. Return each result using the call's correlation ID Only after step four can the model produce an answer grounded in the tool result. If the model requests another tool, the loop repeats with a new call ID. The OpenAI Function Calling guide (https://developers.openai.com/api/docs/guides/function-calling) describes this as a multi-step exchange. Treating the first tool call as a final answer is a common integration error. Compare the wire contracts The main differences are structural rather than conceptual. Concern | Responses API | Chat Completions | Function definition | Flat tools[] item with name, description, parameters, strict | tools[] item with those fields nested under function | Model-proposed call | Typed function_call output item | Assistant message tool_calls[] entry | Correlation identifier | call_id | Tool call id, returned as tool_call_id | Function name | function_call.name | tool_calls[].function.name | Arguments | JSON string in function_call.arguments | JSON string in tool_calls[].function.arguments | Tool result | function_call_output input item | Message with role: "tool" | Common streamed arguments | Typed function-argument delta events | Indexed delta.tool_calls[] fragments | Final text | Typed output items and output-text helpers | choices[0].message.content | Do not correlate calls by array position. Parallel calls and streamed chunks can arrive in an order that differs from the order in which your application finishes. The explicit call ID is the stable join key. Define one strict function schema The following examples expose a read-only get_delivery_status function. The model may request an order status, but it cannot change an order, issue a refund, or access arbitrary storage. The argument schema is deliberately narrow: { "type": "object", "properties": { "order_id": { "type": "string", "pattern": "^ORDER-[0-9]{4}$" } }, "required": ["order_id"], "additionalProperties": false } strict: true asks the supported model to adhere to the function argument schema. The application must still parse and validate the JSON before execution. Pattern support and other JSON Schema features can vary by model or provider, so the same compatibility gate used for Structured Outputs (/articles/structured-outputs-openai-compatible-api) also applies to function arguments. Implement the loop with the Responses API This JavaScript example keeps the exchange stateless by returning the model’s output items with the function result. It uses a deterministic local function so the control flow is visible without introducing another external API. import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.MODELFLARE_API_KEY, baseURL: "https://modelflare.dev/v1", }); const model = process.env.MODEL_ID; if (!model) throw new Error("MODEL_ID is required"); const orders = new Map([ ["ORDER-1001", { status: "in_transit", eta: "2026-08-06" }], ]); const tools = [ { type: "function", name: "get_delivery_status", description: "Return the current delivery status for one order.", parameters: { type: "object", properties: { order_id: { type: "string", pattern: "^ORDER-[0-9]{4}$" }, }, required: ["order_id"], additionalProperties: false, }, strict: true, }, ]; function executeTool(name, rawArguments) { if (name !== "get_delivery_status") { throw new Error(`Tool is not allowed: ${name}`); } const args = JSON.parse(rawArguments); if (!/^ORDER-[0-9]{4}$/.test(args.order_id)) { throw new Error("Invalid order_id"); } return orders.get(args.order_id) ?? { status: "not_found" }; } const first = await client.responses.create({ model, input: "Where is ORDER-1001?", tools, parallel_tool_calls: false, }); const calls = first.output.filter((item) => item.type === "function_call"); if (calls.length === 0) { console.log(first.output_text); process.exit(0); } const toolOutputs = calls.map((call) => ({ type: "function_call_output", call_id: call.call_id, output: JSON.stringify(executeTool(call.name, call.arguments)), })); const final = await client.responses.create({ model, input: [...first.output, ...toolOutputs], tools, parallel_tool_calls: false, }); console.log(final.output_text); For a stateful integration, Responses also supports continuing from a prior response where the route and storage policy permit it. A stateless example makes the complete transcript explicit and avoids assuming that every OpenAI-compatible route stores response state. The critical fields are call.call_id on the model output and the matching call_id on function_call_output. Returning a result with the wrong ID disconnects the data from the model’s request. Implement the loop with Chat Completions The same application function can be used with Chat Completions. The tool definition and result message use the Chat wrapper. import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.MODELFLARE_API_KEY, baseURL: "https://modelflare.dev/v1", }); const model = process.env.MODEL_ID; if (!model) throw new Error("MODEL_ID is required"); const orders = new Map([ ["ORDER-1001", { status: "in_transit", eta: "2026-08-06" }], ]); const tools = [ { type: "function", function: { name: "get_delivery_status", description: "Return the current delivery status for one order.", parameters: { type: "object", properties: { order_id: { type: "string", pattern: "^ORDER-[0-9]{4}$" }, }, required: ["order_id"], additionalProperties: false, }, strict: true, }, }, ]; function executeTool(name, rawArguments) { if (name !== "get_delivery_status") { throw new Error(`Tool is not allowed: ${name}`); } const args = JSON.parse(rawArguments); if (!/^ORDER-[0-9]{4}$/.test(args.order_id)) { throw new Error("Invalid order_id"); } return orders.get(args.order_id) ?? { status: "not_found" }; } const messages = [ { role: "user", content: "Where is ORDER-1001?" }, ]; const first = await client.chat.completions.create({ model, messages, tools, parallel_tool_calls: false, }); const assistant = first.choices[0].message; const calls = assistant.tool_calls ?? []; if (calls.length === 0) { console.log(assistant.content); process.exit(0); } messages.push(assistant); for (const call of calls) { const result = executeTool(call.function.name, call.function.arguments); messages.push({ role: "tool", tool_call_id: call.id, content: JSON.stringify(result), }); } const final = await client.chat.completions.create({ model, messages, tools, parallel_tool_calls: false, }); console.log(final.choices[0].message.content); Here, the first assistant message must be appended to messages before its tool results. Each result uses tool_call_id: call.id. Omitting the assistant tool-call message or returning an unmatched ID produces an invalid conversation history. Stream arguments as fragments, not complete JSON In streaming mode, function arguments can arrive in fragments. A chunk that contains {"order is not invalid JSON; it is incomplete JSON. The safe parser keeps one buffer per call identity: identify the call or indexed call slot from the event; append each argument delta to that call’s buffer; wait for the endpoint’s argument-done or completed signal; parse the complete string once; validate and execute only after completion. Responses uses typed events for function-call arguments and output items. Chat Completions emits fragments in choices[].delta.tool_calls[], where the index identifies the in-progress entry and the final tool call carries its stable ID. Do not execute a function when the first fragment arrives. The AI API Streaming Guide (/articles/ai-api-streaming-guide) explains why connection, first event, first effective output, idle timeout, and completion should be measured separately. Make tool execution safe and idempotent Strict arguments improve structure, not authorization. Before executing any call: allow only registered function names; parse arguments with a size limit; validate the complete schema in application code; authorize the current user or workload for the requested resource; separate read-only tools from tools with side effects; redact secrets and sensitive output before returning it to the model; use a deadline and bounded downstream retries; record a safe operation reference, not raw secrets or private content. For a side-effecting tool, use an idempotency key derived from a stable application request and call identity. Store the completed result before returning it to the model. If a network retry replays the same call, return the stored result rather than issuing a second refund, message, deployment, or database mutation. The model-generated function name and arguments are untrusted input. A valid schema cannot grant permission, prove ownership, or replace a transaction boundary. Decide how to handle multiple calls The examples set parallel_tool_calls: false to keep the state machine easy to inspect. If parallel calls are enabled: correlate every result by call ID, never by completion order; enforce a maximum number of calls per response; apply a concurrency limit in the application; define whether one failed tool cancels, blocks, or coexists with successful tools; return one result for every accepted call; keep side-effecting operations serialized when their order matters. Parallel execution can reduce latency for independent reads, but it increases authorization, ordering, retry, and partial-failure complexity. Enable it because the workload benefits, not because the parameter exists. Understand the Modelflare compatibility boundary Modelflare’s OpenAI compatibility code preserves strict function definitions and maps the supported function-call shapes between Responses and Chat Completions on eligible OpenAI/Codex routes. That mapping is deliberately narrower than the complete Responses tool surface. Application-defined function tools can be represented in both formats. Responses-only hosted tools—such as provider-run search, code execution, or other built-in tool types—are not equivalent to application function calling and cannot be assumed to convert into Chat Completions. Other OpenAI-compatible model families are exposed through raw Chat Completions pass-through unless separately verified. For those routes, the upstream provider decides whether tools, strict, parallel calls, streaming argument fragments, and specific tool_choice shapes are supported. Before enabling a route, run the complete loop rather than checking only whether the first request returns 200: Test | Required evidence | One read-only call | Function name, parsed arguments, call ID, result linkage, final text | Invalid arguments | Explicit validation failure with no tool execution | Unknown function | Rejected by the application allowlist | No tool required | Normal final text without a fabricated call | Streaming call | Complete reconstructed arguments and correct call ID | Repeated request | One side effect or one cached result, not duplicate execution | Multiple calls | Correct correlation independent of completion order | Route fallback | Same model, protocol, tool schema, and result contract preserved | Use Reliable AI API Routing (/articles/reliable-ai-api-routing) to keep route changes explicit, and AI API Key Security (/articles/api-key-security-cost-controls) to separate workloads and permissions. Choose the format by the application contract Use Responses when the application benefits from typed output items, the Responses event model, state-continuation options, or other verified Responses capabilities. Use Chat Completions when the application already owns a stable message transcript and the selected provider’s tool contract is verified. The broader endpoint decision is covered in Responses API vs Chat Completions (/articles/responses-api-vs-chat-completions). The implementation standard should be the same in both cases: explicit tool allowlists, strict argument schemas, application-side validation, authorization before execution, correlation by call ID, idempotency for side effects, and a complete result returned to the model. Those controls—not the endpoint name—make a function-calling workflow reliable. ### Structured Outputs with OpenAI-Compatible APIs: A JSON Schema Guide Canonical URL: https://modelflare.dev/articles/structured-outputs-openai-compatible-api Structured Outputs let an application request JSON that follows a defined schema instead of merely asking the model to “return JSON.” With an OpenAI-compatible API, the same schema can be used through Responses or Chat Completions, but the request fields are different—and compatibility must be verified for the selected model and route. The safe production pattern has three layers: use strict schema output when the model supports it, parse and validate the returned JSON in the application, and then apply business rules to the values. Schema compliance removes many formatting failures; it does not prove that the model’s answer is factually or semantically correct. Structured Outputs are not the same as JSON mode There are three common ways to ask a model for JSON-like output. Method | Produces valid JSON | Enforces your schema | Typical use | Prompt only | Not guaranteed | No | Prototypes where parsing failures are acceptable | JSON mode | Yes, when supported and completed | No | Flexible JSON where the application validates the shape | Structured Outputs with strict: true | Yes, subject to completion and refusal handling | Yes, within the supported JSON Schema subset | Typed extraction and application workflows | The OpenAI Structured Outputs guide (https://developers.openai.com/api/docs/guides/structured-outputs) recommends Structured Outputs instead of JSON mode when the selected model and endpoint support it. The same guide also distinguishes two different jobs: Use a structured response format when the model should answer the user with a predictable data object. Use function calling (https://developers.openai.com/api/docs/guides/function-calling) when the model should request an action from your application. This article focuses on the first job. A later tool loop can reuse strict argument schemas, but its call IDs and result messages follow a different contract. Define one schema before choosing the endpoint Consider a support-triage workflow. The application needs four fields: category: one of billing, technical, or account; priority: an integer from 1 through 3; requires_human: a boolean; summary: a short description for the support queue. The JSON Schema is: { "type": "object", "properties": { "category": { "type": "string", "enum": ["billing", "technical", "account"] }, "priority": { "type": "integer", "minimum": 1, "maximum": 3 }, "requires_human": { "type": "boolean" }, "summary": { "type": "string" } }, "required": ["category", "priority", "requires_human", "summary"], "additionalProperties": false } Making every property required and setting additionalProperties to false produces a stable object for downstream code. It does not make every possible JSON Schema keyword portable. OpenAI supports a documented subset, and another OpenAI-compatible provider may support a different subset or no strict schema mode at all. Send the schema with the Responses API The Responses API places the response schema under text.format. Set a Modelflare API key and choose a model that is currently listed for your API key and has been verified for Structured Outputs. export MODELFLARE_API_KEY="" export MODEL_ID="" curl -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d "{ \"model\": \"$MODEL_ID\", \"input\": \"The customer was charged twice and wants a refund.\", \"text\": { \"format\": { \"type\": \"json_schema\", \"name\": \"support_ticket\", \"schema\": { \"type\": \"object\", \"properties\": { \"category\": { \"type\": \"string\", \"enum\": [\"billing\", \"technical\", \"account\"] }, \"priority\": { \"type\": \"integer\", \"minimum\": 1, \"maximum\": 3 }, \"requires_human\": { \"type\": \"boolean\" }, \"summary\": { \"type\": \"string\" } }, \"required\": [\"category\", \"priority\", \"requires_human\", \"summary\"], \"additionalProperties\": false }, \"strict\": true } } }" In a Responses response, generated content appears in typed output items. SDKs may expose convenience helpers such as aggregated output text or parsed output, but an HTTP client should not assume that the structured object is a top-level field. Read the completed response object, locate its text output, then parse that text as JSON. Send the same schema with Chat Completions Chat Completions places the response schema under response_format.json_schema. The schema body is unchanged; only the wrapper differs. curl -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d "{ \"model\": \"$MODEL_ID\", \"messages\": [ { \"role\": \"user\", \"content\": \"The customer was charged twice and wants a refund.\" } ], \"response_format\": { \"type\": \"json_schema\", \"json_schema\": { \"name\": \"support_ticket\", \"schema\": { \"type\": \"object\", \"properties\": { \"category\": { \"type\": \"string\", \"enum\": [\"billing\", \"technical\", \"account\"] }, \"priority\": { \"type\": \"integer\", \"minimum\": 1, \"maximum\": 3 }, \"requires_human\": { \"type\": \"boolean\" }, \"summary\": { \"type\": \"string\" } }, \"required\": [\"category\", \"priority\", \"requires_human\", \"summary\"], \"additionalProperties\": false }, \"strict\": true } } }" For Chat Completions, the JSON text is normally returned in choices[0].message.content. Parse the content rather than treating the string as an already typed object. The two wrappers can be summarized as follows: Purpose | Responses API | Chat Completions | Schema container | text.format | response_format.json_schema | Format type | text.format.type | response_format.type | Schema name | text.format.name | response_format.json_schema.name | Schema body | text.format.schema | response_format.json_schema.schema | Strict flag | text.format.strict | response_format.json_schema.strict | Result location | Typed output items / output text helper | choices[0].message.content | Modelflare’s OpenAI compatibility layer has explicit conversion logic for these two wrappers on supported OpenAI/Codex routes. That conversion does not create Structured Outputs support in an upstream model that lacks it. For other model families served as raw Chat Completions pass-through, the upstream provider’s exact request contract remains the source of truth. Validate structure and meaning separately Suppose the model returns: { "category": "billing", "priority": 2, "requires_human": true, "summary": "Customer reports a duplicate charge and requests a refund." } This object satisfies the schema, but the application should still perform two distinct checks. Structural validation Use a JSON Schema validator or a typed SDK helper to confirm: the object contains every required property; no unexpected property is present; each value has the required type; the category is in the allowed enum; the priority is within the permitted range. Even when the provider promises strict adherence, application-side validation protects the rest of the system from unsupported routes, integration mistakes, truncated content, and future contract changes. Semantic and business validation The schema cannot determine whether the customer was actually charged twice, whether priority 2 is correct, or whether a human must approve a refund. Those decisions require source data and application policy. For high-impact workflows, treat the model object as a proposed classification. Compare it with authoritative records, enforce permissions and financial limits in deterministic code, and retain a safe audit reference. Never let a structurally valid model object bypass an authorization, billing, or security boundary. Handle incomplete and exceptional output A production parser needs more than a happy-path JSON.parse call. Refusals A model may refuse a request for safety reasons. OpenAI’s structured-output contract makes refusals distinguishable from normal schema output. Check the response status and refusal representation before looking for the JSON object; do not report a refusal as a schema parsing error. Truncation and output limits If generation stops before the object is complete, no schema can repair the missing suffix. Check the endpoint’s completion status and stop reason. Set an output limit that is sufficient for the largest valid object, while keeping the schema and requested content bounded. Unsupported schema keywords Strict output implementations commonly support only a subset of JSON Schema. Start with objects, arrays, primitive types, enums, required fields, and explicit additional-property behavior. Check the current provider documentation before using references, recursive shapes, complex unions, or advanced validation keywords. First-use schema latency Some providers preprocess and cache a new schema. The first request for a new schema can therefore take longer than later requests. Reuse stable, versioned schemas instead of generating a unique schema for every request, and separate first-use measurements from steady-state latency comparisons. Model or route incompatibility An OpenAI-compatible endpoint can accept ordinary Chat Completions while rejecting json_schema, ignoring strict, or forwarding the fields to a model that does not support them. A 200 response containing JSON is still not proof of strict adherence. Test invalid and edge-case inputs and validate the resulting object. Run a compatibility test before production Use a small matrix for every model and route you intend to enable. Test | Evidence to record | Minimal required object | Status, model, route, parsed object, validation result | Every enum value | Whether each allowed value can be produced and parsed | Missing information | Whether the model uses a bounded fallback value or asks for clarification | Safety-triggering input | Refusal representation and application handling | Low output limit | Completion status and truncation handling | Unsupported schema feature | Explicit error versus silently ignored constraint | Responses and Chat wrappers | Whether both produce equivalent application objects | Repeated stable schema | First-use versus steady-state timing | Do not compare models with different schemas, prompts, regions, streaming modes, or output limits and then attribute the result only to model quality. Record the exact test date because provider behavior and model availability can change. Choose the endpoint after testing the workflow Use Responses when the application already works with typed output items, Responses streaming events, or a broader Responses tool workflow. Use Chat Completions when the application has a stable message-based integration and the selected route supports its response_format contract. The broader decision is covered in Responses API vs Chat Completions (/articles/responses-api-vs-chat-completions). Whichever endpoint you select: keep one versioned source for the JSON Schema; translate that schema into the endpoint’s wrapper without changing its meaning; validate the completed response before using it; apply deterministic business rules after structural validation; monitor parsing, refusal, truncation, and compatibility failures separately. For initial client migration, start with the OpenAI-Compatible API Guide (/articles/openai-compatible-api-guide). For event parsing and incomplete streams, use the AI API Streaming Guide (/articles/ai-api-streaming-guide). Check Models & Pricing (/pricing) for the current model and group surface, then verify Structured Outputs on the exact route your API key will use. Structured Outputs are most useful when they narrow a model response into a dependable application interface. They are not a substitute for factual verification, authorization, or billing logic—and “OpenAI-compatible” remains a claim to test feature by feature, not a guarantee to infer from the Base URL. ### DeepSeek V4 Flash: Specs, Parameters, and Modelflare Setup Canonical URL: https://modelflare.dev/articles/deepseek-v4-flash-guide On July 31, 2026, DeepSeek updated deepseek-v4-flash to the public-beta API release DeepSeek-V4-Flash-0731. The model ID did not change. DeepSeek says the update keeps the preview model's architecture and size, while new post-training materially improves coding-agent and tool-use performance and adds native Responses API support. V4 Flash targets text workloads that need reasoning, long context, throughput, and a lower cost profile than V4 Pro. This guide separates three things that are easy to blur together: specifications published by DeepSeek, protocols and prices currently exposed by Modelflare, and behaviors that still need verification on your own workload. Core specifications Item | DeepSeek V4 Flash | Modelflare model ID | deepseek-v4-flash | Current official API version | DeepSeek-V4-Flash-0731 | Architecture | Mixture-of-Experts (MoE) | Total parameters | 284B | Activated parameters per token | 13B | Context length | 1M tokens | Maximum output | 384K tokens | Thinking mode | Enabled by default; default effort is high | Reasoning effort | low, high, or max; xhigh currently maps to high | Official API features | JSON Output, Tool Calls, Responses API, Chat Prefix Completion; FIM only in non-thinking mode | Primary input and output | Text input and text output; do not infer vision support | The useful combination is not merely 284B total parameters, but 13B activated parameters with a 1M-token context window. MoE routing activates only part of the model for each token, aiming to retain model capacity while controlling inference cost. DeepSeek's technical report also describes attention changes designed for long-context efficiency. Whether that becomes lower latency in practice still depends on prompt length, reasoning effort, upstream load, and output size. What changed in the 0731 update DeepSeek describes 0731 as a new post-training release with no architecture or size change. Its published agent results include 82.7 on Terminal Bench 2.1 and 70.3 on Toolathlon Verified, and the company says the agent capability now exceeds the earlier V4 Pro Preview by a wide margin. Treat those numbers as release-direction evidence, not as your production SLA. DeepSeek's public code-agent evaluations used a particular harness, max effort, top_p=0.95, and temperature=1.0; some reported sets are internal. A useful evaluation should pin the model ID, client, tools, timeouts, and task corpus, then record success rate, time to first output, total latency, token use, and repair turns. Choosing thinking parameters V4 Flash enables thinking by default at high effort. A practical starting policy is: Summarization, classification, and formatting: start with low and measure quality; Code changes, complex analysis, and multi-step tools: start with high; A small number of difficult tasks: evaluate max, with explicit latency and reasoning-token budgets; Fast direct answers: set thinking.type=disabled instead of relying on the retired deepseek-chat alias. In thinking mode, temperature, top_p, presence_penalty, and frequency_penalty have no effect even if they are accepted. Tool-using multi-turn conversations must also preserve and replay reasoning_content within the tool-call turn; omitting it can produce an upstream 400. If a client cannot preserve that field correctly, validate the basic non-tool route first. Current Modelflare availability As of August 4, 2026, Modelflare lists deepseek-v4-flash for Chat Completions and Responses. DeepSeek also offers an Anthropic-compatible API upstream, but that does not mean Modelflare currently exposes the same route for this model. Use Models & Pricing (/pricing) as the live protocol source of truth. The current public price snapshot is below, in USD per 1M tokens: Available group | Input | Output | Cached input | Note | deepseek-award | $0.105 | $0.21 | $0.0021 | Available only to API keys eligible for this group | deepseek-stable | $0.15 | $0.30 | $0.003 | Stable group | Prices, group eligibility, and supported protocols can change. Do not hard-code this snapshot as a permanent billing contract. Recheck the live catalog before launch and verify the actual group, tokens, and charge in usage logs. Chat Completions setup Store the Modelflare API key in an environment variable: export MODELFLARE_API_KEY='YOUR_MODELFLARE_API_KEY' Then call the standard public base URL: curl -N -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-v4-flash", "messages": [ {"role": "system", "content": "You are a careful coding assistant."}, {"role": "user", "content": "Review this migration plan and list the rollback risks."} ], "thinking": {"type": "enabled"}, "reasoning_effort": "high", "stream": true }' If an OpenAI SDK does not expose thinking directly, pass it through the SDK's extra_body. Provider-specific fields should remain raw JSON; do not rename them or drop an explicit false value. Responses API setup V4 Flash is also available through Modelflare's Responses endpoint: curl -N -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-v4-flash", "input": "Inspect the proposed deployment sequence and identify unsafe assumptions.", "reasoning": {"effort": "high"}, "stream": true }' Chat Completions and Responses are different wire contracts. A model appearing on both endpoints does not prove identical event types, tool state, error shapes, or continuation behavior. Validate non-streaming, streaming, and tool calls separately before production use. Recommended platform configuration Create a separate Modelflare API key for each application and select a primary group that actually contains deepseek-v4-flash; Add an ordered fallback only when it supports the same model and protocol; Start ordinary text jobs at low or in non-thinking mode, and start complex agent jobs at high; Set client timeouts for long reasoning and long output, not for a single short health prompt; For tools, preserve the complete assistant message, especially reasoning_content and tool_calls; Before launch, check success rate, first output, total latency, output tokens, selected group, and per-request cost on real tasks. Where it fits V4 Flash is a strong candidate for frequent coding assistance, long-document analysis, tool-using agents, batch text processing, and applications balancing capability against cost. Compare V4 Pro or another flagship on the hardest knowledge-heavy and long-horizon tasks. For image input, select a model whose multimodal contract has been explicitly published and verified on Modelflare. Sources and freshness DeepSeek API change log (https://api-docs.deepseek.com/updates/) for the 0731 release status and agent evaluations; DeepSeek models and pricing (https://api-docs.deepseek.com/quick_start/pricing/) for context, maximum output, and official API features; DeepSeek V4 Flash model card (https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) for architecture, parameter scale, and the technical report; DeepSeek thinking-mode guide (https://api-docs.deepseek.com/guides/thinking_mode/) for effort levels, ineffective sampling fields, and tool replay requirements; Modelflare Models & Pricing (/pricing) for current groups, protocols, and platform prices. This article was verified on August 4, 2026. Model snapshots, prices, and protocol support can change; production configuration should use the current official documentation and Modelflare catalog. ### Qwen3.8-Max Released: 2.4T MoE Specs and Modelflare Setup Canonical URL: https://modelflare.dev/articles/qwen3-8-max-guide The Qwen team officially released Qwen3.8-Max on August 3, 2026. It is not the older and easily confused Qwen3-8B, and it is no longer only July's qwen3.8-max-preview: the current official API model ID is qwen3.8-max, which is also the ID used by Modelflare. This is Qwen's largest current Max flagship, aimed at coding, professional workflows, long-horizon agents, and native multimodal work. Qwen also announced that open weights for Qwen3.8-Max and Qwen3.8-27B would arrive the following week. Until the repositories and licenses are actually published, the precise description is “open weights announced,” not “already available for local deployment.” Core specifications Item | Qwen3.8-Max | Modelflare model ID | qwen3.8-max | Architecture | Mixture-of-Experts (MoE) | Total parameters | 2.4T | Activated parameters per token | 95B | Context length | 1M tokens | Maximum non-thinking input | 991K tokens | Maximum thinking-mode input | 983K tokens | Maximum output | 131K tokens | Maximum reasoning tokens | 262K tokens | Input modalities | Text, image, and video | Output modality | Text | Reasoning effort | low, medium, or xhigh; default xhigh | Official features | Prefix Completion, Function Calling, Context Cache, Structured Outputs, Batch, and built-in Responses tools | The 2.4T figure is total model size, not the number of parameters executed for every token. The official release gives 95B activated parameters, which is more useful for understanding the compute involved in MoE inference. A 1M-token context window also does not mean every request should fill it: as input and reasoning budgets grow, latency, caching, and cost need deliberate design. What Qwen3.8-Max targets Qwen positions this release as a broad coding and cowork upgrade, emphasizing three kinds of work: Taking a multi-day software project from an empty directory to a completed result, instead of generating isolated functions; Organizing multi-step deliverables for research, data analysis, office work, design, and other professional workflows; Using image and video understanding throughout planning, execution, and verification rather than as one static recognition step. Qwen presents vendor demonstrations including more than ten days of autonomous coding and hundreds of chip-design optimization turns. These are system-level demonstrations affected by the harness, tools, environment, prompts, and judging method. They do not guarantee the same outcome in your application. Production selection should use your own repositories, documents, tool permissions, and acceptance criteria. Choosing reasoning effort Qwen3.8-Max supports reasoning_effort: low for short answers, lightweight code explanation, and cost-sensitive work; medium as a starting point for daily development, analysis, and multi-step tasks; xhigh for difficult reasoning, long-horizon agents, and complex professional work; it is also the default. Qwen says preserve_thinking is enabled by default so multi-turn work can continue from prior reasoning state. A client that reconstructs message history manually should retain the provider's reasoning fields and tool state rather than concatenating only final text. The model allows up to 262K reasoning tokens, but an available maximum is not a recommended default. Choose effort from task success and unit cost. Current Modelflare availability As of August 4, 2026, Modelflare lists qwen3.8-max for: OpenAI Chat Completions; OpenAI Responses; An Anthropic Messages-compatible route. A protocol label proves that the route exists, not that every QwenCloud-private feature is automatically forwarded. In particular, QwenCloud lists web_search, code_interpreter, web_extractor, t2i_search, and i2i_search as built-in Responses tools. Until each is verified through the exact Modelflare route, prefer client-defined function calling and treat built-in tools as unverified. The current public snapshot for the qwen-award group is below, in USD per 1M tokens: Billing item | Price | Input | $1.239 | Output | $3.71 | Cache read | $0.161 | Explicit cache creation | $1.547 | One-hour explicit cache creation | $2.4752 | The group is available only to eligible API keys. Prices, eligibility, and protocol coverage can change; use Models & Pricing (/pricing) and per-request usage logs as the live source of truth. Recommended Chat Completions setup Store the Modelflare API key in an environment variable: export MODELFLARE_API_KEY='YOUR_MODELFLARE_API_KEY' Start with a text request to validate the basic route: curl -N -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3.8-max", "messages": [ { "role": "user", "content": "Design a safe rollout plan for this service and include validation and rollback gates." } ], "reasoning_effort": "medium", "stream": true }' Use medium to establish a quality, latency, and cost baseline, then raise only the tasks that genuinely need deeper reasoning to xhigh. The upstream default being xhigh is not a reason to give every batch request the maximum reasoning budget. Rolling out multimodal input The official model accepts text, image, and video, but multimodal content shapes vary by protocol and upstream implementation. Roll it out in three stages: Validate authentication, model access, streaming, and billing with text only; Use one fixed test image to validate the content shape, size limits, and output; Add long video, tools, or multi-turn visual feedback only after that, while checking whether retries duplicate billable work. Do not treat image URLs, Base64 payloads, file uploads, and provider object storage as interchangeable protocols. Test the exact content block your client will send. Recommended platform configuration Use a separate API key per application and confirm that its primary group lists qwen3.8-max; Start daily development and analysis at medium, lower short jobs to low, and reserve xhigh for hard long-horizon work; Design chunking and caching for long-context jobs instead of putting an entire knowledge base into every 1M-token request; Validate the Chat Completions, Responses, and Anthropic wire contracts separately; Do not make provider-built-in tools a production dependency until they pass a real request through the target Modelflare route; Compare success rate, first output, total latency, reasoning tokens, cache hits, and per-request cost on a fixed task set; Rerun regression samples when moving from the preview to the production model rather than assuming a family name guarantees identical output. Where it fits Qwen3.8-Max fits complex software engineering, multi-step professional workflows, long-document and video understanding, multimodal-feedback agents, and teams that want one flagship for reasoning and orchestration. It may not be the most economical default for high-throughput, low-latency text transformations. Compare it with Qwen Flash, DeepSeek V4 Flash, or another smaller model on identical inputs. Sources and freshness Official Qwen3.8-Max release (https://qwen.ai/blog?id=qwen3.8) for the release date, 95B activated parameters, coding/cowork demonstrations, and open-weight plan; QwenCloud Qwen3.8-Max model page (https://www.qwencloud.com/models/qwen3.8-max) for context, input/output limits, modalities, features, and official prices; Modelflare Models & Pricing (/pricing) for current groups, protocols, and platform prices. This article was verified on August 4, 2026. Open-weight status, model snapshots, prices, and protocol support can change; use the current official model page, repository license, and Modelflare catalog for production decisions. ### LLM Proxy vs AI Gateway: Architecture, Control, and Tradeoffs Canonical URL: https://modelflare.dev/articles/llm-proxy-vs-ai-gateway An LLM proxy and an AI gateway can sit in the same network position, but they do not necessarily own the same responsibilities. A proxy primarily forwards model requests and responses. An AI gateway usually adds model-aware controls such as routing policy, fallback, usage records, cost attribution, and workload-level access rules. The names are not standards. Some products call a model-aware gateway a “proxy,” while others use “gateway” for little more than a managed endpoint. The reliable way to choose is to compare the responsibilities you need, the behavior you can verify, and the parts your application team would otherwise have to build. Start with responsibility, not the product label Both layers commonly appear between an application and one or more model providers: Application or coding agent ↓ LLM proxy or AI gateway ↓ Model provider and selected model That diagram does not reveal what the middle layer actually does. A useful evaluation starts with a responsibility matrix. Responsibility | Transport-oriented LLM proxy | Model-aware AI gateway | TLS termination and HTTP forwarding | Common | Common | Upstream URL and header handling | Common | Common | Streaming response forwarding | Often | Usually, but the exact event contract still matters | Provider credential isolation | Sometimes | Common, but storage and access boundaries must be verified | OpenAI-compatible request surface | Sometimes | Common, but compatibility varies by model and feature | Model- or provider-aware routing | Limited or application-owned | Common | Retry and ordered fallback policy | Basic upstream retry at most | Often model- and failure-aware | Per-workload rate or quota policy | Usually external | Common | Token, usage, and cost records | Usually external | Common | Request-level latency diagnostics | Usually external | Common | Policy and audit controls | Usually external | Product-dependent | “Common” does not mean guaranteed. For every important row, ask for the exact request contract, failure behavior, record produced, and control available to the operator. What an LLM proxy normally owns A transport-oriented proxy can be enough when the main goal is to put a stable endpoint in front of an upstream API. It may centralize TLS, an upstream hostname, selected headers, request size limits, timeouts, and basic logs. An LLM-specific proxy may also understand server-sent events well enough to forward streaming responses without buffering them. This is valuable infrastructure. It gives an application a controlled network path and can keep provider credentials out of some client environments. It can also provide a convenient place for conventional authentication and network policy. The boundary becomes less clear when the proxy starts translating request formats, selecting providers, counting model tokens, or calculating request cost. Those are model-aware behaviors. At that point, evaluate it as a gateway even if the product still uses “proxy” in its name. A simple proxy is most attractive when: one application uses one provider and a small, pinned model set; the application already owns retries, usage accounting, and incident diagnostics; provider-native features must pass through without translation; there is no need for per-team or per-workload routing policy; the team wants the smallest possible operational layer. What an AI gateway adds An AI gateway treats a model request as more than generic HTTP traffic. It can use the requested model, protocol, key policy, group availability, limits, and prior attempt results when deciding where a request should go. It can then connect the final attempt to token usage, timing, status, and cost records. The feature sets differ. For example, the Cloudflare AI Gateway documentation (https://developers.cloudflare.com/ai-gateway/) groups analytics, logging, caching, rate limiting, retries, and fallback under its gateway. Kong’s AI Gateway metrics documentation (https://developer.konghq.com/ai-gateway/monitor-ai-llm-metrics/) describes model, token, cost, cache, latency, and error metrics. These examples show why “gateway” usually implies an AI-aware control plane, but they do not define a universal minimum. An AI gateway becomes useful when several of these requirements appear together: multiple applications or agents need separately attributable API keys; a requested model can be served through more than one eligible route; limits need to apply before an upstream request creates work or cost; operators need to distinguish authentication, routing, upstream wait, first output, and generation time; model usage and final charges must be explainable per request; a primary route needs an explicit, ordered fallback policy; the team needs one operational view across several model groups. A gateway does not make every provider interchangeable. It also does not remove the need to validate model output, make tool execution idempotent, protect secrets in the application, or test provider-specific features. Use a workload decision matrix The following matrix is more useful than choosing by product category alone. Workload | Main requirement | Sensible starting layer | Why | Internal service using one provider and one model | Stable network path | Direct API or a small proxy | Routing and cost policy may add more machinery than value. | Customer-facing app with several eligible routes for the same model | Availability and attempt evidence | AI gateway | Route selection, bounded fallback, and per-attempt diagnostics need one owner. | Team-wide coding agents | Keys, quota, usage attribution, and offboarding | AI gateway | Shared provider credentials and unassigned spend become operational risks. | Application using a new provider-native feature | Exact wire compatibility | Native endpoint first, then verified proxy or gateway | A normalized API can lag or omit provider-specific fields. | Batch workflow with its own queue and retry controller | Throughput and application-owned recovery | Proxy or gateway with retries disabled or bounded | Multiple retry layers can amplify failures and duplicate work. | Regulated or sensitive workload | Data path, retention, and access evidence | Depends on verified controls | The word “gateway” is not proof of a security or compliance boundary. | The decision can change as the workload matures. Starting directly with one provider and later introducing a gateway is reasonable if the application preserves a clear model client boundary and tests the migration contract. Account for the hidden tradeoffs Another hop must be measured A proxy or gateway adds a network and processing layer. Its useful cost cannot be summarized by one average latency number. Measure upstream-header time, first effective output, first visible text when applicable, total duration, and output speed at realistic concurrency. A gateway that adds a small fixed overhead may still reduce operational recovery time, but that is a workload decision rather than a universal performance claim. Compatibility is feature-specific “OpenAI-compatible” may cover the base URL, authentication header, and a Chat Completions request while differing on Responses events, structured outputs, tool calls, usage fields, or error shapes. Test every feature your application uses. The OpenAI-Compatible API Guide (/articles/openai-compatible-api-guide) provides a migration baseline, and Responses API vs Chat Completions (/articles/responses-api-vs-chat-completions) explains why the endpoint alone is not enough to infer compatibility. Centralization creates a failure domain Moving keys, routing, limits, and logs into one layer simplifies ownership but also makes that layer important. Ask how configuration is versioned, how it is rolled back, what happens when its control plane is unavailable, and whether an existing request can finish during an operator change. Cost records need a defined source of truth Some gateways estimate cost from provider list prices. Others use configured prices, group multipliers, or recorded billing expressions. Confirm when the price is selected, whether it is frozen for the request, how cached tokens and tools are represented, and whether a final charge can be traced back to usage. See AI API Cost Tracking (/articles/ai-api-cost-tracking) for the layers that should remain distinguishable. Exit cost matters Before adopting a normalized request surface, identify which application code depends on gateway-only headers, model aliases, routing names, or log APIs. A useful gateway should make the current route easier to operate without making the underlying protocol impossible to recover. Three practical examples One-provider internal assistant An internal assistant sends a moderate number of non-streaming requests to one pinned model. The application already records its own job IDs and has one server-side provider key. A conventional proxy may be sufficient. Adding cross-provider routing and a separate cost-control plane would not solve a current problem. The architecture should still leave room to introduce a gateway later: keep the model client behind one internal interface, retain provider request IDs, and avoid exposing the provider key to browsers or desktop clients. Multi-route production application A customer-facing application needs a requested model to remain available when one upstream account is saturated. The team also needs to know which route handled the request and which attempt created cost. This is gateway territory, because fallback, billing, and diagnostics need to agree on the same request identity. The fallback should preserve the requested contract. Switching to a different model may change quality, latency, tool behavior, or price and should therefore be an explicit product policy, not an invisible recovery shortcut. The Reliable AI API Routing guide (/articles/reliable-ai-api-routing) explains how ordered groups and request-level evidence fit together. Coding agents across an engineering team Coding agents can create long, tool-heavy sessions from many developer machines. A single shared provider key makes quota, attribution, rotation, and offboarding difficult. A gateway can issue workload-specific keys, restrict access, connect usage to a team or project, and preserve one place to review failures. The gateway still cannot decide whether a tool action is safe or whether a generated patch is correct. Repository permissions, sandboxing, tool approval, and code validation remain outside the model-routing layer. Where Modelflare fits Modelflare is intended to provide an AI-aware access and routing layer rather than only a transparent HTTP relay. A regular API key can select a primary model group and ordered fallback groups. A Smart API Key can evaluate eligible groups using its configured strategy. In both cases, routing searches for an eligible path for the requested model; it should not silently replace the requested model with another one. Group RPM is enforced against the concrete selected group before billing and before an upstream request. When that group is full, an eligible next group can be considered; if no route remains, the request returns 429. This separates a routing-capacity decision from work already sent upstream. Usage logs connect the completed request to status, token usage, cost, and timing metadata. Timing fields distinguish authentication, group selection, upstream headers, first upstream event, first effective output, first visible text, upstream completion, and total handler duration without requiring prompts, response text, complete API keys, or raw request bodies in the timing record. There is also an important compatibility boundary: GPT, Codex, and OpenAI traffic is the fully adapted target. Other OpenAI-compatible model families should be treated as raw Chat Completions pass-through unless their additional behavior has been separately verified. A shared base URL is therefore not a promise that every model exposes the same Responses, tools, or structured-output contract. Use Models & Pricing (/pricing) to inspect the currently exposed models and groups, and use the Modelflare Docs (/docs) for client-specific setup. Production evaluation checklist Before selecting either layer, test the behaviors that matter to your application. Protocol: Does it preserve the exact request and response shapes you use? Streaming: Does it forward events without buffering, loss, or reordered tool arguments? Model identity: Can a route change without silently changing the requested model? Fallback: Which failures are eligible, how many attempts occur, and when does retry stop? Limits: Are rate and quota decisions applied before upstream work and billing? Usage: Are input, output, cached, reasoning, and tool usage represented accurately when applicable? Cost: Is the recorded charge tied to the price and route used for that request? Diagnostics: Can an operator separate gateway time, upstream wait, first output, and generation speed? Secrets: Who can read provider credentials, request bodies, and response bodies? Change control: Can routing and policy changes be reviewed and rolled back? Exit path: Can the application return to a provider-native endpoint without a rewrite? Run this checklist with the same model, prompt class, output length, streaming mode, tools, region, and realistic concurrency. A successful “hello world” request proves reachability, not production compatibility. Frequently asked questions Is every OpenAI-compatible proxy an AI gateway? No. OpenAI compatibility describes some part of the request surface, while a gateway describes operational responsibilities. A proxy can expose an OpenAI-shaped endpoint without owning routing, cost, limits, or diagnostics. Does an AI gateway eliminate provider keys? Not necessarily. Some gateways hold provider credentials, some use bring-your-own-key arrangements, and some provide their own billing relationship. Verify where credentials live and who can use or export them. Will an AI gateway make requests faster? Not automatically. It adds a hop, but it may provide better route availability and clearer diagnostics. Measure the complete request timeline for your workload instead of relying on a generic latency claim. Should fallback choose a different model? Only when the application has explicitly accepted that tradeoff. The safer default is to find another eligible route for the requested model and protocol. Cross-model substitution needs its own quality, compatibility, latency, and price policy. The practical distinction is therefore simple: choose a proxy when you mainly need a controlled transport path; choose an AI gateway when model-aware routing, workload policy, usage, cost, and request evidence need one accountable owner. Then verify those responsibilities individually, because the product label alone is not evidence. ### AI API Error Guide: 401, 403, 429 & 5xx Canonical URL: https://modelflare.dev/articles/ai-api-error-troubleshooting AI API errors are easiest to diagnose when the status code is treated as the beginning of an investigation, not the complete cause. Preserve the request ID, timestamp, endpoint, model, API key name, selected group, response body, and timing before retrying. The goal is to identify which layer rejected or ended the request: client, authentication, access policy, protocol validation, routing, model backend, or downstream connection. What common status codes usually indicate Status | First interpretation | First action | 400 | Invalid request shape, field, or protocol contract | Compare the payload with the selected endpoint; do not retry unchanged | 401 | Missing, malformed, disabled, expired, or invalid credential | Verify the Authorization header and the current key | 403 | Authenticated but blocked by account, model, group, IP, or policy | Check key restrictions and model/group access before testing channels | 404 | Wrong endpoint path or unavailable model identifier | Verify the base URL, path, and authenticated model list | 429 | Request-rate, quota, or eligible-route limit | Inspect quota and group limits; use bounded backoff or configured fallback | 499 | Downstream client disconnected or cancelled before completion | Inspect client deadlines, aborts, proxies, and first-output timing | 500 | Internal processing or adapter failure | Preserve the request ID; retry only if the operation is safe | 502/503 | Bad service provider response, unavailable route, or temporary capacity issue | Check route evidence and use bounded fallback or retry | 504 | A gateway or service provider time budget expired | Compare first-output and total timing with client deadlines | These interpretations are diagnostic starting points. The error body and request record determine the actual reason for a specific call. Diagnose 401 before changing the model A 401 normally occurs before useful model routing. Check: the header uses Authorization: Bearer ...; the application did not include quotes, whitespace, or a placeholder value; the key is enabled and has not expired; the request is sent to the intended Modelflare host; a secret manager or deployment did not retain an older rotated key. Do not paste the full key into logs or support messages. Reproduce with a dedicated test key if exposure is possible, then revoke the compromised credential. Diagnose 403 as an access-policy failure first A 403 does not automatically mean an service provider provider rejected the request. Authentication can succeed while access fails before channel selection because: the key restricts the requested model; the key's IP allowlist does not include the observed client IP; the account cannot use the selected model group; the user or key is disabled by policy; the requested feature is not allowed for that route. Retrieve /v1/models with the same key, inspect its group and model limits, and compare the exact error code. If no channel was selected, changing service provider channels cannot fix the earlier policy decision. Diagnose 429 without creating a retry storm A 429 is a capacity or policy signal, not an instruction to retry immediately in a tight loop. Determine whether the limit belongs to the key quota, account, model group, or selected route. For retryable rate limits: honor Retry-After when present; use exponential backoff with jitter; cap attempts and total elapsed time; keep application concurrency within a known bound; use ordered fallback groups only when they preserve the requested model and protocol. Creating more API keys does not necessarily bypass an account- or group-level request limit. Diagnose 499 from the caller outward Modelflare can record 499 when the downstream client connection ends before the response completes. Start with the caller: Was an abort signal triggered? Did the client, load balancer, CDN, or reverse proxy reach an idle or total timeout? Did the user close the page or stop the agent? Did the stream produce a tool event without visible text? How long elapsed before first effective output? A 499 record alone does not establish a channel outage. Compare other requests in the same time window and the same model/group before changing routing policy. Separate 5xx failures by layer For 5xx, preserve the original status and error details instead of converting every failure into a generic 500 in the client. A validation or adapter 500 can be deterministic and may fail again unchanged. A 502 can indicate an invalid or incomplete service provider response. A 503 can indicate no eligible route or temporary unavailability. A 504 indicates that a time budget expired, but the responsible layer still needs to be identified. Use request timing, selected group, channel evidence available to support, and nearby requests to distinguish a single bad response from a sustained route issue. Decide whether to retry Failure class | Default retry decision | Invalid payload or unsupported protocol | Fix the request; do not retry unchanged | Invalid key or denied access | Fix credentials or policy; do not retry unchanged | Quota exhausted | Increase or reallocate quota intentionally; do not loop | Rate limit | Retry with bounded backoff when permitted | Temporary 502/503/504 | Retry idempotent work with a strict attempt budget | Client cancellation | Retry only if the user still wants the work and duplicates are safe | Tool or write side effect | Require application-level idempotency before repeating | Retries create new work and can create new cost. Record every attempt and avoid hiding the original failure behind a later success. Evidence to capture safely For support or incident review, capture: request ID and UTC timestamp; endpoint and streaming mode; requested model and selected group; HTTP status and structured error code; total time and first-effective-output time; whether the caller cancelled; a sanitized description of the request feature, such as tools or image input. Do not include the full API key, prompt, response content, raw request body, email, or plaintext IP address unless an approved secure process specifically requires it. After locating the failing layer, use Reliable AI API Routing (/articles/reliable-ai-api-routing) for fallback design and the Streaming Guide (/articles/ai-api-streaming-guide) for event and timeout failures. Frequently asked questions Does a 403 prove that a provider is down? No. A 403 can be produced by account, key, model, group, IP, or feature policy before an service provider route is selected. Check the request record and error code first. Is 499 a standard service provider model error? No. In Modelflare request records it represents a downstream cancellation observed before completion. Investigate caller and proxy deadlines before treating it as model failure. Should every 5xx response be retried? No. Retry only operations that are safe to repeat, use a bounded attempt and time budget, and preserve the first error. Deterministic adapter or request problems will not improve with repetition. ### AI API Streaming: SSE, First Output & Timeouts Canonical URL: https://modelflare.dev/articles/ai-api-streaming-guide AI API streaming sends output incrementally while a model is generating instead of waiting for the complete response body. It can improve perceived responsiveness, but it does not automatically reduce model latency, and it adds a protocol contract that the client must parse correctly. The first rule is to match the parser to the endpoint. Chat Completions uses streamed completion chunks, while Responses uses typed response events. A client can receive HTTP 200 and still render nothing if it expects the wrong event shape. Start with an unbuffered request For a Chat Completions model, curl -N disables curl output buffering: curl -N -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "YOUR_CHAT_MODEL", "messages": [{"role": "user", "content": "Explain SSE in three points."}], "stream": true }' For a Responses-compatible model: curl -N -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "YOUR_RESPONSES_MODEL", "input": "Explain SSE in three points.", "stream": true }' Use an exact model ID from Models & Pricing (/pricing). Test the same request once with streaming disabled first; this separates request validation from stream parsing. Treat the event stream as a protocol Server-Sent Events are framed records, not arbitrary JSON fragments. A production client needs to: keep the response body unbuffered through its HTTP library, proxy, and UI layer; accumulate partial network reads until a complete event is available; parse the event type used by the selected endpoint; handle text deltas, reasoning or tool events, completion events, and error events; avoid assuming that every useful response contains visible text; preserve cancellation and final usage information; close the stream after a terminal event instead of waiting forever. When possible, use an SDK that natively supports the chosen protocol. A home-grown parser should be tested with fragmented frames, multiple events in one read, tool-only output, and a clean terminal event. Measure more than time to first token “Time to first token” is often too vague for agent traffic. Separate these measurements: Measurement | What it means | Connection and authentication | Time to reach the gateway and validate the key | Service provider response headers | Time until the selected route starts a response | First effective output | First useful text, reasoning, or tool event | First visible text | First text a user can actually see | Output throughput | Visible generation speed after output begins | Total response time | Time until completion, failure, or cancellation | A tool call may produce effective output before any visible text. For that reason, first effective output is the safer operational metric, while first visible text is useful for user experience. Modelflare usage logs retain request-level timing evidence so these phases can be compared with model, group, status, tokens, and cost. Design timeouts by phase One short global timeout creates ambiguous failures. A streaming client should reason about several budgets: Connection timeout: failure to establish the HTTP connection. Header or first-output timeout: no response or effective event after the request was accepted. Idle timeout: the stream started but no new event arrived for an unexpectedly long period. Overall deadline: the complete operation exceeded the workload's business limit. Reasoning-heavy or tool-driven requests can legitimately wait longer before visible text than a short chat completion. Choose budgets from observed workload behavior, and cancel intentionally when the user no longer needs the result. Understand cancellation and 499 records If the client, reverse proxy, browser tab, or caller deadline closes the downstream connection, the gateway can record the request as client-cancelled. A 499 label in a request record is evidence that the downstream connection ended before completion; it is not proof by itself that the selected model or channel failed. When investigating cancellation, compare: the client deadline and abort signal; proxy buffering and idle timeouts; time until first effective output; whether any event reached the client; the selected model and group; the request ID and cancellation timestamp. Diagnose a stream that returns no visible text Use this order: Repeat the request with "stream": false. Confirm the model supports the selected endpoint. Capture raw events before the UI transforms them. Check whether the response contains a tool call or reasoning event instead of text. Verify that an intermediary is not buffering the body. Confirm that the parser handles the endpoint's terminal event. Review usage logs for status, first output, total time, and cancellation. If non-streaming works and raw streaming events arrive, the failure is usually in client parsing or rendering rather than authentication or model access. If no raw event arrives, continue with the AI API Error Guide (/articles/ai-api-error-troubleshooting). Production streaming checklist Match the parser to Chat Completions or Responses explicitly. Disable buffering in the HTTP client and every controlled proxy layer. Test text, reasoning, tool-only, and error events used by the application. Distinguish first effective output from first visible text. Use phase-appropriate timeouts and an intentional abort signal. Treat retries as new requests unless the operation has a safe idempotency design. Preserve request ID, status, selected model, group, and timing for diagnosis. Compare the endpoint decision in Responses API vs Chat Completions (/articles/responses-api-vs-chat-completions). Frequently asked questions Does streaming make the model generate faster? Not necessarily. Streaming exposes output earlier; generation speed and time before the first event still depend on the model, route, context, reasoning, and tools. Why does curl work while the application shows nothing? The application or an intermediary may buffer the body, or the parser may expect Chat Completions chunks while receiving Responses events. Capture raw frames at the application boundary before changing the model or route. Should a client retry after a stream disconnects? Only when repeating the operation is safe. A disconnected stream may have already caused service provider work or tool activity. Use bounded retries and application-level idempotency where duplicate side effects matter. ### AI API Key Security & Cost Controls Canonical URL: https://modelflare.dev/articles/api-key-security-cost-controls An AI API key should identify one workload boundary, not an entire organization forever. Separate keys make access, quota, routing, rotation, and usage records easier to control without exposing one credential across unrelated applications. Modelflare API keys can combine an expiration date, finite or unlimited quota, model limits, an IP allowlist, and routing policy. These controls are most effective when they are designed together rather than added only after an incident. Start with one key per workload Create separate keys for boundaries that need independent access or accounting, such as: production API and background workers; development and CI; a coding agent or local tool; a customer-facing application; a temporary contractor or migration; a high-cost batch workload. Use names that describe environment and purpose, such as prod-support-agent or dev-evaluation. A clear name makes request records and rotation decisions understandable without revealing the secret. Keep the secret out of client-visible code Store the full key in a server-side secret manager or protected environment variable. Do not put it in: browser JavaScript or mobile packages that users can inspect; source control, container images, screenshots, or tickets; analytics events, error trackers, shell history, or application logs; URLs, query strings, or copied command examples with real values. For a browser application, send requests through a backend that authenticates the user and holds the Modelflare key. If a key may have leaked, create a replacement, move the workload, verify traffic on the new key, and revoke the old one. Use quota and expiration as independent boundaries A finite key quota places a hard usage boundary on that key. An expiration date limits how long the credential remains valid. They solve different risks: Control | Primary purpose | Finite quota | Limits the amount of usage attributable to the key | Unlimited quota | Avoids a key-level cap; account credits and other policy still apply | Expiration | Ends access after a defined time | Disable or revoke | Stops new access immediately | Development, contractor, and evaluation keys should usually have both a finite quota and an expiration. Production keys may need a larger or unlimited key quota, but they still benefit from separate workload attribution and a rotation plan. Quota is not a substitute for concurrency limits or client retry control. A burst can consume the remaining amount quickly, and repeated failures can create additional requests. Restrict models when the workload is narrow Model limits prevent a key from requesting models outside its intended set. This reduces accidental use of an incompatible or unexpectedly expensive model and gives a clearer 403 boundary when the client asks for something it should not access. Keep the allowed set aligned with the endpoint. A key that can list a model still needs a client and protocol that support that model's request format. Review the live contract in Models & Pricing (/pricing) instead of copying model IDs into a long-lived policy document. Add IP rules only when the network path is stable An IP allowlist checks the client IP observed by Modelflare against individual addresses or CIDR ranges configured on the key. It is useful for servers with known egress, but it can break traffic when: a cloud workload changes outbound NAT addresses; requests move through a new proxy or gateway; IPv4 and IPv6 paths differ; a developer switches networks; the configured CIDR is broader or narrower than intended. Verify the real egress path before enabling the restriction, keep an emergency rotation path, and do not treat IP allowlisting as a replacement for secret protection. Choose regular or Smart routing intentionally Key type | Routing behavior | Best fit | Regular API Key | One explicit primary group plus optional ordered fallback groups | Workloads that require a known route order and group policy | Smart API Key | Evaluates account-eligible groups using balanced, stability-first, or low-price-first strategy | Workloads that want broader automatic route selection | Both key types still send the requested model and protocol. Routing policy chooses an eligible group; it does not guarantee that every group supports every model or feature. A regular fallback list should be tested in order. A Smart API Key should be evaluated with real latency, price, and compatibility requirements before production. For detailed behavior, read Reliable AI API Routing (/articles/reliable-ai-api-routing). Rotate without an avoidable outage Use an overlapping rotation: Create a new key with the same intended restrictions. Store it in the application's secret system. Deploy the new credential without deleting the old one. Verify model access and one real request. Confirm new usage records appear under the new key. Remove the old key from every deployment and automation job. Disable or delete the old key. Monitor for unexpected requests using the retired credential. If compromise is active, revoke first and accept the controlled interruption. The overlap procedure is for planned rotation, not continued exposure. Connect key policy to cost review Key-level isolation makes cost review actionable. Filter usage logs by key, then compare model, group, input and output usage, status, retries, and cost. A finite quota can stop further use, but the request records explain what consumed it. See AI API Cost Tracking (/articles/ai-api-cost-tracking) for a repeatable review process. API key checklist One key per application, environment, or accountable workload. No secrets in browser code, repositories, logs, URLs, or screenshots. Finite quota and expiration for temporary or experimental use. Model restrictions for narrow workloads. IP rules only for a verified, stable egress path. Explicit regular fallback order or an intentionally selected Smart strategy. Rotation tested before the old key is removed. Usage and cost reviewed by key after every policy change. Frequently asked questions Does an unlimited key remove all spending limits? No. It removes the key-level quota cap. Account credits, model pricing, group policy, request limits, and other service controls still apply. Can multiple keys bypass a rate limit? Not necessarily. A limit can apply at account, model group, or route level. Create keys for access and attribution, not as an unverified rate-limit workaround. Is an IP allowlist enough if a key leaks? No. Network rules reduce where a key can be used, but network paths change and configuration can be wrong. Rotate exposed credentials and keep the secret protected at rest and in deployment. ### What Is an AI API Gateway? Routing & Usage Canonical URL: https://modelflare.dev/articles/what-is-ai-api-gateway An AI API gateway is a service boundary between an application and one or more model backends. The application sends a request to the gateway; the gateway applies authentication and access policy, selects an eligible route for the requested model, forwards the request, and records the result. That definition is more useful than saying “one API for every model.” A gateway can centralize the parts that should be consistent, but it does not make every model or provider support the same protocol, tools, input types, or latency profile. Where an AI API gateway sits A typical request path has six stages: Stage | Responsibility | Client | Chooses a model, protocol, input, and streaming mode | Compatible endpoint | Accepts the selected wire format, such as Chat Completions or Responses | API key policy | Checks account access, quota, expiration, model limits, IP rules, and routing configuration | Model route | Selects an eligible model group and channel without changing the requested contract | Model backend | Executes the request and returns protocol-specific output | Usage record | Connects status, model, group, tokens, timing, and cost to the request | In Modelflare, the public base URL is https://modelflare.dev/v1 (https://docs.modelflare.dev/guides/endpoints/). Clients can use compatible endpoints while keeping model availability, protocol support, and current pricing discoverable through Models & Pricing (/pricing). What a gateway can centralize Authentication and key policy Applications use one Modelflare API key instead of putting several service provider credentials into every deployment. Separate keys can be created for production, development, agents, and automation so each workload has its own quota, expiration, model access, IP rules, and routing policy. Model discovery The authenticated model list shows the model IDs available to the current key: curl -sS https://modelflare.dev/v1/models \ -H "Authorization: Bearer $MODELFLARE_API_KEY" This is an access check, not a universal compatibility promise. A returned model still needs to support the endpoint and features used by the client. Routing and fallback A regular API key can use one primary model group and ordered fallback groups. A Smart API Key evaluates the concrete groups available to the account according to its routing strategy. In both cases, routing chooses where the requested model is served; it should not silently replace that model with a different one. For production design, see Reliable AI API Routing (/articles/reliable-ai-api-routing). Usage and cost visibility A gateway can connect the completed request to its selected model and group. Modelflare usage logs expose request status, token usage, response timing, and cost so an operator can investigate one request instead of estimating from a monthly total. The accounting layers are described in AI API Cost Tracking (/articles/ai-api-cost-tracking). What remains model- and protocol-specific An AI API gateway does not erase the service provider contract. Verify these boundaries separately: Endpoint support: a model may support Chat Completions, Responses, Anthropic Messages, Gemini-style requests, or only a subset. Streaming events: Chat Completions chunks and Responses events are different client contracts. Tools and structured output: schemas and supported tool types can vary by model and provider. Multimodal input: image, audio, files, and other inputs require explicit model and endpoint support. Provider-specific fields: some compatible endpoints intentionally pass private controls through rather than translating them. Latency and limits: route availability does not make every backend share the same first-output time or request limit. The safest migration starts with the OpenAI-Compatible API Guide (/articles/openai-compatible-api-guide) and tests the exact features used by the application. A practical gateway evaluation Evaluate a gateway with the real workload, not one short prompt: Create a dedicated API key with the intended quota and routing policy. Retrieve /v1/models and select a model visible to that key. Confirm the supported API format on the live catalog. Send one non-streaming request and preserve the returned status. Test streaming, tools, structured output, and multimodal input separately when required. Review the resulting usage record for model, group, tokens, timing, and cost. Exercise the fallback path without changing the requested model or protocol. Repeat with representative context size and client timeouts before production cutover. When a gateway is a good fit An AI API gateway is useful when a team needs consistent key management, access to several model families, explicit routing policy, centralized usage records, or a stable integration boundary for agents and applications. A direct provider integration can still be appropriate when one application depends on a provider-only feature that has no compatible gateway contract, or when the team intentionally wants to own provider credentials, routing, settlement, and diagnostics itself. The decision is not “gateway or provider” in the abstract. It is whether the gateway preserves the exact protocol and features the workload needs while reducing operational complexity around access, routing, and evidence. Frequently asked questions Does one gateway mean every model uses the same request format? No. A gateway can expose several compatible formats, but the client, endpoint, model, and service provider provider must still agree on one contract. Check the catalog before switching an existing request between Chat Completions and Responses. Does fallback automatically choose a different model? Modelflare fallback groups are routes for the requested model. They are not an instruction to substitute an arbitrary model. Every candidate group must expose the same model and protocol capabilities required by the request. What should be measured before production? Measure authentication success, model access, non-streaming output, first effective streaming output, total response time, usage, cost, and failure behavior. A successful health-check prompt alone does not prove production compatibility. ### AI API Cost Tracking: Tokens and Model Groups Canonical URL: https://modelflare.dev/articles/ai-api-cost-tracking AI API cost tracking is most useful when every charge can be connected to one request, one model, one model group, and one measured unit of work. A monthly total can show that spend changed; a request-level record can explain why. Modelflare combines model pricing, measured usage, group policy, and request logs so users can review cost without estimating from visible text alone. The four layers of a request cost 1. Model pricing Each model has its own pricing basis. Many text models use input and output tokens, while image, audio, reranking, or other APIs may use different metered units. Some models also use versioned, advanced pricing rules for features that cannot be represented by one flat token rate. Use Models & Pricing (/pricing) for the current model and API-format contract instead of copying a price into application code. 2. Measured request usage The gateway records the usage returned or derived for the completed request. For token-priced text models, input and output are kept separate because their rates may differ. Streaming text visible in a terminal is not a reliable cost oracle. Tool items, reasoning, cached input, normalized tokens, or provider-specific usage fields may affect the bill even when they are not displayed as ordinary assistant text. 3. Model-group multiplier The selected routing group can apply a sales multiplier to base usage billing. For example, a group ratio of 0.9 applies 90% of the base model charge for that request. The multiplier belongs to usage billing. It does not change how many credits a top-up adds. 4. The final request record The usage log connects the calculated amount to the request's model, group, status, token usage, timing, and other safe operational metadata. Failed or cancelled requests can follow different settlement paths, so the persisted result is more authoritative than a client-side estimate. What to compare in usage logs When spend changes, compare requests along these dimensions: Model: Did traffic move to a model with a different input, output, or feature price? Group: Did the selected group or fallback group use a different multiplier? Protocol: Did the request move between Chat Completions and Responses with a different usage shape? Input size: Did prompts, retrieved context, files, or tool results grow? Output size: Did completion limits or agent loops produce more output? Status and retries: Did failures cause additional completed attempts? Timing: Did longer generation correlate with more output rather than just queueing? Use a stable request ID or application-specific API key to isolate a workload instead of comparing unrelated account traffic. Set practical cost boundaries Separate API keys by workload Create different keys for production, development, automation, and personal tools. Each key can have its own name, expiration, group policy, and finite or unlimited quota. Choose groups intentionally Do not select a group only by its label. Check live model availability, the group multiplier, access requirement, RPM, and fallback policy. A cheaper primary group with an acceptable fallback can be more predictable than an implicit routing choice that is difficult to explain. Limit input before limiting output Large system prompts, repeated conversation history, retrieved documents, and tool results often dominate input usage. Trim context that no longer affects the answer, and avoid resending unchanged data when the client can safely reference or cache it. Review agent loops One user action can create several model requests. Track each tool round and retry rather than treating the final visible answer as one API call. Why client-side estimates drift A local tokenizer or character count can be useful for planning, but it may differ from billed usage because: providers can normalize or count content differently; cached, reasoning, image, audio, or tool units may use separate rates; gateway settlement uses the actual selected model and group; failed requests and refunds depend on the real request lifecycle; pricing can change while old requests retain their recorded result. For financial review, reconcile the persisted usage and credit-ledger records rather than deriving remaining credits from one display field. A repeatable review process Filter usage logs to one API key and time range. Group requests by model and selected group. Compare input, output, status, and cost per request. Inspect outliers for retries, long context, tool loops, or fallback. Confirm the live price and group policy before changing routing. Set a finite key quota when the workload needs a hard spending boundary. Recheck the same metrics after the change. Clear cost control starts with attribution. When model, group, usage, and request outcome remain connected, teams can optimize the actual source of spend instead of guessing from aggregate totals. Turn that review into an enforceable workload boundary with the quota, expiration, model, IP, rotation, and routing controls in AI API Key Security (/articles/api-key-security-cost-controls). ### OpenAI-Compatible API Guide: Change the Base URL Canonical URL: https://modelflare.dev/articles/openai-compatible-api-guide An OpenAI-compatible API lets an existing client keep a familiar authentication header, JSON request shape, and streaming pattern while sending traffic to a different gateway. The change can be as small as replacing the base URL and API key, but compatibility is still a protocol contract—not a promise that every model supports every endpoint or provider-specific field. This guide explains the safe migration path for applications, scripts, and AI tools that already speak an OpenAI-style API. What OpenAI compatibility actually covers The most reusable parts of the contract are: Bearer-token authentication through the Authorization header. JSON requests and responses on versioned /v1 endpoints. Common endpoints such as /v1/models, /v1/chat/completions, and /v1/responses. Server-sent events for supported streaming requests. Familiar request fields such as model, messages, input, stream, and tool definitions when the selected protocol supports them. Compatibility does not mean that one model can be moved freely between Chat Completions and Responses. A model may be exposed only through one verified protocol. Provider-specific reasoning, search, or multimodal fields may also require raw pass-through rather than gateway-side translation. Treat the live Models & Pricing (/pricing) catalog as the source of truth for the model, group, and API format you intend to use. Prepare the migration Before changing application code: Create a dedicated key in API Keys instead of reusing a personal or unrelated integration key. Select a primary model group that can access the intended model. Add ordered fallback groups only when they also support that model and fit the same cost and reliability policy. Record the current endpoint, model ID, streaming setting, and tool usage so the before-and-after behavior can be compared. The canonical Modelflare OpenAI-compatible base URL is: https://modelflare.dev/v1 Most SDKs expect the base URL to end at /v1 and append /chat/completions or /responses themselves. Check the client's documentation before adding a second endpoint suffix. Verify authentication and model access first Store the key in an environment variable rather than source code: export MODELFLARE_API_KEY='YOUR_MODELFLARE_API_KEY' Then confirm that the key can list its available models: curl -sS https://modelflare.dev/v1/models \ -H "Authorization: Bearer $MODELFLARE_API_KEY" A successful response proves that the hostname, TLS path, and key are valid. It does not yet prove that a particular request format is valid for every returned model, so the next check must use the intended endpoint. Send one request with the intended protocol For a Chat Completions model listed as compatible with that format: curl -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "YOUR_CHAT_MODEL", "messages": [ {"role": "user", "content": "Reply with the active model name."} ], "stream": false }' For a Responses-compatible coding model: curl -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.6-sol", "input": "Reply with the active model name.", "stream": false }' Use the exact model ID shown in the live catalog. A model_not_found result usually means the selected key or group cannot access that model, not that changing letter case will fix the request. Validate streaming separately Non-streaming success is not enough for an application that depends on streaming. Repeat the request with "stream": true, confirm that events arrive incrementally, and make sure the client does not buffer the complete response before rendering it. When diagnosing a slow stream, separate: time spent authenticating and selecting a route; time until service provider response headers; time until the first effective content or tool delta; generation throughput after visible output begins. Modelflare usage logs retain request-level timing metadata for this purpose without storing prompts, response text, raw request bodies, API keys, emails, or plaintext IP addresses. Production migration checklist Keep the API key in a secret store or environment variable. Pin the base URL to https://modelflare.dev/v1 (https://docs.modelflare.dev/guides/endpoints/). Use a model that explicitly supports the chosen endpoint. Test non-streaming and streaming modes independently. Test tools, structured output, reasoning controls, and multimodal input if the application uses them. Preserve explicit zero or false values when they carry meaning in the client request. Set client timeouts for the real workload instead of a short health-check prompt. Review usage logs after cutover for status, latency, token usage, selected group, and cost. Once the protocol boundary is verified, the client can usually keep its existing request lifecycle while Modelflare provides model access, routing policy, and per-request visibility behind the compatible endpoint. For the operational layer behind that endpoint, read What Is an AI API Gateway? (/articles/what-is-ai-api-gateway). If the client can use more than one OpenAI format, compare Responses API vs Chat Completions (/articles/responses-api-vs-chat-completions) before changing paths. ### Reliable AI API Routing & Diagnostics Canonical URL: https://modelflare.dev/articles/reliable-ai-api-routing Reliable AI API routing is not one retry loop. It means giving each API key a clear primary group, fallback groups, model capabilities, and request limits, while retaining enough user-facing evidence when a request slows down or fails. A good routing design makes the result explainable and prevents a group switch from silently changing the requested model or billing policy. Start with an explicit key policy Modelflare supports two routing approaches: A regular API key uses one explicit primary group and an ordered list of fallback groups. A Smart API Key evaluates the groups currently available to the account and continues through eligible candidates according to its strategy. Fallback order is not traffic splitting. It is a priority sequence. When the primary group is unavailable, requests continue through the later groups in order. Create separate keys for separate applications or environments. This makes group access, quota, expiration, and usage records easier to reason about than one shared key used everywhere. A fallback must preserve the request contract Before adding a group to a fallback list, verify that it: exposes the requested model; supports the same endpoint and streaming behavior; permits any required Service Tier or provider-specific field; has an acceptable usage multiplier and request-rate limit; is actually unlocked for the account. A fallback group is not a license to substitute an arbitrary model. The requested model and protocol still define the wire contract. Understand group request limits Request-rate limits are applied according to the account and the group that handles the request. Separate API keys make usage easier to analyze, but they do not automatically bypass account or group limits. When the current group reaches its limit, an API key with fallback groups can continue to the next available group. If no fallback is available, the request receives 429. Client retries should use bounded backoff rather than immediately creating a burst of new requests. Use request-level performance metrics Usage logs expose the product-level metrics needed to understand a request: Metric | What it helps explain | Total response time | Time from submitting the request until completion | First response | Time until the first effective text, reasoning, or tool event | First visible text | Time until text appears for requests expected to produce text | Visible output speed | Generation speed after visible output begins | Output tokens | The amount of output produced by the request | A slow first response usually means the request waited longer before generation began. A normal first response followed by low output speed points more toward the generation phase. Combine these metrics with status, model, group, and time range to distinguish a one-off delay from a sustained issue. Tool-call-only responses may not contain visible text, so “first response” is the safer first-effective-output signal for Responses traffic. Use evidence that is safe to retain Modelflare timing metadata does not include prompts, response text, raw request bodies, API keys, emails, or plaintext IP addresses. This keeps the operational record useful without turning latency diagnostics into a second content store. For an incident, capture: request ID and timestamp; requested model and selected group; endpoint and streaming mode; downstream status; status, total response time, and first-response timing; whether the client cancelled before the request completed. That evidence lets support verify group switching, slow model generation, and client cancellation without exposing internal service topology in ordinary logs. Production reliability checklist Give every application its own API key. Keep the primary and fallback order intentional. Verify model and protocol support in every candidate group. Set a client timeout long enough for the real workload. Use bounded retries with jitter for retryable failures. Do not retry authentication, quota, or model-access errors as if they were transient. Test both non-streaming and streaming requests. Monitor 429, first response, output throughput, and cancellation. Review group pricing before treating a fallback as equivalent. Reliability comes from preserving the request contract while retaining enough evidence when a request fails. Ordered fallback groups reduce dependence on one group, while performance and usage records make remaining failures explainable. Use the AI API Error Guide (/articles/ai-api-error-troubleshooting) to interpret authentication, access, rate-limit, cancellation, and 5xx failures. Apply the controls in AI API Key Security (/articles/api-key-security-cost-controls) before expanding fallback access. ### Responses API vs Chat Completions Canonical URL: https://modelflare.dev/articles/responses-api-vs-chat-completions Responses API and Chat Completions both send prompts to language models, but they organize input, output, tools, and streaming differently. Choosing between them should start with the client and model contract—not with the assumption that a newer endpoint automatically works for every provider. The shortest decision rule is: Use Responses API for a coding agent or application that already expects Responses items, tool events, and the Responses streaming lifecycle. Use Chat Completions for broadly compatible chat clients and for provider families that expose their OpenAI-style models through raw Chat Completions pass-through. Always verify the selected model's supported API format in Models & Pricing (/pricing). Protocol comparison Question | Responses API | Chat Completions | Primary input shape | input and typed input items | A messages array | Output model | Typed output items and events | Assistant message choices and deltas | Streaming | Responses event stream | Chat completion chunk stream | Tool activity | Typed tool-call and tool-output items | Tool calls attached to assistant messages | Best fit | Agents, coding tools, and Responses-native applications | Chat clients and widely supported OpenAI-compatible providers | Model portability | Only models verified for Responses | Only models verified for Chat Completions | The table describes the wire contract. It does not imply that Modelflare converts every provider feature between the two formats. When Responses API is the better fit Choose /v1/responses when the client already treats a model run as a sequence of typed items rather than one assistant message. This is common in coding agents that need to distinguish visible text, reasoning summaries, function arguments, custom tool input, and other events. A minimal request looks like: curl -sS https://modelflare.dev/v1/responses \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.6-sol", "input": "List the three checks to make before an API migration.", "stream": true }' Responses streaming should be validated end to end. A client that understands Chat Completions chunks but not Responses events can connect successfully and still fail to render useful output. When Chat Completions is the safer choice Choose /v1/chat/completions when the application is built around system, user, assistant, and tool messages, or when the selected provider family documents an OpenAI-compatible Chat Completions endpoint. curl -sS https://modelflare.dev/v1/chat/completions \ -H "Authorization: Bearer $MODELFLARE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "YOUR_CHAT_MODEL", "messages": [ {"role": "system", "content": "Answer concisely."}, {"role": "user", "content": "What should an API health check verify?"} ], "stream": true }' For non-OpenAI provider families, Modelflare can expose raw Chat Completions pass-through so private request fields such as provider-specific reasoning or search controls reach the service provider service unchanged. That behavior is intentionally narrower than claiming full Responses compatibility. Do not choose by model name alone Three separate checks are required: The API key can access the model. Access depends on the key and available model groups. The model supports the endpoint. A model in /v1/models is not automatically valid for both request formats. The client understands the stream. Responses events and Chat Completions chunks are different client contracts. If any of these checks fail, switching the endpoint path alone may turn a clear compatibility error into an empty or partially rendered response. Tool and structured-output migration Before moving an existing integration: compare the tool definition schema used by the client; verify how tool-call IDs and tool outputs are returned; preserve explicit optional values instead of dropping 0 or false; check whether the provider requires private fields that must pass through unchanged; test tool-only responses that may not contain a visible text delta; confirm how the client detects completion and usage. The same prompt producing similar text is not a sufficient protocol test. A real test should include the features the application depends on. A practical selection flow Open Models & Pricing (/pricing) and choose the model and group. Confirm the supported API format. Follow a client-specific guide in Modelflare Docs (/docs) when one exists. Send one non-streaming request. Send one streaming request. Exercise tools or structured output. Review the request in usage logs for status, timing, token usage, and cost. Responses API is not a universal replacement for Chat Completions, and Chat Completions is not obsolete. The correct format is the one jointly supported by the client, the selected model, and the service provider provider contract. After choosing the format, use the AI API Streaming Guide (/articles/ai-api-streaming-guide) to implement its event parser, timing metrics, cancellation, and timeout policy.