Why AI Agent Cache Hit Rates Collapse: GPT, Claude, and Auditable Gateways
A source-backed guide to third-party agent cache failures, GPT and Claude prompt-caching mechanics, reproducible hit-rate measurement, and Modelflare's public compensation boundary.
AI agent caching is not a checkbox. It is a protocol, routing, and accounting contract. This article explains why an opaque third-party agent can show a very low cache hit rate even when the upstream model supports prompt caching, how GPT and Claude measure reuse differently, and what an auditable gateway should guarantee.
The decision in one sentence
Do not put a production, cache-sensitive agent behind a third-party agent or gateway that cannot expose native cache usage, preserve a stable prefix, keep model and route affinity, and reconcile the result against a durable ledger. “Supports caching” is a feature claim; a measured, eligible hit rate is an operating fact.
This is not a claim that every third-party agent fails. Public documentation cannot establish a vendor-wide percentage without a fixed corpus, a fixed model, a known observation window, and request-level usage evidence. The reliable conclusion is narrower: an opaque translation layer creates several independent ways to turn a valid provider cache into an almost-always-cold path. Treat that lack of evidence as a reason not to use the path for cache-dependent production work.
Define the denominator before discussing a percentage
The provider usually reports three different buckets. Let R be tokens read from a cache, W be tokens written to a cache, and U be input tokens processed without reuse. For cache-eligible prefixes, the comparable hit rate is:
eligible_hit_rate = R / (R + W)
For all input sent to the model, the reuse share is:
input_reuse_share = R / (R + W + U)
Those numbers answer different questions. A dashboard that divides by all input can look low when the agent sends many short, ineligible requests. A dashboard that divides only eligible traffic can look high while a large dynamic suffix still dominates spend. Report both, plus the eligibility rule and the time window.
The provider fields are the evidence, not a green badge in an agent UI:
| Signal | OpenAI field | Claude field | What it proves |
|---|---|---|---|
| Reused prefix | input_tokens_details.cached_tokens |
cache_read_input_tokens |
Tokens actually served from a matching entry |
| New cache entry | input_tokens_details.cache_write_tokens |
cache_creation_input_tokens |
Tokens charged to create or extend an entry |
| Unreused input | input_tokens - input_tokens_details.cached_tokens - input_tokens_details.cache_write_tokens (derive carefully) |
input_tokens after the breakpoint |
Input outside the cached prefix; semantics differ |
| Correlation | request ID plus model and route | request ID plus model and route | Which provider attempt produced the usage |
Worked example, using synthetic numbers rather than Modelflare production telemetry: R=720,000, W=80,000, and U=200,000 give an eligible hit rate of 720,000 / 800,000 = 90%, but an all-input reuse share of 720,000 / 1,000,000 = 72%. Publishing only one of those values hides the denominator.
input_tokens_details.cache_write_tokens=0 is not proof of “no cache.” A request can be a cache read and therefore create no new entry. Conversely, a missing usage field in a third-party response is not a zero; it is an observability failure that must be recorded as not auditable.
Why an agent gateway loses hits even when the model supports caching
The failure is usually in the path between the application and the provider. These are the high-frequency breakpoints:
- Dynamic bytes arrive too early. Timestamps, request IDs, user names, experiment flags, or a changing “current date” in the system prompt alter the prefix before the reusable instructions.
- Tools are reserialized. Adding, removing, reordering, or nondeterministically serializing a tool schema changes the exact prefix. A harmless-looking JSON key order can become a cache miss.
- Fallback changes the cache identity. A load balancer that moves between model aliases, regions, organizations, or provider credentials does not share one universal cache entry.
- The adapter drops native controls. Removing
cache_control,prompt_cache_key, retention options, or usage details turns a provider feature into an invisible best effort. - The prompt is below the provider threshold. Short agent turns can be valid requests but still be ineligible for caching.
- The TTL window is crossed. A five-minute Claude entry or a model-specific OpenAI retention window can expire during a human-in-the-loop pause.
- Parallel warm-up races. Concurrent first requests may all arrive before the first response makes a cache entry available.
- History is rewritten. Summarization, compaction, truncation, or a different serialization of prior messages changes the prefix instead of appending to it.
None of these failures requires a dishonest vendor. They are predictable consequences of a gateway that treats an agent request as free-form text instead of preserving a provider’s cache contract. The practical warning is stronger than a marketing comparison: if the agent cannot show which breakpoint failed, you cannot price or debug cache-dependent traffic.
GPT and Claude use related ideas, not interchangeable semantics
Both providers require an exact reusable prefix, but the control surface and usage accounting are different. The table below summarizes the official guides checked on 2026-08-25; model names, minimums, and retention policies can change, so recheck the linked pages before changing a production policy.
| Dimension | OpenAI prompt caching | Claude prompt caching |
|---|---|---|
| Reusable unit | The full rendered context prefix, including instructions, tools, history, and multimodal parts | The ordered prefix through a cache_control breakpoint: tools, system, then messages |
| Minimum length | The current guide lists 1,024 visible tokens for GPT-5.6+ and generally 2,048 for older models | Model-specific minimums currently span roughly 512–4,096 tokens; shorter prompts run without caching |
| Controls | Implicit caching; explicit breakpoints and a stable prompt_cache_key on supported models |
Top-level automatic caching or explicit block breakpoints; up to four breakpoints and a 20-block lookback |
| Retention | GPT-5.6+ supports a 30-minute TTL; earlier models expose retention modes with provider-defined typical windows | Five-minute default TTL, refreshed on use; optional one-hour TTL at a higher write price |
| Price shape | On the current GPT-5.6+ guide, writes are 1.25× base input and reads are 0.1× | Five-minute writes are 1.25× and one-hour writes are 2×; reads are 0.1× |
| Usage evidence | input_tokens_details.cached_tokens and, when applicable, input_tokens_details.cache_write_tokens |
cache_read_input_tokens, cache_creation_input_tokens, and post-breakpoint input_tokens |
| Frequent invalidator | Model/tool/settings changes, machine overflow, or a prefix change before the breakpoint | Model/system/tools/message changes, a missing previous message, or a concurrent warm-up miss |
OpenAI notes that cache entries live on individual machines and that a prompt_cache_key helps group and route requests but does not pin a machine or guarantee a hit. Claude documents exact matching, workspace-level isolation, and explicit diagnostics such as tools_changed and messages_changed. A gateway that hides those distinctions cannot honestly convert provider documentation into a single “cache rate” promise.
The price multipliers explain why this matters. If one write is followed by one read, the cached input portion costs about 1.25 + 0.10 = 1.35× the uncached input unit on the cited OpenAI model, rather than two uncached units. Ten fully reused requests cost about 1.25 + 9×0.10 = 2.15× instead of 10×. These are official illustrative calculations, not a Modelflare invoice or a promise for every model.
The adapter tax is measurable
Use a failure matrix instead of a screenshot. Keep the application prompt, model, credentials, and request rate fixed; change one variable at a time and retain the raw usage object.
| Controlled change | Expected provider signal | What an opaque agent may hide | Release interpretation |
|---|---|---|---|
| Append only the user turn | R rises after the first W |
Rewritten history or a new route | Prefix preservation works |
| Add a timestamp to the system prompt | R falls to zero or a new W appears |
Which bytes changed | Dynamic prefix is a cache breaker |
| Reorder one tool property | tools_changed or a new write |
Serializer behavior | Tool schema must be deterministic |
| Split traffic across aliases or machines | Lower and more variable R |
Selected route and cache key | Affinity policy is missing |
| Sleep past the documented TTL | New write after expiry | Expiry time and retention mode | Measure human-pause workload separately |
| Fire two identical first requests in parallel | One or both may write | Warm-up race and attempt order | Do not infer capacity from a cold burst |
The test must retain the request ID, stream mode, first event time, exact model, selected route, cache fields, and UTC timestamp. Redact prompts, keys, and customer content. If a gateway returns only a normalized “input tokens” total, label the run not auditable; do not turn missing dimensions into a fabricated zero or a favorable estimate.
A reproducible audit record
The following is a small, synthetic record format. It is deliberately provider-neutral at the envelope and preserves native fields inside usage. It is not a benchmark result.
{"request_id":"demo-001","timestamp":"2026-08-25T02:00:00Z","model":"MODEL_ID","route":"route-a","stream":true,"usage":{"input_tokens":1000000,"input_tokens_details":{"cached_tokens":0,"cache_write_tokens":1000000}}}
{"request_id":"demo-002","timestamp":"2026-08-25T02:00:03Z","model":"MODEL_ID","route":"route-a","stream":true,"usage":{"input_tokens":1000000,"input_tokens_details":{"cached_tokens":720000,"cache_write_tokens":80000}}}
Run at least five phases: a serial warm-up, an append-only conversation, a dynamic-system mutation, a tool-order mutation, and a TTL-gap replay. For Claude, map the native read/write fields into the same report without replacing them; for OpenAI, retain both cached and write fields when returned. Compare R/(R+W) and R/(R+W+U) per phase, route, model, and UTC day. A single blended number is not enough to decide whether an agent is safe for production.
What Modelflare guarantees—and what it does not
Modelflare’s public status page currently describes an OpenAI Cache Hit Rate Guarantee. For eligible requests that satisfy OpenAI’s caching requirements and form a valid cache, the daily rate is calculated by UTC calendar day. If the measured rate falls below the applicable tier, the shortfall is compensated and shown in Dashboard → Token Usage Analysis. The public tiers currently shown are 65%, 75%, and 85% for the published group levels.
The boundary is intentional:
- The guarantee applies to eligible, valid OpenAI cache traffic, not to a short or constantly changing prompt that never qualifies.
- It does not turn a changing model, tool schema, route, or TTL into a reusable prefix.
- It is calculated per UTC calendar day and credited through the documented account surface; it is not an assertion that every individual request will hit.
- The native usage evidence and eligibility calculation remain visible requirements. A compensation rule is useful precisely because the denominator is explicit.
That is the difference between an auditable service promise and a third-party agent slogan: the former names the eligible population, observation window, threshold, and credit destination.
Price advantage and operating legitimacy are separate proofs
The current public pricing page also displays a 0.015 first-top-up offer ratio for the applicable campaign/group, with the page’s stated minimum single top-up and eligibility terms. This is a price incentive; it must not be conflated with the cache guarantee. Check the live pricing page for the exact model scope, campaign status, and settlement rules before making a purchase decision.
Modelflare’s public trust and legal materials identify Havenbyte LLC as the operator and describe the operation as United States based. They list Stripe as the payment processor for payment handling, billing, invoices, refunds, and fraud controls. These are useful custody and accountability signals, not a substitute for reading the terms or testing the API. The article makes no claim about a particular state registration, certification, or provider affiliation beyond those public materials.
Recommendation for production agents
Choose a third-party agent only when it can pass this checklist:
- It forwards the provider’s native cache controls and returns native read/write usage.
- It keeps the stable instruction and tool prefix byte-identical, then appends dynamic state.
- It exposes the selected model, route, organization/region boundary, TTL mode, and request ID.
- It records serial and parallel warm-up behavior instead of averaging them away.
- It defines eligible traffic, denominator, UTC window, threshold, and compensation in writing.
- It lets you export request-level evidence without exposing secrets or customer content.
If any of those answers is “no,” do not use the path for a workload whose economics depend on cache reuse. Use a direct provider path for the experiment, or choose an auditable gateway such as Modelflare and verify the exact route with the same corpus. For adjacent controls, see reliable AI API routing, AI API cost tracking, pricing, and trust.
Sources and verification date
The following primary sources were checked on 2026-08-25. They describe provider contracts and Modelflare’s current public policy; they do not establish a universal hit rate for an unnamed third-party agent.
- OpenAI Prompt Caching guide
- OpenAI API pricing
- Anthropic Prompt Caching guide
- Anthropic cache diagnostics
- Modelflare live status and cache guarantee
- Modelflare pricing and current offer terms
- Modelflare trust and operating entity
FAQ: Is a low third-party hit rate always fraud?
No. It may be a valid result of short prompts, a changing prefix, route fan-out, TTL expiry, or a missing usage field. The correct finding is “not reproducible” or “not auditable” until the controlled test shows the cause.
FAQ: Can I compare Claude and GPT with one number?
Only after normalizing the denominator and preserving each provider’s native fields. Compare the same corpus and workload phase, not a blended dashboard total.
FAQ: Does compensation remove the need to design a stable prefix?
No. Compensation limits the financial downside for eligible OpenAI cache traffic; it cannot make an ineligible request reusable or repair an agent that rewrites the prefix.