AI API Latency Metrics: TTFT, First Response, and Output Speed

AI API latency is not one number. For a streaming request, record when the upstream headers arrive, when the first non-empty SSE event arrives, when the first meaningful content or action appears, when the first visible text appears, and when the response completes. These milestones answer different questions and should not all be labeled “TTFT.”

For a text-only chat, time to first visible text is usually the user-facing latency. For a reasoning or tool-using workflow, the first effective output can occur earlier as reasoning metadata or function-call arguments. Generation speed is a separate measurement taken after output starts.

Build one request timeline before naming a metric

Use a monotonic clock and one origin for all client- or gateway-level timestamps. A streaming request can be represented as follows:

t0  gateway receives request
t1  gateway starts upstream request
t2  upstream response headers arrive
t3  first non-empty SSE event arrives
t4  first effective content or action arrives
t5  first visible text delta arrives
t6  upstream body completes or closes
t7  gateway handler finishes

Not every request has every milestone. A non-streaming response has no useful SSE timeline. A function-call-only response may never produce visible text. A cancelled stream may have a start but no normal completion. Store absent values as absent rather than as zero, because zero would falsely imply that the event happened immediately.

Modelflare records request-level metadata under other.timing. The fields map to the timeline like this:

Field	What it measures	Important boundary
auth_ms	Token and user authentication duration	Local phase duration, not upstream time
distribution_ms	Channel selection and routing-policy duration	Local phase duration
body_read_ms	Time spent reading the downstream request body	Can expose a slow upload before routing
upstream_headers_ms	Upstream request start to response headers	Its origin is upstream start, not initial gateway receipt
first_sse_event_ms	Gateway receipt to first non-empty upstream SSE event	An event can contain metadata rather than useful output
first_response_ms	Gateway receipt to first effective content or action delta	Includes visible text, reasoning summary, function arguments, or custom tool input
first_text_delta_ms	Gateway receipt to first visible text delta	Exists only when visible streaming text appears
upstream_done_ms	Gateway receipt to upstream body completion or close	A close is not automatically a successful completion
total_handler_ms	Gateway receipt to the latest known relay completion point	Closest gateway-side end-to-end duration
visible_output_tps	Output tokens divided by time from first visible text to upstream completion	A visible-output diagnostic, not system-wide TPS

This mapping prevents a common error: subtracting timestamps that do not share an origin. For example, upstream_headers_ms is a duration beginning when the upstream request starts, while first_response_ms begins when the gateway receives the request.

TTFT has more than one practical meaning

The conventional definition of time to first token is the duration from query submission to the first received output token. The NVIDIA NIM benchmarking guide notes that this generally includes network latency, queueing, and prompt prefill, and that empty initial responses should not count.

That definition works for a text-generation benchmark, but modern API streams can emit several things before visible text:

• a response-created or metadata event;
• a reasoning summary delta;
• function-call argument fragments;
• custom tool input;
• an empty heartbeat or provider-specific envelope.

Use explicit names in dashboards and incident reports:

Name	Recommended meaning	Best use
Time to headers	Request start to upstream headers	Network, proxy, and upstream admission diagnosis
Time to first event	Request start to first non-empty SSE event	Transport liveness only
Time to first effective response	Request start to useful content or action	Agent and reasoning workflow responsiveness
Time to first visible text	Request start to text a user can render	Chat and user-perceived responsiveness

If a chart says only “TTFT,” document which row it means. Otherwise two teams can report different values for the same request and both appear correct.

Separate responsiveness from generation speed

End-to-end latency measures the whole request from submission until the full response is received. It includes queueing, prefill, decoding, network transfer, tool pauses represented within the upstream stream, and gateway work visible inside the chosen boundary.

For a simple completed text stream:

end_to_end_latency = final_response_time - request_start_time
visible_generation_window = final_response_time - first_visible_text_time
visible_output_tps = output_tokens / visible_generation_window_seconds

Inter-token latency, also called time per output token, is normally the average spacing between output tokens after the first token. NVIDIA’s AIPerf definition excludes TTFT and divides the remaining duration by output_tokens - 1. Its reciprocal approximates per-user decode throughput for sufficiently long outputs.

Do not calculate token-level ITL by counting SSE events or text Delta objects. One event can contain zero, one, or several tokens, and token boundaries depend on the model tokenizer. Modelflare’s visible_output_tps uses the recorded output-token count and the visible generation window; it does not claim to be a per-token event-spacing measurement or total system throughput.

These metrics answer separate questions:

• low first-text latency with slow output speed feels responsive at first and then sluggish;
• high first-text latency with fast output speed feels stalled and then finishes quickly;
• low per-user output speed does not prove low system throughput under concurrency;
• high aggregate TPS does not guarantee good latency for one user.

Reasoning and tools can precede visible text

In a text-only flow, first_response_ms and first_text_delta_ms may be nearly identical. In a reasoning or function-calling flow, the gap can be meaningful.

Suppose a model emits function-call arguments at 1.8 seconds, the application or provider-side tool flow continues, and visible text begins at 6.4 seconds. The system has produced an actionable result at 1.8 seconds, but the user has not seen prose until 6.4 seconds. Calling both values TTFT hides whether the delay occurred before planning, during a tool step, or before answer rendering.

Interpret the fields according to the product experience:

• for a terminal agent, first effective action may be the best responsiveness signal;
• for a chat UI that displays only prose, first visible text is the relevant user signal;
• for a tool-call API consumed by code, visible text may be absent and should not be required;
• for a reasoning model, an early SSE envelope proves connection progress but not useful progress.

The AI API Streaming Guide covers SSE parsing, cancellation, and idle-timeout behavior. Latency metrics should be computed only after the parser distinguishes envelopes from effective output.

Diagnose the slow stage in a fixed order

Start at the upstream boundary, then move toward local phases. This avoids blaming authentication or routing when the request spent most of its time waiting for the model.

Symptom	Primary metric	Likely layer to inspect	Next check
Slow before headers	upstream_headers_ms	Network path, upstream admission, provider queue, proxy route	Compare channel, region, status, and concurrency
Headers are fast, useful output is late	first_response_ms minus early phases	Model queueing, prompt prefill, reasoning, upstream scheduling	Compare input tokens, model, route, and inflight count
First event is fast, effective output is late	Gap from first_sse_event_ms to first_response_ms	Metadata-only events, heartbeats, reasoning startup	Inspect safe event types, not raw content
Effective action is fast, visible text is late	Gap from first_response_ms to first_text_delta_ms	Tool/reasoning phase or response composition	Check requested tools and output mode
Text begins quickly, then crawls	visible_output_tps and upstream_done_ms	Decode throughput, contention, long context, network backpressure	Compare output length and channel inflight count
All upstream metrics are normal	auth_ms, distribution_ms, body_read_ms	Local auth, policy selection, client upload	Inspect only the phase that is elevated

No single metric proves a root cause. High upstream_headers_ms, for example, groups several possibilities that require route, region, provider, and concurrent-load evidence to separate.

Read three synthetic traces

The following values are illustrative diagnostics, not Modelflare production averages or provider benchmarks.

Trace A: waiting before upstream headers

Metric	Value
upstream_headers_ms	6,100 ms
first_sse_event_ms	6,300 ms
first_response_ms	6,350 ms
first_text_delta_ms	6,400 ms
total_handler_ms	9,200 ms
visible_output_tps	42

Most of the wait happens before headers. Generation after visible text is comparatively healthy. Check the selected upstream route, provider admission, network path, region, and concurrent load before optimizing the client renderer.

Trace B: useful action before visible prose

Metric	Value
upstream_headers_ms	240 ms
first_sse_event_ms	310 ms
first_response_ms	2,900 ms
first_text_delta_ms	8,700 ms
total_handler_ms	10,200 ms
visible_output_tps	55

Transport is alive early and an effective action appears at 2.9 seconds, but visible text waits another 5.8 seconds. If the request uses reasoning or tools, inspect that phase. Increasing a header timeout would not address this pattern.

Trace C: fast start, slow generation

Metric	Value
upstream_headers_ms	260 ms
first_sse_event_ms	330 ms
first_response_ms	420 ms
first_text_delta_ms	430 ms
total_handler_ms	20,430 ms
visible_output_tps	9.8

The request becomes visible quickly but spends about 20 seconds generating. Compare output length, context length, selected channel, inflight count, and provider behavior. A first-output timeout would pass and therefore cannot detect this failure mode.

Compare latency only under controlled conditions

A fair comparison holds the workload contract stable. At minimum, record:

• exact model identifier and route or group;
• input-token and output-token distributions, not just averages;
• streaming versus non-streaming mode;
• enabled reasoning effort and tools;
• region and client-to-gateway network path;
• concurrency or arrival rate;
• sampling settings and maximum output length;
• warmup policy, retry policy, and excluded failures;
• sample size, time window, and percentile method.

Compare p50, p95, and p99 rather than presenting one average. Keep failures visible: removing timeouts and errors can make a route look faster precisely when it is less reliable. Never compare two providers with different prompts, output lengths, concurrency, or endpoint behavior and present the result as a model-speed ranking.

Retain diagnostic metadata without retaining content

Latency diagnosis does not require storing prompts, responses, API keys, or plaintext client identities. A useful request record can contain:

• a generated request ID and timestamp;
• model, group, and selected channel reference;
• status and terminal outcome;
• input, output, and cached-token counts when available;
• the timing fields above;
• stream event and text-delta counts;
• channel inflight count;
• coarse region and a privacy-reviewed network trace identifier;
• a coarse classifier such as upstream_headers_slow or generation_slow_tps.

Apply retention limits and access controls even to metadata. Request IDs, route choices, timing patterns, and token counts can still reveal operational behavior when combined.

Begin an investigation with the request timeline, not a single TTFT label. Then use Reliable AI API Routing to compare the exact route selected for each attempt and AI API Error Troubleshooting to pair timing with the terminal status. This produces evidence that can separate admission delay, model startup, reasoning or tool work, visible generation, and local gateway overhead.