JEV - Why a Jev that cannot chat fits an agent

An agent is often missing a judgment that can enter a branch, not another stretch of long text. This piece explains the three questions, what the workflow eval is comparing, and why a hot path cannot wait out a chat.

Folkbench folkbench.com

Co-author: folkbench.com (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.)

Ten pieces in this series:

What an agent is often missing is not another stretch of long text. It is a judgment that can enter control flow directly. The usual ones are whether this alert stays closed, whether this invoice gets paid, whether this trace needs a person, and whether the next customer-service sentence should be escalated. These nodes are not there for the model to write an analysis. Code needs a value it can compare and branch on. That is what Jev does.

TypeSafe's formulation means a function call of frontier intelligence. State comes in. A typed probabilistic decision goes out. It does not generate strings. The sentence for a person, and the judgment for a program, are not the same exit.

Fix the output first

The output of current large models is a string. Software that needs to continue still has to parse it, validate it, and guard against drift. An extra field appears. The paragraph looks complete, and the enum does not match. A person in a chat box notices and asks again. Code does not notice, and walks on with the error. Buried several calls deep, a smarter model cannot put it back.

Jev fixes the output space in advance. You say what it is allowed to return, and it only assigns probability inside that space. A type error cannot happen, mathematically. It cannot hand back a shape you did not define. That is not the same thing as the judgment being correct. The probability can lean the wrong way, and the option can be the wrong one. Type safety closes the shape, not right and wrong. Once the shape is fixed, the branches after it can be written. You do not have to fish a value out of a paragraph first.

There are only three questions.

Noul is yes-no. The model gives a probability from 0 to 1. Near 1 is yes. Near 0 is no. Near 0.5 is unsure. It does not attach a separate confidence. The probability is the signal. Choice is single choice. You list the options first, with a cap of 255. What comes back is the chosen option, the distribution over every option, and a confidence. Code uses that distribution to decide whether it acts or hands the case to a person. Score is levels. The levels run from 2 to 10, and you write what each level means. What comes back is a score, the distribution over the levels, and a confidence. The score can land between two levels, as a position on the scale.

A yes-no is written as a condition. Single choice and levels are different: the first looks at which option it landed on, the second thresholds the score. A good question is still narrow. A judgment that a person who knows the domain can make in a second after reading the material is the kind to give it. Whether the customer is asking for a refund is that kind of question. Reading the whole letter and then deciding the best action is not. The second sentence wants slow reasoning. Split it, then ask. The split questions face the same state, independent of each other, and come back together in one request. The weights stay in code. When policy changes, you change the numbers. You do not rewrite the whole prompt.

The fork is what stalls people

What actually stalls people is often a node of this shape. An alert arrives with records the machine already has, and the conclusion is close it, send it to an analyst, or isolate now. An invoice hangs on an order and a delivery record, and someone has to decide pay, hold, or send it back. A customer-service agent has finished, the tool calls are all in the trace, and someone has to decide whether this trace gets looked at, and how soon. The customer writes again, and the thread and the account state are both there. How the next sentence should continue, and whether it should escalate, is the same shape of question.

The part rules can freeze is code. What is brittle is the exceptions that cannot be finished. The receipt matches, the reason is vague, and one step in the trace looks odd. Handwritten logic shatters on these. You stuff the whole policy into one prompt and let the model think it through once, and you write fewer conditionals. The exit turns back into a piece of text. For text to enter control flow, you write another layer of parsing. That layer can be wrong on its own.

TypeSafe's workflow eval measures this way of connecting. The eval splits a task into many narrow questions. What code can decide, code decides. The model only answers the judgments code cannot settle. The four public flows are a security incident, agent-trace observability, invoice handling, and customer service. Under the same flow, a model on the workflow is more accurate than stuffing the whole policy into one prompt, and it costs less and takes less time. Averaged across the four tasks, the models under test move in that direction.

The actions after that follow the probability, not a label that has been slammed shut. The result that leaves the system is still one discrete action. The engineering in the middle has to be done the same way every time.

The eval measures closeness

This eval does not argue with you about whether the flow itself was written wrong. It assumes the harness is right. The reference answer is not a gold label marked by hand, question by question. GPT-6 Astra and Claude Fable 5.1 answer every question under high thinking, and the two answers are averaged. Other models use the vendor's default reasoning. Only after the flow is fixed can they be compared. The eval compares how close a model comes to those two large models' judgments, plus speed and cost.

So the chart does not prove that Jev understands the business better than Astra. Astra and Fable are the ruler here. Jev has to come close to their judgments, and it also has to open a gap on latency and cost. If the questions were split wrong, a closer ruler does not help. Splitting the questions is the work of the person who writes the flow.

Jev sits far out on the frontier of "fast and cheap". The higher multiples on the official site, about 193.6 times faster and 444.6 times cheaper, are the high end of this eval. That multiple is not every call. The calls here are closer to the automation load you would actually ship.

What it comes close to is the probability on those narrow questions after the split, not the writing of a long analysis. Jev does not write that long analysis.

A hot path cannot wait

Latency is hard for an agent. A person can still wait three seconds. Once layers wrap layers, they cannot. The same chain also has retrieval, a database write, and the next call, and every hop spends the same time. Inside 70 to 500 milliseconds, a judgment can sit on a hot path. Outside that range, the call stack treats it as blocking.

Talking to a person, current frontier models commonly take from 3 seconds to more than 300 seconds end to end. That speed makes sense for a copilot, or for a coding agent with a person watching. As a condition on a hot path, that speed does not. Jev does not emit a sentence token by token. The probabilities that should come back arrive together. The speed comes from there.

TypeSafe also uses Jev to check. Prompts, reasoning traces, and outputs from other models, Jev scores them, and it also acts as a guardrail and looks for jailbreaks. Generation stays with the chat model. Whether this gate is passed is decided by a model that does not generate strings. That is a joint in an agent, not chat. If the nod and the review still sit on long text, you have to find another program to read it. Jev closes the result into probabilities and levels, and the review can be written as code.

The sentence written for the user is still the chat model's job. Jev does not take that sentence. It takes the routing in front, and the check behind. What an agent is missing here is not one more, longer explanation. It is a judgment whose type is already fixed, so code can walk with it.

Sources

Series