JEV - How far "can't hallucinate" actually goes

TypeSafe says Jev can't hallucinate. The sentence only holds for type: an answer cannot leave the table fixed in advance. Calibration and correctness are separate, and the workflow eval lines up with the average of Astra and Fable.

Folkbench folkbench.com

Co-author: folkbench.com (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.)

Ten pieces in this series:

TypeSafe says Jev "can't hallucinate". The sentence sounds like hallucination has been banned in one stroke. The same piece narrows it. What they wrote is that schema matching is a guarantee, not a statistic, so type errors can be written as 0%. One counterexample would be enough to refute that. They say it is mathematically impossible.

The only layer that sentence can stand on is type. Jev gave up string generation. It hands back a structure fixed in advance, not a passage of text generated on the spot. Whether the probability is accurate, and whether the judgment is right, are two other matters. If a reader does not split them, "it did not write prose" gets heard as "the answer is correct".

It cannot leave the table

There are only three question types, and all of them are sealed. Noul asks whether one thing holds, and what comes back is a probability from 0 to 1. Choice options have to be written by the caller in advance, at most 255 on one table. What comes back is the chosen option, with a probability on every option and a confidence. Score levels also have to be written in advance, at least two and at most ten. The score can land between two levels. The answer carries the probability of each level and a confidence at the same time. That score is still a position on the scale, not a fresh paragraph.

It cannot leave the option table. The model also cannot suddenly write a piece of prose and talk the answer into shape. The docs say this straight. What the model hands back is a distribution over the options or levels the caller submitted. It does not hand back a value outside that distribution. Code can use these values directly, without first digging fields out of a generated paragraph.

Chat models usually do not hand work in this way. They hand in strings. A string might be a normal reply, and it might be a hallucination. Software still has to parse it and validate it. If validation fails, the program cannot use it directly. TypeSafe puts type safety and hallucination together, and treats type safety as the minimum condition for automation. If the type is wrong deep in a call chain, later branches connect to the wrong place. The official writing also says that current models, however smart, still hallucinate, and still make type errors.

Staying inside the type does not stop the judgment from being wrong. Choosing the wrong item on a closed question is still wrong. On the public security-alert flow, the model has to choose among close it, hand it to an analyst, and contain it now. All three options are on the table. If the model chooses the wrong one, the type is still legal.

The official account does not call System One always correct. They wrote that System 1 thinking gives the impression of being error-prone. They believe System One models can be made more reliable than the alternatives. They leave the specific reasons for later. That is their direction. It is not "always correct".

Choice has another hard edge. Once the options pass 255, one table cannot hold them. A Wikipedia race that walks links has to pick among hundreds or thousands of links at a step. They switch to two stages: score independently, then make a choice. It occasionally slows down. That slowdown showed up in the Wikipedia-race demo. High cardinality is not infinite. "No hallucination" here still only means it will not hand back a link that is not on the table. Options cannot be added without a limit.

Confidence is a separate question

Calibration is a separate matter. The official account is that answers carry probability and confidence. When confidence is higher, accuracy is higher. Similar inputs return similar answers. On the other side, they write about chat models. Even if you prompt one to report a confidence, it is often overconfident, and unstable. If a task can be done right 95% of the time and the model does not say which 5% it is unsure about, that task cannot be automated.

The confidence on Choice and Score is not a separate sentence of certainty the model reports. The number is computed from the shape of the distribution. Probability piled on one option or one level, and the number is high. Probability spread across several places, and the number is low. The caller acts when the number is high, and hands the case to a person when it is low. Noul does not attach a separate confidence. The yes-or-no probability is itself the signal. The official comparison table writes probability and confidence together. Applied to the three questions, Choice and Score have both, and Noul has only the probability.

A confidence of 1.0 only means the distribution is pressed entirely onto one result. It describes the shape of this answer. When the distribution is pressed entirely onto the wrong option, confidence can still be high. A large number does not mean the result is right.

A type error cannot come out. That is mathematically impossible. Calibration has no guarantee of that kind. What the official account gives is a correspondence: higher confidence, higher accuracy. If that correspondence breaks, the type still cannot escape, and confidence can no longer be a threshold. So this layer has a mechanism, and it has a claim. Whether errors are fewer when the number is higher, and whether similar inputs come back with similar distributions, both have to be looked at separately. The field being present every time only means the caller can read the number.

Right is not the world

Right and wrong have to be split once more. The workflow eval measures how close a model comes to a reference inside the same flow. That reference is not an objective answer from the world. They assume the flow's code is right, and they do not argue about whether the labels are right. The reference labels are the average of GPT-6 Astra and Claude Fable 5.1. Both use high thinking, and each answers every question in the flow. The other models are compared at each vendor's default reasoning.

Jev coming close to that average means coming close to the consensus of those two models. It cannot be written as truer than the facts. If the consensus leans, an answer stuck to the consensus leans too. The official writing says that using the average of these two as the reference biases answers toward OpenAI's and Anthropic's models.

During the comparison, the large models are wrapped in the System One LLM adapter, which forces them to emit structured decisions too, so they can be compared inside the same question types. The official account admits that asking with probabilities is usually slower and more expensive than giving a decision without probabilities. Their conclusion is that this is the most accurate way to take a decision from a large model. That accuracy is compared with a decision that does not carry probabilities. It does not mean the decision matched an external fact.

The type-error numbers for large models on the chart come from OpenRouter, and they are biased. More complex queries may be routed to a stronger model. Jev's 0% is not the same kind of statistic. The official writing says their number is not an empirical result. Schema matching is guaranteed, so the chart can be filled with 0%. One side is an observation after routing. The other side is a number filled in from a guarantee. These are not the same error rate.

The side-by-side with GPT-5.6 Terra used default reasoning. The official account picked that model because, as they read it, its intelligence is on average the closest to Jev. In the recorded run, the only disagreement is "Churn likelihood level". The official view is that the question is ambiguous and the answer cannot be said cleanly. On an ambiguous question, a distribution is more honest than a hard label.

"Can't hallucinate", in the end, means it does not produce output outside the type. That is why they write type errors as 0%. Calibration has to be accepted on its own, by whether confidence can be used as a threshold. Whether the judgment is right, this flow eval cannot give the world's own answer. What it lines up with is the average of Astra and Fable.

Sources

Series