JEV - Some questions do not belong to it

After using it, the questions that do not belong are explanations, replies for a person to read, inventing a name in an open set, and keeping a reasoning trace. What does belong is still a closed yes-no, category, level, and whether to escalate.

Folkbench folkbench.com

Co-author: folkbench.com (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.)

Ten pieces in this series:

After using it, what I am clearest about is not what Jev can do. It is which kinds of talk I should not ask it.

I take it as a joint, not as a mouth. Instinct can only answer questions that instinct can answer. What has to be unfolded, or what has to lay the reasoning out for a person to see, is the wrong object if you ask Jev. Speaking something through to a person is also the wrong ask. It has no strings. Ask, and there is nowhere to put the answer.

What I should not ask

If you are also coming over from a chat model, the easiest things to ask wrong are the kinds below.

I should not ask it to explain why. The habit I brought from chat is to open by asking it to explain. That habit does not work on it. When I ask it to explain, what arrives is only a judgment. A reason has to be written as sentences. It cannot write them. The return has no "because". If someone needs to see how the cause arrived, it cannot give that. If I really need a stretch of reasons, I write it myself, or I ask a model that can speak, and have it write from the judgment already made.

I have asked it, wrongly, to write a reply for a person to read. What has to be sent is a passage. Which sentence comes first, and how heavy the tone is, lives in the words. What it gives is options and probabilities. Those do not match the passage that has to be sent. However high the probability, what comes back is one of the items I gave in advance, not a passage it writes now. I later reversed the order. It only judges which kind this thing belongs to, and whether this step should escalate. The sentences go to a chat model, or I write them myself.

I have asked it to think up a name on its own inside an open set. Usable names are many, and only a small slice is written into the question. It can only pick among the items I gave. What is not among the items, it cannot produce. If I cannot name the candidates myself, it is not its turn yet. I narrow with rules first, or I pull a small handful out with retrieval, and then I close the question. Later I changed naming to this: I fix a few candidates myself, then let it pick. A brand-new one outside the list, I leave to the step that can speak.

There is another kind that looks like heavy work which ought to be given to a model. The evidence has to be turned back and forth, and the reasoning has to be kept. Materials push against each other. You want it to pick a side, and you also want it to keep how the match was made. It does not have that back and forth. One request is a fast judgment from what I put in the state. The process does not become text that can be turned through. The back and forth I mean is often a conclusion that depends on the next page, and that page is not in the state yet. I put that dependence in code. Ask one closed judgment first, then I go get the evidence, and when it comes back I change the state and ask again. The sequence of which page was looked at first, and what was ruled out, it cannot write. If a trace has to be kept, I give it to a model that can speak.

What I should ask

What I should ask, I now only recognize a few shapes. Whether it is this kind, I ask directly. The categories have to be the ones I fixed in advance, and it only has to point at one. Severity, I do not let it report a fine number. I draw the levels, and I let it say which level it lands on. Escalation, whether this step should be handed to a person, is also a closed question, and I ask it in the same pass. A shape outside these, I do not stuff into it.

I also keep a coarse ruler. If a person who understands the matter can point, by instinct, while looking at this state, then it looks like its question. If pointing still takes a long deduction, I hand it straight to a model that can speak.

I prepare the state. I close the question. I choose the fields that enter the context, and I do not bring what is irrelevant to this question. The boundaries of options and levels are written in the question, not in a place where I hope it will improvise. It is not responsible for rounding an open question, and it does not turn the next page of material for me. Turn a page, and the state has changed. That is the next request, and I still have to prepare it.

I have stepped on a question that was not closed well. The distribution flattens. Several options crowd together, and none stands a head above. I first took that as it being slow. It was not. I had given a joint the piston of a mouth. Options overlap, or a situation that will actually occur was not written into the list, and the probability can only spread. Then change the question, or hand the step back to a model that can speak. Changing the question means merging the overlapping items, and giving the uncovered case a "none of these". Handing it back means letting the model that can speak say the situation clearly first, and then I decide whether to close it again. Do not keep tuning it. While the question is still open, the distribution is still spread.

Choice and Score

I try not to fill Choice up to 255. The official cap is 255. Beyond that, they themselves have to score first and then choose, and it also gets slower. 255 is not a number I am here to fill. The more edge cases I list as options, the more the question looks like a list that was never closed. If my question naturally has hundreds of exits, I first ask whether rules can cut the candidates, and whether retrieval can cut them. A small handful has no fixed count. I can still tell, at a glance, how these items differ, and only then is the cut far enough. If a glance cannot tell them apart, I am sending the request too early. Cut it to a small handful, and then I take it to choose.

I treat Score as ordered levels, not as a continuous exact decimal. When I write the question, I usually draw the levels from 2 to 10. What I want is a level, not a false precision like 7.63. I write the levels in plain words. Whether this level and the next one would be handled differently, I think through first. If they would, I split them. If both levels end by triggering the same code, I merge them into one level. No branch in the code is waiting for two digits after the decimal. It sometimes lands a number between two levels, and I still read it as a level, with the threshold written in code. If the meaning of a level cannot be written clearly, I change the description of the level. I do not pick at the decimal.

While watching the demos

I watched the official Doom demo and the Wikipedia race, and I did not use them to conclude my own questions. Watching Doom, I am easily carried off by the screen. The screen looks like a game being played. What goes in is structured state, not the picture. On the Doom side the scale is about 10 queries a second, about $7 an hour. That scale only shows that a realtime joint holds, and that a loop can turn at that density. It does not show that it can play the game. The picture was not eaten. The next step is code outside, asking with the state.

The Wikipedia race shows that choosing links at high cardinality is useful. It does not show that it browses the web better than a model that can reason. Under the non-reasoning comparison, it takes fewer steps. That is because the other side did not turn reasoning on, so the demonstration looks better. Turn reasoning on, and the demonstration looks less good. The step count changes, and who is quicker changes. I will not bring "fewer steps" back to my own task and treat it as better at finding a path. Whether the path continues, and where it stops, is still decided by the program outside. I take this as: do not judge everything from a toy. Whether a demonstration is pleasant to watch does not decide, for me, which sentence I should ask next.

When I use it myself, the split is fairly hard. What can be exhausted, and whose conditions are stable, I still write as rules. I do not go ask it. What instinct can answer in one point, that is when I ask it. Before I ask, the question has to be closed, and I have to be able to prepare the state. When it is time to open up and say the content, I give it to a chat model. Ask the wrong thing, and the slowness and the cost are my problem.

Sources

Series