JEV - People step from the middle of the loop to the edge

Catching a failure and looking at every question are separated. High-confidence yes-no and single choice enter a branch directly. A person only handles the slice that bounces back, and confidence has to actually work as a threshold.

Folkbench folkbench.com

Co-author: folkbench.com (Folkbench is a reviewable evaluation and selection platform for AI APIs, model services, and related sites. It draws on published service information, prices, availability, latency, and evidence windows to help users compare and choose a path.)

Ten pieces in this series:

For the past two years, almost every serious AI system I have seen has assumed a person inside the loop, what people call human in the loop. The output cannot be counted on. The system finishes a passage and does not itself know whether the passage can be used directly, so a person has to catch it.

Catching it, kept up, turns into a person looking at every judgment. That is what I ran into. Calling a person only when something has gone wrong is rare. A person sitting in the loop the whole time is common. When there are few questions, that glance still looks like responsibility. When there are many, the person is only passing them through, and after the pass the work goes on anyway. Those two years I took this as standard equipment for a serious system. Output shaped like that, and a person has to sit in the middle. I do not read it as laziness.

Later I separated two things. Catching it is a person standing at the side, waiting for the moment that is unsure. A person in every loop means every question has to pass through their hands. I used to treat these as one thing, and then I could not take the person out of most loops. If I do not even know which question is unsure, I can only look at all of them myself.

What the model handed me was a passage. The passage can be written very completely, and it can look a lot like a conclusion, but it cannot enter control flow. I have to read first, read which way it wants to take the matter, and then set the branch myself. I cannot remove that step. Code wants a value, and the passage is not yet a value. The person stays in the middle of the loop because the branch has to wait until I have finished reading.

What I reviewed was not the occasional exception. Release or escalate, I had to finish a reason first, one piece after another. After enough reading, I treat a complete piece of writing as thinking that is already finished. The person was still sitting in the middle of the loop, and the judgment had followed the passage.

A judgment small enough to enter code

Jev turns that around. The judgment is pressed small enough, and closed enough, to enter the control flow of code directly. The question is written shut before it is asked: yes-no, or single choice, or else a Score. The model only answers the question that has already been closed. It does not tell the whole matter as a passage. The answer enters a branch. A person is no longer in the middle.

The smallness I mean is the question. The business itself has not gotten smaller. The business stays as large as it should be. What gets smaller is the sentence asked of the model. That sentence has to be able to enter a branch in code. What cannot enter a branch is still a note a person has to read, and the person is back in the middle of the loop.

I hold to this when I write the question. If the options can be written, and the boundaries do not fight, the question can be asked. If I only write "use your judgment", that is not a closed question. Either I still judge it myself, or it becomes a longer passage. I keep those. I do not ask the model.

More precisely, what it turns around is not the person. It is "every judgment needs a person to glance at it". Every question carries a confidence. When the confidence can be trusted, the system sends only the least sure slice to a person. The person steps from the middle of the loop to the edge. On the part that is sure, the person no longer appears.

I keep four shallow pictures in mind: whether an alert is closed, whether an invoice is paid, whether the next customer-service sentence escalates, and whether an agent run needs a person to look. They are the same kind of thing. The business is split into closed small questions first, and code walks from the result. The person used to stand in the middle of these loops. After the split, most branches do not have to call a person.

I only handle what bounces back

This is how I use it. High-confidence yes-no and single choice go straight to a branch. I do not open the content. A Score that lands on a middle level, with a flat distribution, is what bounces to me. One that only lands on a middle level, with a distribution that is not flat, I also do not look at. Code continues. There is no third way. I do not let it write a reason first and then decide, from that, whether I look. Done that way, a person still passes every question. What bounces back is only a small slice, and I can take it. When I used to pass every question, I could finish looking and still not say what I had judged.

When it bounces over, look at the distribution, not at a long piece. A long piece kidnaps a person. Once the reason is spread out, the conclusion has usually already been chosen, and the words write downward along that conclusion. I may have another judgment, and I still have to take the passage apart first. I cannot point to the sentence it got wrong. I am only less sure than it is, and I feel awkward pushing back, and by the end of the taking-apart I often nod along. A distribution does not argue with me. How heavy each level is, is laid out. The flat place I can see at a glance. If I change the decision, I change it, without first having to beat a passage written very full.

I am still in this system, and I am absent from most of the loops. The time it bounces back is the time that belongs to me. What I have in hand is a distribution that did not separate. I pick a side myself. Code walks by what I picked. What has already separated finishes inside the code, and it is not my turn to look. I no longer write a passage to negotiate with the model, and I do not wait for it to send me an explanation.

Confidence has to work as a threshold

I dare to use it this way on one condition. Confidence has to actually work as a threshold. If the model can do the questions but cannot say which time it is unsure, a person still cannot leave the middle of the loop. The official account says what they train with RLCD is calibration: higher confidence is more accurate, and similar inputs give similar answers. I trust that direction. The first sentence makes a threshold meaningful. The second sentence I care about more. If similar questions are one way today and another way tomorrow, the threshold I set will not stay fixed. That is why I dare to move the person out of the middle of the loop.

If confidence is decoration, the thing reverses. The person has not stepped to the edge. The person has been replaced by a machine that is better at performing certainty. It gives every question a very full number, and the number has nothing to do with whether it is accurate. If I use it as the condition for letting something through, I let go when I should not. In the way I use it now, it does not do that. High confidence, I give to code. What bounces back, the distribution is flat, with no polite sentence of uncertainty attached.

There are also times a person has to step back in. What has not been closed well should not be automated. An option is missing a piece, or two boundaries are tangled, and confidence only fills a number in on a bad question. I go back and change the question first. I do not prop the gap up with the model. Consequences too large to buy with a probability threshold, I leave inside the loop. All four pictures have a loop of that kind, and however high the confidence, I do not give it to a threshold. That is not the model failing. That loop should not be automated.

After the person steps to the edge, the questions I look at are far fewer. The ones I am sure of, I no longer look at. The least sure slice is still in my hands. Loops that should not be given to a threshold, a person stays inside.

Sources

Series