How we fix an AI agent's mistakes: data first, prompt last

For years, one way to teach a model the cases of a product was fine-tuning: training it on your own examples. That road is closing. According to its deprecations page, since May 2026 OpenAI no longer lets organizations that have never fine-tuned create new training jobs, and from January 2027 its existing customers won't be able to either.

If the model is rented, as we argued in the first post, then what has to learn from mistakes is the harness. Here is how we do it: every reply can be reported; a mistake is fixed in the data first, then in the missing fact, and only then in the prompt; and each fix is proven by a family of test cases that run more than once. This is the last post in the series.

Make every mistake reportable on the spot

Back to the note from the second post. The project has two open Novak jobs, the panel installation and a battery quote, and suppose this time Werk gets it wrong: it marks the quote as done. The person replies "it was the installation," and Werk undoes the change and makes it on the right card. For them, the problem ends there.

For the harness, that's where the work starts. A mistake that only gets fixed in the chat will happen again with the next client who has two open jobs. That's why every reply from Werk can be reported: pick a reason and, if you like, add a voice note or a few words explaining it.

It pays to have this from day one, when it still seems early, and to decide who reads the reports. What someone writes in a report is part of their team's work. At Werk, an agent reads those reports, not a person.

Data first, instructions last

Faced with a model's mistake, the temptation is to go straight to the prompt and add a line. First, read everything the model received on that turn, in full: not what the code was supposed to send, but the exact request that went out to the provider, with the instructions, the conversation, the cards and the message. Then the questions come in order.

The first is whether the data was right. If the conversation arrived incomplete, the right card wasn't among the candidates, or two parts of the context contradicted each other, the model had no way to get it right, and the fix belongs in the harness. In the Novak case, the data was right: the model saw both cards and where each one matched.

The second is whether a fact was missing. The model lacked what the person knew by heart: the installation was theirs and they had started it that morning, while the quote belonged to someone else. No instruction can replace a fact the model can't see. The fix is to show it who holds each card and what state it's in.

Only after those two questions do you look at the instructions, when the model has correct, complete data and still gets it wrong. Even then, what you write is a principle, never a rule for the case. A line like "if they say Novak, it's the installation" fixes this note and breaks next week's, when the quote is the one that's done.

From one mistake to a family of cases

A fix has to be proven, and one case won't prove it. The Novak report gives you a class of mistake: two open jobs for the same client, and a message that doesn't say which.

From that class you write a family of cases: the same situation in other words and other languages, as a voice note, with two different clients who share a name. And the opposite case too, with a single open job, where there's nothing to be unsure about. If the fix only works for "done with the Novak job," the mistake is still there.

These cases are best written by someone other than whoever makes the fix. Whoever fixes something tends to test exactly what they fixed; whoever starts from the class of mistake tests what the person needs. That way special cases live in the test battery, not in the prompt: the instructions state principles, and the cases check that those principles hold.

Measure more than once

A model doesn't always answer the same message the same way. τ-bench (2024), a benchmark of agents that serve customers with tools, measured it: even the best agents of the time solved fewer than half of the tasks. And in its retail setting, the chance of solving the same task correctly on all eight tries fell below 25%.

That's why every case runs more than once. A fix that passes once and fails on the next run isn't done.

A case that fails one time in ten is a mistake someone will run into on a roof, sooner or later.

And when a fix goes in, run the whole battery, not just the new cases. That's how you know a mistake was fixed without breaking something else.

The harness learns with use

Models will keep getting things wrong now and then, like anyone working in a hurry. What you can build is what happens next: every reported mistake becomes a family of cases, and those cases get checked every time something changes.

Developers and office workers already have a harness built for their work. The person installing solar panels on a roof now has one too, and every mistake someone reports makes it better for everyone.