I can’t write a line of Python. I couldn’t tell you what a rw-semaphore is if you paid me, and until last week I’d never heard the phrase “streaming tool-call parser” in my life.

And yet three days ago, I sat down with Claude and built a system that takes a local AI model, hands it a job, checks its work against a written standard, has a second AI critique it for taste, runs two versions of the same job side by side and picks the better one blind, and sends failed work back for a fix with specific notes, automatically, until it passes.

None of that required me to know how to code. It required me to understand two things: what a model is, and what a harness is. Once you get that distinction, you can direct serious engineering work without touching a single line of it yourself. That’s the real story here, more than the local AI part.

The engine and the car

Here’s the distinction that unlocked all of this for me. A model is the raw intelligence: the thing that reads a prompt and generates a response. When people talk about “which AI is smartest,” they’re talking about the model. GPT, Claude, Gemini, the open-source ones you can run on your own computer. That’s the engine.

A harness is everything you build around the engine so it actually gets you somewhere. The fuel line. The transmission. The dashboard that tells you when something’s wrong. The seatbelt that stops a small mistake from becoming a disaster. A brilliant engine bolted to nothing is a very expensive paperweight.

Most people, myself included until recently, spend all their attention on the engine. Which model is best this month. Which one just got upgraded. Which one benchmarks highest. Almost nobody talks about the harness. But it’s the harness that decides whether any of that horsepower actually turns into a car that drives somewhere useful.

Why this mattered to me. I run AI models on my own hardware at home, a Mac Studio and a machine called a GX10, instead of only renting time from OpenAI or Anthropic’s cloud. It’s nearly free once you own the hardware. You just pay for electricity. That’s a real advantage for privacy and cost. The catch: those local models are noticeably less polished than the frontier cloud models. They need a much better harness to behave. This project was about building that harness, and it turned into a lesson that applies to any AI system, local or not.

Aren’t you just going back and forth with it?

Fair question. I’d rather answer it here than have you wonder further down.

There’s a mode most modern coding AIs have now, usually a command called something like /goal. It’s a completely different thing from a normal chat. You don’t type something, get a reply, correct it, type again. You write one paragraph describing what you want built and how to prove it’s actually done, hit enter, and walk away. The AI reads its own output, decides its own next step, checks its own work, and only comes back to you when it’s finished or genuinely stuck. No back and forth at all, at least not from a human. That’s the actual feature I was chasing this whole time. Everything else in this article exists in service of making that one feature work reliably.

Here’s the discovery that kicked off this entire project, months before the three days I described above. That autonomous mode already just works on the big frontier cloud models. Point Opus or GPT at a /goal task and it handles it, start to finish, almost every time. Point the exact same feature at a local model running on my own hardware and it goes off the rails. Not slightly worse. Off the rails. The transcript below, the one with zero files produced after six minutes, that’s what /goal looks like on an unmodified local model. Nobody was correcting it turn by turn during those six minutes. It was completely alone. Its own decisions were bad ones. It kept making the same bad decision every few seconds until I finally killed it.

That’s worth sitting with for a second. What looks like arguing with the AI in that transcript is actually the model, entirely unattended, stuck in a hole it dug for itself, with nobody typing anything into it at all. A bad back-and-forth conversation you can just walk away from. A system that’s supposed to be working autonomously while quietly failing for six minutes, or six hours, is a completely different kind of problem. That’s the one this whole project set out to fix.

The tool I actually use for this is called pi, a terminal coding agent. It’s intentionally bare bones. It doesn’t ship with a hundred built-in features the way some assistants do. If you want a capability it doesn’t have out of the box, like /goal, you build it yourself as an add-on.

I have zero coding background. That should have ended the story right there. Instead, it’s exactly what forced me into the kind of work I describe later in this piece. I found an existing open-source add-on someone else had built for /goal-style execution, had Claude fork it as a starting point, and pointed it at the local model I was testing at the time, an earlier version of Qwen, before I ever landed on the 27B and 35B versions this article is mostly about.

It didn’t work. The existing add-on assumed a level of self-correction that only frontier models actually have. Local models needed something built around how they actually fail, not how a much smarter model fails. So I built my own version from scratch. That add-on, more than anything else I’ve described so far, is the real harness this whole article is named after.

I was genuinely frustrated for a while. It felt like the one feature I wanted most, autonomous execution I could point at a goal and walk away from, was locked out for anyone not running a frontier model with a monthly bill attached. But that frustration is exactly what kept me going. If I could get a model running on hardware I already own, free after the electricity bill, to reliably execute a /goal-style task the way a much more expensive model does, that unlocks a lot. Cheap and private, running on hardware I’d already paid for. That was worth the fight.

What it looked like before

I asked one of my local models to build a simple browser game and just walk away. Should be easy for something that can write code. Instead, it opened the file, wrote nothing but the filename into it, got an error, and then started repeating the exact same broken action over and over. Not for a minute. For the entire session, once every couple of seconds, forever, until I killed it.

That’s not a smart machine having a bad day. That’s a machine stuck in a groove it can’t see itself in, like a scratched record.

	Before	After
Result	0 for 3 on real tasks	6 for 6, unattended
What happened	The model got stuck in the same broken loop, every single time I asked it to build something. Three attempts, three failures.	Every task, every model, every mode I tested came back clean. One run even survived the computer running it crashing mid-task and picked back up on its own.
Verdict	Not reliable enough to trust with real work.	Total number of times I had to step in and help: one.

The obvious assumption is that the model itself was the problem. Get a smarter model, this goes away. That assumption turned out to be almost entirely wrong. Finding out why is the whole point of this article.

It wasn’t the model. It was the plumbing.

I went digging. What I found had nothing to do with the model being dumb. Picture a waiter taking an order back to the kitchen. The waiter mishears “table four wants the burger with no onions,” the ticket that actually prints in the kitchen just says “table four,” and the cook sends out an empty plate. The diner complains. The waiter, not knowing what actually went wrong, reprints the exact same broken ticket for the next table too, because that’s the only ticket format he’s ever seen.

That’s almost exactly what was happening, translated into software terms. A piece of plumbing between the model and the outside world was corrupting big pieces of the model’s work in transit, silently. The model would try to write a whole file’s worth of content. By the time it reached the other side, only the filename survived. The system then filed that broken attempt away as “this is just what a normal turn looks like.” The model, which pays close attention to its own recent behavior, started copying the broken pattern because that’s what it had just seen itself do.

I wasn’t looking at a stupid model. I was looking at a good model being fed corrupted instructions and then blamed for the empty plate.

The defining mistake of the whole project was assuming a machine that did something inexplicably dumb, repeatedly and identically, must be dumb. When a system behaves that strangely and that consistently, look at the plumbing before you blame the intelligence.

Once I fixed the plumbing, one small thing that used to break the entire game just… started working. The model hadn’t gotten smarter overnight. Somebody had finally handed it an unbroken order ticket.

Then a totally different question showed up

Fixing the plumbing solved reliability. The model would now finish the job every time. But that raised a second, sneakier question I hadn’t thought to ask yet.

Finishing isn’t the same as finishing well.

A model can complete every step you asked for and still hand you something mediocre. It can even convince itself the job is done right when it isn’t, the AI equivalent of a contractor telling you the wiring’s fine without ever flipping the switch. So the next three days weren’t about reliability anymore. They were about quality, and about how you teach a system what “good” actually means before you ever ask it to produce anything.

The restaurant that taught itself to cook better

The clearest way I can explain what I built is to picture a restaurant kitchen, because the whole system maps onto one almost perfectly.

Imagine you hire a promising but unproven line cook. You could just tell them “make something good” and hope for the best. Or you could set up an actual system: a written recipe standard so “good” is defined before anyone starts cooking, a kitchen inspector who checks the dish against that standard with real evidence (not the cook’s word for it), a food critic who tastes it and judges something the recipe card can’t capture, and a head chef who sends a dish back with specific notes instead of a shrug when it’s not right.

That’s the four pieces I built, one at a time, run automatically, no human in the loop:

1. The recipe card. Ten checkable rules for what “good” means, written down before the cook starts. Never handed to the cook directly, just used to write better instructions.
2. The inspector. Checks the finished dish against the recipe, line by line. If it can’t find proof something was actually done, that counts as a fail, even if it looks fine.
3. The food critic. A separate judgment: not “did you follow the recipe” but “would I actually want to eat this.” Catches things the checklist can’t.
4. Sent back to the kitchen. If it fails, the cook doesn’t get a lecture. They get a short, specific punch list and one more shot. Then it’s re-inspected from scratch.

The recipe card: writing down what “good” means

I took a public checklist, built by researchers studying what separates genuinely excellent work from work that merely looks fine, and boiled it down into short lists for the kinds of work my models do. Ten lines for code. Ten for research reports. Things like “every error must show the user something, never fail silently” or “every number in a report needs a source.” Nothing fancy. Just specific and checkable.

The important part is where that checklist lives. I never hand it to the model and say “follow this.” Hand a promising-but-inexperienced cook a fifty-page culinary textbook mid-shift and watch the dish get worse, not better. Weak systems get overwhelmed by too much guidance in the moment. Instead, the checklist shapes the instructions I write before the work starts. It becomes the standard the inspector checks against afterward.

The inspector: proof, not vibes

After a model finishes a task, I bring in a second AI, one that had no hand in building the thing, to grade it against the checklist. Not a vibe check. Every single line needs a citation: point to the actual file, the actual test result, the actual proof. If it can’t find proof something was tested, that’s an automatic fail, even if the output looks perfectly fine on the surface. “I’m pretty sure it works” doesn’t clear inspection. You have to have flipped the switch and watched the light turn on.

This alone caught something I found genuinely surprising. I had a model build a small game, it looked completely finished, played fine when I clicked around in it. The inspector failed it anyway, because nowhere in the model’s own testing had it ever actually driven the game into its “you lose” state to confirm that screen worked. It just assumed that part was fine. Turned out it wasn’t. That’s a real defect a human reviewer, moving at normal speed, would very plausibly have missed too.

The food critic: a second opinion that asks a different question

The inspector answers “did you follow the recipe.” That’s necessary, but it’s not the same question as “is this actually good.” So I added a second, separate judgment: a food critic who never saw the recipe-following inspection, who only asks would I ship this, would I actually want this.

This is where the project surprised me the most. I had my model build the same small game two different ways, one instructed to prioritize doing everything correctly, one instructed to prioritize looking and feeling great. The rule-follower scored higher on the checklist. Case closed, right?

The food critic, judging both blind, without knowing which one was which, picked the other one instead. And it was right. Buried underneath the polished version’s good looks was a real bug: a weapon in the game that quietly dealt damage twice per hit instead of once. Every checklist line had passed. The rulebook-checker had no idea anything was wrong. It took a different kind of scrutiny, one that actually plays the game instead of just checking boxes, to catch it.

	Version A: followed every rule	Version B: built for polish
Checklist score	5 out of 8	7 out of 8
What it looked like	Plain but every rule-following line checked out clean.	Scored better on paper. Hid a real double-damage bug underneath the shine.
Food critic’s pick	No	No, once it actually played it

Neither judge alone would have caught the full picture. That’s the actual lesson: checking the rules and judging the taste are two different skills. You need both, because they catch different kinds of failure.

Two cooks, one dish, judged blind

Once you’ve got an inspector and a critic, a new option opens up almost for free: why send just one cook into the kitchen? I started building the same thing twice at once, on two different machines, each pushed toward a slightly different style, then let the blind food critic pick the winner without ever being told which machine cooked which dish.

The whole thing, building two versions and picking a winner, took eight minutes. And the blind judging disqualified one version for a bug that would have shipped completely unnoticed if I’d only tried once. Two cheap attempts, judged honestly, beat one careful attempt almost every time, as long as the judging stays blind. The moment the critic knows which cook is the “good” one, you’re not testing the food anymore, you’re testing your own bias.

Sending a dish back to the kitchen, correctly

The last piece: what happens when something fails inspection? The lazy version is to hand the cook the entire inspection report and say “fix everything.” That overwhelms an inexperienced cook and usually makes things worse. The version that actually works is a short, specific punch list, the two or three most related problems, nothing more, with instructions on exactly how to prove each one is actually fixed this time. Then a full re-inspection, from scratch, no shortcuts for what passed last time.

I capped it at three attempts. Either it passes clean, or it stops and shows you honestly what’s still broken, instead of looping forever pretending progress is happening.

The lesson that ties it all together

A few years ago I had a quick 24-hour speaking trip to North Carolina. I travel constantly, so I figured I didn’t need to think too hard about packing. Showed up with a dying phone and no charger. Realized at the venue I’d left my clicker at home. Then my slides wouldn’t load. One disaster after another. Every single one was avoidable. The lesson stuck with me hard: no matter how experienced you think you are, if you don’t have a checklist, you will forget something that matters.

The AI version of that same lesson showed up in the middle of this project. It was almost funny how directly it mapped. I built a game with a “you win” screen that only appeared after clearing dozens of levels. Totally reasonable requirement. Except there was no fast way for the inspector to actually reach that screen and confirm it worked. The model, trying to prove something impossible to prove quickly, got stuck trying to invent a workaround for forty straight minutes and never finished.

The fix wasn’t a smarter model. It was one sentence in the instructions: build in a secret side door, a special web address that starts the game one level from winning, so the win screen can be checked in five seconds instead of five minutes of actual play. The moment that existed, the very same model built the entire thing correctly on its first try, no help needed, no do-overs.

The rule that generalizes past AI entirely. If you can’t cheaply prove something works, you can’t reliably know that it does, no matter how capable the person or system doing the work is. Design the checkpoint before you design the work. That’s true for a video game’s win screen. It’s true for your own business processes too. Anything you can’t easily check, you’re just hoping about.

How three days actually went

Day 1: Pick the model, learn its quirks. Chose between two local models by running the same real task on both and comparing. Also learned the model rewrites its entire memory from scratch on every single turn, so the size of what you feed it matters enormously. That single fact shaped almost every decision after it.

Day 2: One job type goes reliable. The other stays broken. Research-style tasks started succeeding after I gave the model a smarter set of tools instead of smarter instructions. A coding task kept failing the same broken way, over and over. At the end of the day I still didn’t know why.

Day 3: Found the real bug, fixed the plumbing. Recorded the raw data flowing between the model and the outside world and caught the corruption in the act. Fixed it in four separate layers so no single point of failure could break it again. The task that failed three times in a row finished clean in one long, unattended run.

Day 4: Every remaining test, on both models, came back clean. Ran the full test matrix across both AI models and both job types. Six for six. One run even survived the physical machine running it crashing halfway through and picked back up on its own once it came back online.

Day 5: The quality system: recipe cards, inspector, critic, retries. With reliability solved, I spent the day building and proving out the whole restaurant system above. It closed with the cleanest possible result: a fresh task, built with a secret side door for testing baked in, passed every single check on its very first attempt.

6/6	1	8 min	1st try
tasks passed	human step-in, total	to build 2, pick 1	clean pass, final test

Why I could direct this without knowing how to build it

Here’s the part I actually want people to take from this, more than any of the technical detail above. I’ve talked before about my rule of thumb for working with AI: ten percent of the work is me setting the vision and the spec, eighty percent is the AI actually executing it, and ten percent is me doing final quality review. This project is the cleanest example of that rule I’ve ever run.

I didn’t touch a line of code. What I actually did, across three days, was a short list of real decisions: approve the plan before big changes got made. Decide that a small speed cost was worth it for a big reliability gain. Say yes to running an experiment that would take an hour to find out if a whole approach was worth pursuing. Notice when a report felt incomplete and ask “have I actually proven this helps, or does it just sound like it should?” That last one, by the way, led directly to discovering the taste-critic catching the hidden bug. If I’d just accepted the first “yes it works better” answer, I’d have shipped the worse version and never known.

None of those decisions required knowing what a rw-semaphore is. They required judgment, the willingness to ask “prove it” instead of accepting a confident-sounding answer, and knowing enough about models and harnesses to ask the right question at the right layer. That’s a learnable skill. It’s not a coding skill.

The enterprise version of this lesson

This is the part I think about constantly when I’m advising a company instead of tinkering on my own hardware. It’s the opposite instinct from what most people assume.

As a hobbyist, I can chase the newest model the moment it drops. If it breaks something, I lose an evening. No real cost. A company can’t operate that way. Not because they’re slow or bureaucratic. It’s because upgrading the model is never just upgrading the model. Every time the underlying engine changes even slightly, everything you built around it, every rule, every check, every recipe card, has to be re-verified against the new version. In this project, upgrading my model server by two versions took an entire day of careful re-testing just to confirm nothing had quietly broken. For a side project, that’s a fun Saturday. For a business running real operations on top of it, that’s a full QA cycle, with real risk if you skip a step, every single time a new model ships.

If you’re chasing the newest, shiniest model every few weeks, you’re signing up for that entire QA cycle every few weeks, forever. Most companies can’t afford that rhythm. Honestly, most don’t need to.

I think about this the same way I think about buying a genuinely great pair of headphones instead of replacing cheap ones every year. Spend once, properly, on the thing you’ll use constantly. You stop paying the replacement tax over and over. For a business, the equivalent move is picking one solid model and investing seriously in the harness around it: the checklists, the inspection, the retry logic, the specific rules for your specific use case. That harness is where the real, compounding value sits. The model underneath it becomes almost a commodity you can swap out later, carefully, on your own timeline, once there’s an actual proven reason to.

The strategic takeaway. Don’t ask “which model should we use.” Ask “what does good output look like for this specific job, and how do we prove we’re getting it.” Answer that once, build the harness around it. The model becomes the easiest part to replace later. Most companies have that order backwards.

One thing to try this week

You don’t need local AI models or three free days to use any of this. Pick one thing you already ask an AI to do regularly. Write down, in five bullet points, exactly what “good” looks like for that specific task, the way I wrote my recipe card. Then, next time you get an output, actually check it against those five points instead of skimming it and moving on.

That’s the entire system, shrunk down to something you can do in fifteen minutes. Everything else I built this week, the inspector, the critic, the retries, is just that same idea, made automatic and running at scale. Start with the five bullet points. See what it catches.

Field notes from a 72-hour local AI build, July 2026. Written by hand, no ghostwriter, with an AI system doing the eighty percent I described above and me doing the ten and ten around it.