At some point late on a Sunday night, after a whole weekend of grinding, my scoreboard finally showed what I’d been chasing: 100%. Two rounds in a row. Every question answered correctly, twice, with receipts.

That was the moment I could walk into a client’s office and say: this AI assistant tells the truth. Not “it seems pretty good.” Not “it usually gets it right.” I can prove it, on demand, with a script.

Let me back up and tell you how I got there, because this was one of the most instructive weekends I’ve had with AI. It’s also where I almost gave up on local models entirely… for exactly the wrong reason.

The setup

I built an AI assistant for a payments company here in the US. Leadership asks it questions in plain English: how many deals entered underwriting this week, top merchants by volume this month, what does our SOP say about a chargeback dispute. The assistant answers from their live systems: the CRM, the accounting platform, the data warehouse, the SOP library. All of it runs on a local model on hardware they control, because their data (merchant financials, banking details) is exactly the kind of thing that should never leave the building.

Here’s the problem nobody warns you about. The demo works on day 1. Getting a business to trust it takes much longer. I’ve seen this pattern with every finance-adjacent automation I’ve built: it takes a team a month or 2 of checking the system’s work before they stop double-checking it. And they’re right to. In finance, a confidently wrong answer is worse than no answer.

My favorite framing of this comes from Robert Smith at Vista Equity Partners. Talking about AI in the enterprise, he said you cannot have a wire randomly not go out. That line lives in my head. Generative AI is inherently non-deterministic. The same question can produce a different answer tomorrow. Businesses run on the opposite: deterministic systems and processes, where the wire goes out every single time. So the job is to wrap non-deterministic AI in deterministic code until the outcomes are deterministic. That wrapper is where the power is.

What an eval actually is

The wrapper has a name: evals. An eval is an exam for your AI. You write questions where you know the correct answer, you make the AI take the exam over and over. You grade it with code, never with vibes.

You hear about evals constantly in AI circles. It’s not until you build one for a real business that you realize how hard they are. Here’s what mine ended up looking like, because the details are the whole game:

The questions. A core set of real leadership questions, plus an adversarial set designed to trick the model: questions with no answer in the data, questions that tempt it to guess, ambiguous date ranges. If your exam only has softballs, you’ve built a participation trophy.

The oracles. For every question, a small piece of code computes the correct answer directly from the database. No AI involved. That’s the answer key. The word “oracle” sounds fancy but it just means “the truth, computed the boring way.”

The grader. Another piece of deterministic code compares the AI’s answer to the oracle. Exact-match where possible, structured comparison where not. The grader has no opinions. It cannot be charmed by a well-written wrong answer, which is the failure mode humans fall for every time.

The receipts. This is my favorite part. Every eval run cross-checks the gateway’s audit log to confirm the model actually called the data tools. Why? Because early on I caught the model “passing” a question by reciting a plausible answer from its instructions without ever touching the database. The reply text looked right. The audit log showed zero tool calls. It was, in the most literal sense, making it up. Now any answer without a fresh tool call in the audit log is an automatic fail, no matter how correct it looks. That single rule is the difference between “sounds right” and “is right.”

The manifest. Every run gets a signed manifest with a fixed model, temperature, seed, and question set, so 2 runs are actually comparable. When I later swapped models to compare (a Qwen baseline against a Gemma challenger), the harness pinned everything else so the comparison meant something.

I almost quit because I thought the model was stupid

Now the confession. For a stretch of this project I was convinced the local model just wasn’t smart enough. It flailed. It picked wrong tools, mangled date windows, gave answers that were close but wrong. I remember thinking: this is why everyone stays on the frontier cloud models. The little model can’t hack it.

I was wrong. The way I found out changed how I think about local AI. The model wasn’t stupid. The tooling was. The missing key was the MCP server: the layer that defines what tools the model can call and how they’re described. Once I ground the tool semantics properly (what “approved” actually means in this business, which data source is canonical for volume numbers, how date windows resolve against the company’s timezone), the same “stupid” model started acing questions it had been fumbling for days. The intelligence was there the whole time. It was starving for well-defined tools.

If you take one technical lesson from this piece, take that one. When a local model fails, interrogate your tool layer before you blame the model.

The last 10% took a machine 12 hours

Working in Claude Code, I got the eval scores to maybe 80-90% of where they needed to be over that weekend. It felt like each remaining point cost more than the previous 10 combined. Classic last mile.

So I did something I do more and more now: I handed the last stretch to a different AI. I gave Codex a /goal (a mode where you describe the finish line and the proof, then walk away) and let it grind. It worked the problem for almost 12 hours. Tightening tool descriptions, fixing edge cases the grader kept catching, rerunning the battery after every change. When it was done, the evals hit 100%… and then hit 100% again on a second consecutive round, which was my bar for “done.” One perfect round can be luck. Two in a row is a system.

Partially this weekend was perseverance. Partially it was just playing around, vibing with the models until I understood how they fail. Both mattered. I learned more about AI in those 2 days than in some entire months.

The lesson that bit me: live data moves

One morning mid-project, the evals broke. Questions that passed the night before started failing. I hadn’t changed anything. What happened?

The data changed overnight. I was running evals against the live dataset. The business kept doing business overnight: new deals, updated records. My “correct answers” had drifted out from under my exam. To the AI’s credit, it actually flagged the discrepancy rather than papering over it, which is the one time I was glad to be surprised. The lesson stands. I’d do it differently today: evals belong on a frozen snapshot or a historical window that can never change. Your answer key must be carved in stone, or you’re grading against sand.

Where this leaves you

If you have an AI assistant anywhere in your business (or you’re about to), do this before you trust it with anything that matters: write down 10 questions you already know the answers to, including 2 or 3 trick questions with no answer in your data. Run all 10 through your assistant every week and score it with a spreadsheet. That’s a manual eval. It gets you 80% of the value of everything I described above.

The wire has to go out every time. Until you’re measuring, you’re just hoping.