Last Saturday I typed the same prompt into 2 different AI agents and let them race.

The job: audit a production codebase I built for a VC firm. It’s a custom CRM that manages their deals. It has grown to about 650,000 lines of application code (over 1.4 million lines if you count everything in the repo). The ask was simple to state and brutal to execute: find every place where a user can create something but can’t edit or delete it. Every missing lifecycle action, across every entity, in the whole system.

One agent ran GPT-5.6, a frontier model in the cloud. The other ran DeepSeek V4 Flash 0731, the July 31 release that I consider a monumental moment for local AI, on 2 DGX Spark machines sitting in my office, connected with a single QSFP cable, serving the model with tensor parallelism across both boxes. I wrote about that setup when I built it in my first 24 hours with DeepSeek. Same prompt, byte for byte. Same codebase. Same tools available.

Then I had Claude verify both reports against the actual source code, line by line, without reading either report first.

Round 1 was a bloodbath

GPT-5.6 came back in 24 minutes with 32 findings. It had spawned 8 subagents on its own: readers to digest the big files, reviewers to check its own work, a verifier to run the project’s test suites before it signed off. Every finding carried a file and line number. When Claude spot-checked 7 of the 32 against the source, all 7 held up. It even caught a genuine correctness bug that had nothing to do with the original question, buried in a code path nobody had looked at in months. Cost: about $4.

DeepSeek came back with 8 findings. 5 of them were flat-out wrong.

And they were wrong in the worst possible way. It claimed a delete function didn’t exist when it shipped months ago. It claimed there was no way to deactivate a user when the button renders on every row. It claimed an entire management UI was missing when 2 of its features were live in production. If I had handed that report to a developer, they would have spent days building things that already exist.

Here’s what stung: the model never checked. Not one of those claims came with a citation. It looked at part of the code, formed an impression, and stated the impression as fact.

The part where I almost drew the wrong conclusion

My first instinct was the obvious one. Frontier model wins, local model loses, story over. I’ve been running local models long enough to know that instinct is usually lazy, so I dug into the session transcripts instead.

Do you know what I found? DeepSeek had the exact same subagent tools available that GPT used. It called the subagent tool exactly once, with {"action": "list"}. It looked at the menu of helpers it could have delegated to. Then it never used any of them. It did the whole 650,000-line audit alone, in a single pass, with 75 shell commands, and wrote up its impressions.

Why would it do that? Because my harness told it to. Sort of.

I’ve written before that the model matters less than the harness, the scaffolding you build around a model that turns raw intelligence into reliable work. My harness for local models is a custom extension for Pi, my coding agent. It has guidance baked in that says: before you finish non-trivial work, spawn a fresh verifier subagent and have it check your work.

That guidance never reached the model. The harness checked whether a tool named Agent was available before injecting it. The subagent extension had renamed its tool to subagent months earlier. The check silently failed, every session, for weeks. The one instruction designed to produce exactly the behavior GPT showed spontaneously was disconnected the entire time.

It got worse the deeper I looked. 3 separate problems, all mine:

1. The dead delegation guidance. The stale tool name above. Two-line fix once found.

2. No evidence contract. Nothing in my harness required the model to prove a claim before making it. All 5 of DeepSeek’s errors were the same failure: asserting something doesn’t exist without searching for it. So I added rules to the guidance: every finding must cite file and line. Before you claim a feature is absent, run the searches that would find it, and only claim absence when they come back empty. Before you finish, re-open every cited location once and drop anything that doesn’t hold.

3. Turn budgets tuned for a weaker model. My subagent helpers had an 8-turn budget, a leash I added months ago when a smaller model would loop forever if you let it. DeepSeek doesn’t loop. But point an 8-turn reader at an 866KB JavaScript bundle and it gets cut off mid-analysis every time. I raised the budgets to 20.

Run it again

After shipping those fixes (each one gated through my test suite, because harness changes you don’t test are just new bugs), I quarantined every prior report so the model couldn’t cheat by reading its own answer key. Then I ran the identical prompt again.

The difference was honestly a little eerie.

Fixed DeepSeek fanned out 4 parallel reader subagents on its first dispatch. It ran verification greps before allowing its own claims. It built the report incrementally instead of trying one giant write. It ran 31 minutes and 42 seconds start to finish with zero human intervention. It produced a 24-entity matrix with file and line citations on every single cell.

All 5 of the original false claims were gone. Not corrected. Gone, replaced by accurate statements of what actually exists. It found 2 real issues that even GPT had missed, including one that mapped directly onto a feature request the client had literally asked me for that week. Total errors: 2, both overstatements rather than fabrications. Coverage: roughly two-thirds of what GPT found.

Same model weights as the disaster run. Every single gain came from the harness.

Here’s the whole experiment in one table:

	GPT-5.6 (cloud)	DeepSeek, day 1	DeepSeek, fixed harness
Time to finish	24 min	~15 min	31 min 42 sec
Findings	32	8	24 entities, 10 ranked gaps
Wrong claims	0 found	5	2
File + line citations	Every finding	None	Every finding
Helpers spawned	8	0	7, in parallel
Babysitting needed	None	None	None
Cost	~$4	$0	$0
Where the code went	Their servers	Never left my office	Never left my office

That last row is the one I’d circle if you run a business with code or data you’d rather not upload to someone else’s cloud.

There was one more model quirk worth naming, because it took a fourth fix. DeepSeek has a habit I started calling say-then-stop: it ends a turn by announcing its next step (“Let me add the Funds section”) and then just… stops. Doesn’t execute the step. Doesn’t fail. Stops. A bare “continue” always got it moving again, which meant I was babysitting a supposedly autonomous run. So I taught the harness to detect that exact signature (turn ends, no tool call, last sentence is a narrated intention that isn’t a question) and send the “continue” itself, capped so a genuinely stuck run still reaches me. That’s what made the 31-minute unattended run possible.

What the hardware was doing the whole time

I checked on the 2 DGX Sparks mid-run, while the parent agent and 4 subagents were all hammering the cluster at once. Both GPUs sat at 95% utilization. Temperatures were 65°C and 71°C, nowhere near throttling. The KV cache, the memory pool that holds every active conversation, was at 5.3% of its 1.1 million token capacity. The prefix cache, which lets concurrent agents share the parts of their context that overlap, was hitting 94.5%.

Translation: 5 agents auditing a 650,000-line codebase simultaneously barely woke the thing up. The request queue never had a single job waiting. I had been wondering whether I’d need a third machine for more concurrency. The measurements say no. The scaffolding around the model was the bottleneck all along, which is becoming the theme of this entire year.

So which model should you use?

For analyzing a massive codebase: the frontier model. It isn’t close. GPT-5.6 did something DeepSeek never managed even after all my fixes. It held the frontend and the backend in its head at the same time and asked whether they agree. Its best findings were cross-layer: a UI that checks one permission column while the API checks a different one, a shortcut code path that silently skips the validation the main path enforces. That kind of synthesis across distant parts of a system is still what you’re paying frontier prices for. DeepSeek, even fixed, found what was visible surface by surface. It went from confidently wrong to honestly incomplete. That’s a massive upgrade, but incomplete is still incomplete when the codebase is this big.

On a smaller codebase? I genuinely think fixed DeepSeek would hold its own. Two-thirds coverage with citations, at zero marginal cost, on hardware I own, with client code that never leaves my office. There’s a real class of work where that trade wins.

If you’re a business owner reading this, that’s the practical takeaway. The question was never “is local AI as smart as the frontier models?” It usually isn’t. The question is which jobs in your business need frontier judgment and which jobs need a tireless worker that costs nothing per run and keeps your data in the building. Client code audits, document processing, internal research, first drafts: once the scaffolding is right, a local model handles an enormous amount of that. I wrote about when local is good enough if you want the fuller argument.

My favorite pattern hasn’t changed, it’s only gotten stronger: use the frontier model to plan and architect, hand the bounded implementation work to DeepSeek, then bring the frontier model back to review the result. You get frontier judgment at the 2 points where judgment matters and nearly free execution in the middle. After this weekend I trust that split more than ever, because now I know the middle leg’s failures were mostly my own scaffolding.

I keep seeing people on X dismiss local models after one bad session. I did the same thing before an overnight run changed my mind. Here’s what I’d ask them now: was the model wrong, or did your harness never give it a chance? Nobody has built a great harness for DeepSeek yet. I use Pi with custom extensions and I still found 5 separate ways my own setup was strangling the model, in a single day of looking.

My harness is public, by the way. The extension with all of this in it, the evidence contract, the capability-aware delegation guidance, the auto-continue logic, lives at github.com/humanrouter/pi-local-harness. Take whatever is useful.

If you run a local model, here’s the one thing to do this week: pull up your last disappointing session and read the transcript, not the output. Look at what tools the model had, what your scaffolding told it, and what it was never told. I thought I was benchmarking a model. I was actually benchmarking my harness, and the harness lost.