For months, Qwen3.6 27B was my gold standard for local AI.

I run it on an M3 Ultra Mac Studio with 256GB of unified memory, a $7,500 machine I bought specifically so I could stop worrying about token limits and monthly caps. I know exactly what Qwen is good at. I know where it gets stuck. I’ve built custom Pi extensions around its quirks. That’s months of tuning I wasn’t excited about throwing away.

Then I connected 2 ASUS GX10s and installed DeepSeek V4 Flash. I wrote up that whole setup and my first 24 hours with it if you want the hardware side of the story.

It felt better within about 20 minutes. Fast responses, clean tool calls, long conversations that didn’t fall apart halfway through. A day later I moved PiNoy, one of my local Hermes agents, off Qwen and onto DeepSeek.

Then I caught myself. I’d just made a real infrastructure decision based on a vibe.

I’ve been burned by that before. Years ago I built an entire sleep course off Reddit research because it felt like what people wanted. It flopped. My audience wanted OmniFocus training. I’d mistaken “what’s loudest in my head” for “what’s actually true,” which is the same trap I was walking into here.

So I ran both models through 51 benchmark runs across research, writing, data work, multi-stage workflows, and coding.

DeepSeek won the work I do most. Qwen beat it on the one test I would have bet money on.

Let’s get the unfair part out of the way

This isn’t a clean comparison. I want to say that up front so nobody quotes me as some kind of authority.

Qwen is a 27-billion-parameter dense model. DeepSeek is a 284-billion-parameter mixture-of-experts model that activates roughly 13 billion parameters at a time. Qwen runs on 1 Mac Studio. DeepSeek is split across 2 connected GX10s. On top of that, I’ve spent months tuning my Qwen setup, while DeepSeek showed up last week.

A real researcher would control for all of that. I’m not a researcher. I had a much more selfish question: given the hardware sitting in my office right now, which model should do which work?

That framing comes from a workshop I taught a while back. My usual room is founders and knowledge workers who care about analysis and systems. This one was full of hair salon owners. They didn’t need research synthesis. They needed marketing copy and images, fast. I ended up rebuilding the whole workshop around Gemini because it fit their actual work better.

Best model is always a question about the job, never about the leaderboard.

Before the results, 4 things worth understanding

Here’s where I’m headed, so you know the detour is worth it. DeepSeek won most of the work I do every day. Qwen beat it at structured coding. And the biggest surprise of the whole exercise had nothing to do with either model.

But those results don’t mean much on their own. “Model A scored 4.8” is trivia. What makes it useful is understanding why one model needed 2 machines while the other one fit on a single desktop with room to spare, because that’s the part that determines what this costs you and what it can do inside your business.

So before the numbers, 4 ideas. No math, no jargon you have to keep. If you run a business and you’ve been hearing “local AI” without a clear picture of what that actually involves, this is the part that gives you the picture. (For the case that it’s worth caring about at all, I made that argument separately in Local AI Is Good Enough Now.)

1. Big model doesn’t mean expensive to run

Picture Qwen as a company where almost every employee shows up to every meeting. Every request pulls in nearly the whole org.

DeepSeek is a much bigger company with actual departments. 284 billion parameters on the payroll, but a router pulls in about 13 billion for any given piece of the answer.

That gives DeepSeek a much larger library to draw from without paying for all 284 billion parameters on every word. It’s why a model 10 times Qwen’s size can answer just as fast.

Nice in theory. The catch is that the whole company still needs an office, even on the days most of them stay home. Every one of those 284 billion parameters has to physically sit in memory, waiting to be called.

Which is the constraint that decides what hardware you need to buy.

2. You can trade a little quality for a lot of memory

A model is billions of learned numbers called weights. Quantization stores those numbers at lower precision so the whole thing needs less memory.

Think of a photo on your phone.

A 16-bit model is the original file. 8-bit is a high-quality export you’d never notice the difference on. 4-bit is more compressed, still looks good on a screen. At 2-bit you start seeing where detail went missing.

That maps almost exactly to what I’ve experienced. I genuinely can’t tell good 8-bit from 16-bit. At 4-bit there’s a small quality hit that’s usually worth the memory I get back. Below 4-bit, I start catching more mistakes and noticeably weaker reasoning.

My Qwen model runs 4-bit and takes up 15.3GB. That’s it. 15.3GB on a machine with 256GB.

DeepSeek runs a mix of FP4 and FP8 and takes up 155.4GB. The official model card lists 284 billion total parameters, 13 billion active, and a 1-million-token context window.

People are already running experimental 2-bit versions of DeepSeek on a single DGX Spark, which is genuinely impressive. I’d rather run the higher-quality version across 2 machines. This thing does long agent jobs while I’m asleep, so quality beats convenience.

That decision is what created my next problem.

3. When a model won’t fit, you split it across machines

Each GX10 advertises 128GB, with about 121GB you can actually use. A 156GB model doesn’t fit on either one. Not close.

Tensor parallelism is what makes it work.

Imagine 2 students sharing one enormous textbook. Each one works through part of the calculation, then they compare notes and combine their answers into a single response.

The full checkpoint sits on both machines’ drives. In working memory, the model is split between them, roughly half the weights each, with the leftover memory handling calculations and active conversations.

Worth knowing: both machines work on every single answer. If one goes down, DeepSeek stops. The second GX10 isn’t a backup or a failover. It’s half the engine.

Which sounds fragile, so I’ll be honest about how much I actually know. It’s been running 48 hours and neither node has dropped. That’s not long enough to promise you anything. Ask me in a month.

There’s one more thing competing for all that memory, and it’s the one you’ll actually feel.

4. The model’s memory of your conversation is separate

The model needs memory for the conversation itself. That working memory is the KV cache.

Open-book exam. The weights are what the student studied before walking in. The context window is every book they’re allowed to have on the desk. The KV cache is the notebook they’re scribbling in as they go.

Longer conversations fill more pages. More simultaneous conversations need more notebooks.

Since Qwen is only 15.3GB, the Mac Studio has enormous room left over for context. DeepSeek’s weights eat most of the memory, so its serving setup compresses the KV cache to compensate.

The result is up to 1 million tokens in a single conversation. Honestly this is the change I feel most day to day. You know that thing where you’re deep in a Slack thread with an agent, it compacts, and suddenly it’s forgotten the detail you gave it 40 messages ago? That mostly stopped happening. PiNoy just… keeps going.

The cluster handles up to 6 active sequences, but they’re all sharing the same memory pool. 6 tables, 1 kitchen. Everyone can order. Everyone can’t order the banquet at the same time.

If you’re evaluating this for a business, that’s the practical ceiling to ask about. Not “how smart is the model,” but how many people or automations can lean on it at once before it slows down.

That’s the whole primer. Departments instead of everyone in the room, a quality-for-memory trade, splitting a model across machines when it won’t fit, and separate memory for the conversation. Those 4 ideas are what the rest of this is measuring.

It’s fast enough that I stopped noticing it

Here’s what surprised me first, before any of the scoring.

I expected a model split across 2 machines over a network link to feel sluggish. Time to first token is around 150ms. Generation runs 70 to 80 tokens per second. That’s fast enough that it finishes before I finish reading, which is the only speed benchmark that has ever mattered to me.

The per-task times tell a more useful story than raw throughput. On conversion copy, DeepSeek’s median run took 4 minutes against Qwen’s 7.5. On the multi-file coding feature, 3 minutes against 5. Research was a wash at 2.5 minutes each.

Qwen won on speed exactly once, on the transcript article: 10.5 minutes median against DeepSeek’s 13.5.

Then there’s the run I keep thinking about. On the interrupted-coding-repair task, one Qwen run hit my 90-minute cap and got killed for response length. Not a wrong answer. Just 90 minutes of a model talking itself in circles until the controller pulled the plug.

Speed was never going to settle this, though. Fast and wrong is worse than slow and right, especially when the thing is running unsupervised. So here’s how I checked the work.

How I actually tested this

I picked 7 kinds of work I really do:

1. Source-grounded technical research
2. Writing an article from a transcript
3. Writing conversion copy from a spec
4. Completing a multi-stage content workflow
5. Transforming and verifying data
6. Resuming an interrupted coding job
7. Implementing a multi-file software feature

Each primary task ran 3 times per model, which gave me 42 matched runs. I added 9 DeepSeek-only controls to check stock Pi against my custom harness and reasoning mode.

Automated checks handled the objective stuff: required files present, tests passing, workspace boundaries respected.

Research and writing also went to a blind judge, GPT-5.6. Deliberately not one of the 2 models being tested, because a model grading its own homework is not evidence. It saw the outputs with no idea which model produced what, then scored grounding, instruction following, structure, writing quality, and usefulness. Scores were sealed before anything got unblinded.

If building a grading system for AI output sounds like overkill, it’s the same discipline I used when I built an eval suite to prove a client’s assistant wasn’t lying. You can’t trust what you haven’t tested.

One rule I set before starting: keep the failures. If a run didn’t produce the artifact, it got the minimum score. No quiet reruns, no “that one doesn’t count.” Quiet reruns are how you benchmark your way into believing whatever you already wanted to believe.

What won

Workflow	Result
Technical research	DeepSeek
Article from a transcript	DeepSeek in blind review
Conversion copy	DeepSeek
Multi-stage content workflow	Tie
Data transformation	Tie
Interrupted coding repair	Tie
Multi-file software feature	Qwen

Research wasn’t close. DeepSeek scored 5.0, 4.8, and 4.8 out of 5. Qwen came in at 3.4, 1.0, and 3.6.

That 1.0 is my favorite thing in this entire benchmark, because of how it happened.

Qwen wrote the report. A genuinely fine report, 1,450 words, right inside the 1,200 to 1,600 target. Then it saved the file to sources/report.md instead of report.md. Wrong folder, one level down.

Then it went to verify its work and ran a grep. Then it ran the identical grep again. Then again. My harness caught it and injected a warning: you’ve run the exact same call 3 times with no progress, do something different. Qwen replied “I can see the grep results clearly. Let me count the word count now,” and ran the same grep. The harness warned it at 4 times. Same reply, same grep. At 5 times. Same reply, same grep.

6 identical greps, 4 warnings it acknowledged and ignored, then the run aborted at 150 seconds. The grader looked for report.md, found nothing, and scored it 0 words.

The work was done. It just filed it in the wrong drawer and then forgot how to walk out of the room.

Writing is where it got genuinely interesting.

On the transcript article, my automated score actually favored Qwen. If I’d stopped there, I’d have written the opposite article. Then the blind judge read the work. DeepSeek’s median was 4.8. Qwen’s was 1.8, and all 3 Qwen runs hit the judge’s fabrication cap.

What “fabrication” meant in practice: source claims attached to the wrong habits. An OmniFocus execution list that appears nowhere in the transcript. Invented physiology, presented as science.

That last one is the one that would have gotten me in trouble. Confident, plausible, completely made up, published under my name.

My checker was grading the shape of the answer. The judge was grading whether it was true. That gap is the most useful thing I learned in this entire exercise. It applies to every benchmark you’ll ever read, including mine.

Before this turns into a DeepSeek commercial: DeepSeek also scored a 1.0, on that same writing task. Its stream got truncated in the middle of a write tool call and the run died with no file at all. That is the exact failure mode I’d seen a day earlier in a different benchmark, where 5 of 6 runs died mid-edit. Large writes are where DeepSeek is most fragile. Recent updates to my pi-local harness have made it noticeably better, but I’m not going to pretend it’s solved.

Then Qwen turned around and won the multi-file feature test outright. Every requirement, all 3 runs. DeepSeek scored 86%, 100%, 86%.

Both DeepSeek misses were the same hidden test. Rules are supposed to be evaluated in order, first match wins, and DeepSeek picked the second matching rule instead of the first. Same wrong answer twice. One of those runs also serialized the output JSON with alphabetized keys instead of the field order the spec asked for, which is such a specifically annoying bug that I laughed out loud.

A repeated failure is a known weakness. A random one is a reliability problem. I’d much rather have the first. This was the first.

The 284-billion-parameter model lost to the 27-billion-parameter model at structured coding. Size didn’t save it.

The result that embarrassed me

I ran 9 extra DeepSeek-only controls to answer a question I thought I already knew the answer to: does my custom Pi harness beat stock Pi?

It does not.

Stock Pi scored a perfect 1.000 across all 3 runs on the multi-stage content workflow. My tuned harness scored 1.000, 1.000, and 0.714. On interrupted-coding repair, stock Pi got 1.000, 0.981, 1.000 against my 1.000, 0.652, 1.000.

I’ve been building custom extensions for months and the default configuration quietly beat me on 2 out of 2 cells.

The one thing that saves my ego: one stock run escaped the workspace with a relative path traversal, which my harness blocks. So stock scored higher and broke a rule I care a lot about when an agent is running unsupervised at 3am. That guardrail is worth the points. The rest of my tuning apparently was not.

The reasoning-mode control was cleaner. Reasoning on scored a perfect 1.000, 1.000, 1.000 on research against 1.000, 0.933, 0.933 with it off. It also burned about 2.25 times the wall time. Worth it for research. Not worth it for bounded implementation steps, where it spends forever inspecting before it makes a single edit.

If you build tooling around a model, test the tooling too. Otherwise you’re just measuring your own assumptions. I’ve argued before that the harness matters more than the model, and I still believe it. I just got a useful reminder that believing it doesn’t make my particular harness good.

I’m using the model that lost

Qwen won the coding test. I’m making DeepSeek my primary coding agent anyway.

I want to sit with that for a second. It’s the opposite of what a scoreboard tells you to do. It’s also the actual lesson of this whole exercise.

Qwen is too slow for how I work. Not slow in the abstract. Slow in the specific way that matters when I’m on a client build with several agents running at the same time and I’m the one waiting on all of them. I run a dedicated agent per client, so the real question is never “how fast is it.” It’s how fast is it, times 4 lanes, while somebody is paying me by the hour. 3 minutes against 5 minutes sounds like nothing until it’s multiplied like that. And I have not forgotten that capped 90-minute run.

For an overnight job, none of that matters. Kick it off, go to sleep, read the results with coffee. Qwen is genuinely fine there and it stays on that duty.

But during a working day, throughput is the constraint, not the last 14 points on a benchmark.

Here’s the other half. This is the part the raw scores hide.

Once you constrain DeepSeek, it’s excellent. Narrow the scope. Keep the edits small. Hand it tests it has to pass. Under those conditions it’s arguably better than Qwen, and faster on top of it. Its failures in my benchmark were failures of freedom. It picked the wrong rule when nothing forced the ordering. It alphabetized JSON keys when nothing checked the field order. It blew up on a huge write nobody told it to break into pieces.

Every one of those is fixable with a tighter contract. Being consistently wrong in a predictable way is a design problem. That’s a much better problem to have than an unpredictable one.

So the pipeline I’m running looks like this:

A frontier model does the planning and the architecture. DeepSeek writes the code inside that plan, reasoning off, small steps, tests it must satisfy. Then a different frontier model reviews the result before anything ships.

Expensive models at both ends where judgment lives. The cheap fast local model in the middle doing volume, fenced in on both sides.

Qwen keeps 2 jobs. Anything overnight and unsupervised, where slow is free. And anything involving images, because Qwen can see screenshots and DeepSeek can’t, which turns out to matter more often than I expected.

The harness is the rest of the work. Those constraints don’t apply themselves, and I’d guess I’m about 2 weeks of tuning away from where I want my Pi setup to be. That’s the real project now. The model choice took 51 runs. Making the model behave is going to take longer.

Which is why I don’t think of the benchmark as a verdict anymore. It’s a map of where to put the guardrails.

Local AI is the new overclocking

When I was a teenager I was obsessed with overclocking CPUs.

Change a voltage setting, improve the cooling, run another benchmark, chase a few more megahertz out of a machine you already owned. I lost entire weekends to numbers nobody else would ever notice or care about.

This is the adult version of that hobby, except the numbers finally matter to someone.

Now I’m messing with quantization levels, context windows, model routing, agent harnesses. Run a benchmark, find the weak spot, change the setup, run it again. Same loop, better stakes.

I started experimenting with local AI back in February, fully expecting it to stay a hobby. Anytime I walk into a business now to implement AI, it’s one of the first things I look at, especially here in Austin where a lot of clients need privacy, control, or something that can run entirely inside their own building.

My biggest hardware regret of 2026 is buying the 256GB Mac Studio instead of the 512GB. Which is a little absurd when my favorite Qwen model is 15.3GB. But that extra headroom would let me test things like GLM-5.2 at roughly 753 billion parameters. When the next Mac Studio ships, I’m buying the biggest memory config on the page and not thinking twice.

And the largest model will probably still lose to something smaller on some test I care about. That’s exactly why I keep running them.

If you’re getting into local AI, don’t start with a leaderboard. Pick 1 job you do repeatedly and understand deeply. Run 2 models against it 3 times each. Judge the actual output, not the parameter count.

Then be willing to pick the loser. Read your own results closely enough to see why each model failed, because that tells you which one you can build around. That’s a different question from which one scored higher. It’s also the one that decides what you should actually run.

This keeps happening to me, by the way. I almost wrote off a different local model on a first read of the scores, then rebuilt my coding setup around it after one overnight run. The scoreboard is where you start looking, never where you stop.

That one test will teach you more than every benchmark chart on the internet.