It was close to midnight and I was about to shut my laptop. A new open-weights model called Laguna-S-2.1 had come out that day from the poolside team, and I’d spent part of the evening running it through my usual battery. The results were fine. Not exciting. On the short coding tasks I test with, it didn’t beat Qwen3.6-35B-A3B, the model I already run in production on my GX10 in the office. My honest reaction was “okay, cool, next.”

Then I scrolled X one more time before bed. The poolside people kept saying the same thing about their model, over and over: it’s built for long-horizon work. Tasks that run for hours. Not clever one-shot answers.

And that stopped me. Because long-horizon is exactly the thing I’ve been fighting for months.

If those names look like license plates

Here’s the part worth understanding, because it explains the whole tradeoff.

The A3B in Qwen3.6-35B-A3B means 3 billion active parameters. The model holds 35 billion in total, but for any single word it generates it only wakes up about 3 billion of them. That design is called a mixture of experts. Picture a consulting firm with 200 people where every question gets routed to the 2 or 3 specialists who actually know the answer, instead of dragging all 200 into the room every time.

The payoff is speed. You get something close to the knowledge of a big model at the running cost of a small one. The older approach, called a dense model, fires every parameter for every word. Same size on paper, much slower to run.

Here’s the part worth being precise about, because I had it wrong at first. Laguna is a mixture of experts too. It’s the same architecture set to a very different dial: 118 billion parameters in total with about 8 billion active per token, against Qwen’s 35 billion total and 3 billion active. Laguna keeps 256 specialists on staff and pulls 10 of them into every meeting, roughly 7% of the model firing at any moment.

So Laguna isn’t slower because it’s built differently. It’s slower because it’s a much bigger building with more people in every room. Nearly 3 times the active parameters per token and more than 3 times the total weights to hold. On my hardware that works out to about 120 tokens per second from Qwen against 39 from Laguna. Both are quantized down to fit the box. Both use a trick called speculative decoding to claw some speed back.

So this was never a fast-versus-slow story, or a small-versus-big one. Two models built the same way, tuned to different points on the same dial. Fast is not the same thing as smart. The fast one is the one that lost the long job.

The problem I’d already spent hours on

I run local models on machines in my office because they’re almost free. You pay for electricity. That’s it. No token bill, no monthly cap, no meter running while an agent grinds away at something boring at 3am.

The catch is that a model on your desk is only half the system. The other half is the harness, the software that hands the model a job and keeps it honest. My harness of choice is pi. And to make pi work with Qwen for anything long-running, I had to build a whole extension: loop breakers, size guards, a ledger, nudges that shove the model forward when it starts spinning. I spent hours and hours on that thing. Fine-tuning it so /goal and /loop could survive a multi-hour job.

Why did that extension need to exist? Because Qwen is fantastic in short bursts and falls apart over long stretches. I knew that. I’d been building scaffolding around a known weakness and calling it solved.

So when a model shows up claiming it’s specifically good at the exact thing I’ve been compensating for, that’s worth 6 hours of my machine’s time. I was going to be asleep anyway.

Why I bother benchmarking at all

I don’t test new models for sport. I have clients asking for local models now, which means I need real answers about what these things can and can’t do before I put one inside somebody’s business.

So I test across a spread of jobs rather than one score. Agentic workflows, where the model has to plan and carry out several steps without a human nudging it along. Tool calling, which sounds boring and is probably the single most important skill on my list. If you want to run something like Hermes or OpenClaw, a model that fumbles tool calls is useless no matter how well it writes. Then coding, which is where most of the demand is coming from.

Here’s what that battery said the night before the long test. Every cell is the average of 5 fresh runs, with both models in their best configuration, which for both means thinking turned off. Higher is better, 1.0 is perfect.

Short tests (n=5)	Qwen3.6-35B-A3B	Laguna-S-2.1
Coding, tier 1	0.803	0.963
Coding, tier 2 (harder)	0.946	0.976
Writing a test suite that catches bugs	0.356	0.877
Agentic investigation (6 tasks)	0.921	0.903
Writing to constraints	0.963	1.00
Needle retrieval at ~199K context	12/12	12/12
Tokens burned on tier 1	12K	7K

Look at that and you can see why I shrugged. Laguna wins most cells, but the margins are small enough to be noise on anything except test-suite writing, where it more than doubles Qwen. It loses on agentic investigation, which is the thing I care about most for Hermes and OpenClaw. Nothing here says “rebuild your stack around me.”

Run enough of these and a pattern shows up. Models are not ranked on one ladder from dumb to smart. They have shapes. One is quick and broad. Another is patient and narrow. The mistake is asking which is better instead of asking which job you’re handing it.

The experiment I set up before bed

Here’s the nice thing about having machines that run at home. When you close your laptop, the work doesn’t have to stop.

I gave my Claude session a job for the night: build a test that’s genuinely long, then run both models through it and tell me what happened in the morning.

The task it built was mean, in a good way. A 10-module Python library, 259 tests, with 32 bugs deliberately seeded into it. Then it deleted one module entirely and left only the written spec, so the model had to write a parser from scratch to make those tests pass. Starting score: 169 of 259 passing. Both models got the same job, the same starting point, 400 turns to work with, and up to 5 hours of wall clock.

That’s the shape of real work. Not “write me a function.” More like “here is a codebase, it’s broken in ways nobody told you about, go fix it and don’t stop until the tests are green.”

What I woke up to

Qwen never wrote a single file.

It explored for about 20 turns, which looked healthy. Then it locked into a loop and ran the test suite over and over. 211 identical test runs. 250 turns. Zero edits. The score sat frozen at 169 the entire time, and I eventually had it killed. It wasn’t confused about the problem. It just could not convert understanding into sustained action.

The detail that stung: on a smaller version of the same test, Qwen solved it in 31 turns. Scale is the trigger. Make the job big enough and the wheels come off.

Laguna went from 169 to 257 of 259 passing. It took 61 turns and about 55 minutes. Zero regressions, meaning it never broke a test that was already passing, which is the failure mode I care most about when something is editing my code unsupervised. The score climbed the whole way: 169, then 234, then 243, 249, 255, 257. That’s a model actually working, not thrashing.

Same hardware. Same task. Same night.

The same comparison with the numbers underneath it:

The long test	Qwen3.6-35B-A3B	Laguna-S-2.1
Tests passing at the end	169 of 259 (no change)	257 of 259
Files written	0, across 250 turns	18 writes across 10 files
Bugs fixed (of 32 seeded)	0	30
Wrote the missing module from spec	no	yes, +65 tests in one go
Broke something that worked	never acted	0
Rewrote a file with identical content	never acted	0
Longest stretch with no progress	221 test runs	3 turns
Times it ran the test suite	211	8
Turns / wall clock	250 turns, killed at 26 min	61 turns, 55 min
Tokens generated	8.3K, all of it looping	73.8K of actual code
How it ended	degenerate loop, I killed it	stopped early at 257 of 259

The two rows I keep coming back to are the test-suite runs and the tokens. Qwen ran the tests 211 times and generated 8,300 tokens, which is the signature of a model reading the same failure over and over without ever deciding to change anything. Laguna ran the suite 8 times and wrote 74,000 tokens, batching a pile of fixes between each check. That’s how a person would do it.

The part I’d be lying if I left out

Laguna quit at turn 61 with 2 tests still failing. It had 339 turns left in the budget.

It didn’t crash or spiral. It decided it was done when it wasn’t. That’s a real weakness. For what I want to use this for (an agent running unattended while I sleep) something that stops early at 99% finished is still a problem I have to solve.

The good news is that this is a harness problem. Those I know how to fix. When I ran the whole thing again through pi with a completion gate in front of it, the gate rejected the model’s first claim of “done”. The model went back and corrected itself instead of quitting. That’s the fix working live, not on paper.

Also worth saying plainly: this was one run at that size. Not 5. I usually insist on 3 or more runs per test because I’ve watched the same setup score wildly differently at the same settings. When each run takes hours you make a tradeoff, and I made it knowingly. The gap between “collapsed at zero edits” and “88 tests fixed” is wide enough that I’m comfortable, but I’m not going to pretend it’s statistically airtight.

All-purpose models and specialists

The bigger lesson I’m taking from this is about how I evaluate local models at all.

Qwen is a generalist. It handles vision, which matters to me for a bunch of workflows that involve screenshots and documents. It’s roughly 3 times faster than Laguna on my hardware. For quick interactive work it’s still the right call. It’s still what serves on my production port.

Laguna has no vision. It’s built for code. On paper that’s a narrower model. My first benchmark treated that narrowness as a downside because I was measuring the wrong thing. I was testing peak performance on short tasks. The question that actually matters for the way I want to use these things is whether a model can stay coherent for 4 hours without a human in the chair.

Those are different questions. A model can win one and lose the other badly. I now think the local model world is splitting into generalists and specialists. Picking one “best local model” is going to age about as well as picking one best kitchen knife.

After all this testing I can finally say what Laguna is actually for, which is more useful to me than knowing where it lands on a leaderboard. Its purpose is long-running coding work. Not research, not vision, not quick answers. Give it a big broken codebase plus hours to sit with it. It does the thing I could never get Qwen to do.

Have you been benchmarking your models on the shape of work you actually give them? I hadn’t been. My battery was full of short tasks because short tasks are easy to score.

The technical details, for the people who want them

Both models run on the GX10, one at a time, because 121GB of memory won’t hold both. Both are quantized to NVFP4 to fit. Both use speculative decoding, which is a smaller helper model guessing the next few tokens so the big one can verify a batch instead of grinding out one at a time. The acceptance length is how many of those guesses survive on average, so higher means the trick is working harder for you.

	Qwen3.6-35B-A3B	Laguna-S-2.1
Architecture	MoE	MoE (256 experts, top-10)
Total parameters	35B	118B
Active per token	3B	~8B (~6.8%)
Quantization	NVFP4	NVFP4
Speculative decoding	MTP-3	DFlash
Acceptance length	3.50	5.46
Output, single stream	119.7 tok/s	39.1 tok/s
Output, 4 streams	255.9 tok/s	79.5 tok/s
Time to first token	99 ms	293 ms
Serving stack	vLLM	vLLM 0.25.1

That time-to-first-token number is the one people underrate. 99ms versus 293ms feels like nothing on paper. Sit in front of both for an afternoon of back-and-forth work and Qwen feels alive while Laguna feels like it’s thinking about whether to answer you. For interactive use that gap decides which one you actually reach for.

Now flip it. On a job that runs for an hour with nobody watching, time to first token is meaningless and tokens per second only sets the wall clock. Laguna finished the long task in 55 minutes at 39 tok/s. Qwen, at 3 times the speed, finished nothing at all in 26 minutes of trying. Speed only counts once the model is going somewhere.

One config note if you go running these yourself: keep thinking mode OFF for both. For Laguna it buys nothing on quality, hurts agentic work, and triples the tokens. For Qwen it does help, quite a bit on test writing, at roughly 10 times the tokens and 16 minutes a pass. I left it off for every number on this page.

What I’m doing with it now

I’m rebuilding my pi extension so it runs both models, routing by job. Qwen for the fast interactive stuff and anything involving images. Laguna as the executor for coding: bug triage, root cause, proposed fixes, the long grinding sessions on bigger projects.

That extension started life as a Qwen-specific thing, which was always a bit of a lie about what it actually is. Underneath, the loop breakers and guards and ledger were never Qwen-specific at all. So I’m pulling it apart into a neutral core with thin adapters per model family, and I’m open sourcing it. If you have a machine and a local model, you should be able to point it at your own hardware and get a coding agent that survives long jobs.

I went from almost deleting this model to planning my next quarter of building around it. All because I decided to burn an idle night on a test instead of trusting a benchmark that was asking the wrong question.

Pick the one job you’d most want an AI to do while you sleep. Build a test that runs as long as that job really takes, then set your models loose on it overnight. You’ll learn more from that one night than from a month of reading benchmark charts.