Benchmark Results

Capability matrix (built-in probe test)

The standardized test (ollama-agent probe) measures per model and per endpoint: complexity levels 1–5, largest context size handled (needle test up to 256k, call timeout 10 min), response times, and stability. All 25 models were tested on both lab servers (probe show for the full table); here are the usable candidates on the fast server (11434):

Model	Levels	Context (actual)	Tok/s	warm	Note
gemma4:26b-a4b-it-qat	5/5	256k ✓	35.3	6.6s	full declared window usable (256k in 4m08s) — best all-rounder
gemma4:e4b	5/5	128k (model max)	94.7	0.8s	128k in 25s! — ideal router + fast worker
qwen3.5:9b	5/5	256k ✓	98.7	4.0s	256k in 1m22 — fastest large-context model
gemma4:12b-it-qat	5/5	256k ✓	71.2	3.7s	balanced
qwen3-coder:latest	5/5	256k ✓	30.9	1.0s	256k just under the timeout (9m08) — code reference
gpt-oss:20b	5/5	128k (model max)	52.0	2.2s	solid
krishairnd/Gemma-4-Uncensored	5/5	128k (model max)	54.0	2.1s	uncensored option
nemotron-cascade-2	5/5	128k	28.7	4.8s	256k failed on content (needle not found)
glm-4.7-flash:q4_K_M	5/5	128k	26.8	12.9s	q8_0 variant: 64k timeout, not worth it
llama3.1:8b	2/5	128k	77.7	0.5s	fast, but fails at logic + tool chaining

Failed (selection): mistral/mistral-nemo (calculate incorrectly, 0–1/5), deepseek-r1 (no tool support), qwen3.5:27b and gemma4:31b (2–4 tok/s, context breaks down early), gurubot/girl (0/5). On the 11436 server the same models pass the same levels, but everything is 2–5× slower and contexts end 1–2 levels earlier (timeouts).

Two lessons from this run:

1. The matrix is a server property, not a model property. The same gpt-oss:20b handles the 32k test in 27 seconds on the NVIDIA instance and runs into the 10-minute timeout on the CPU instance. That’s why the probe measures per endpoint, and the agent calculates using the minimum across all servers the pool can route to.
2. The probe reveals configuration mistakes: the lab servers had been mentally swapped — port 11434 is the NVIDIA instance (RTX 3080 Ti), port 11436 the deliberately CPU-only instance with a 262k default context. The measurements (77 tok/s = GPU class, size_vram in /api/ps) uncovered this before the wrong assumption made it into operation.

The detailed values (all level timings) live in state_dir/model-matrix.json; the agent automatically uses the measured context sizes for history compaction. Level results (which model handles which complexity) were identical on both servers — capability is model-side, speed is server-side.

Measured with cmd/modelbench against an Ollama 0.30 server (RTX GPU with 12 GB VRAM). To test your own server:

go run ./cmd/modelbench -host http://<server>:11434 -models "mistral:latest,llama3.1:8b,..."

Three disciplines per model:

• Arithmetic: simple correctness + response speed (the first call’s time includes model loading)
• Tool call: does the model correctly invoke a tool (structured = native format, salvaged = rescued by the parser, none = not at all) and does it relay the result?
• Classification: suitability as a router model — sorting 4 test inputs into categories (JSON schema enforced, thinking disabled)

Results (as of 2026-07, Ollama 0.30.10)

Model	Size	Arithmetic	Tool call	Classification	Time/classif.
mistral:latest	4.4 GB	ok, very fast	structured ✓	4/4	1.9s
llama3.1:8b	4.9 GB	ok	structured ✓	4/4	2.1s
qwen3-coder:latest	18.6 GB	ok	structured ✓ (6.6s)	4/4	1.2s
gemma4:12b-it-qat	7.2 GB	ok	structured ✓	4/4	4.6s
qwen3.5:9b	6.6 GB	ok, but slow (thinking)	structured ✓	4/4	3.2s
gpt-oss:20b	13.8 GB	ok	structured ✓	4/4	12.7s
mistral-nemo:latest	7.1 GB	wrong (17+25=32!)	structured ✓	4/4	2.5s
deepseek-r1:8b	5.2 GB	ok, slow	no tool support	4/4	44s

Findings

1. mistral:latest is the best router: fastest correct classification, smallest model. llama3.1:8b is an equally good fallback.
2. llama3.1:8b is the best workhorse for chat and tool calling: reliable, native tool format, no thinking overhead.
3. qwen3-coder is the first choice for code — and also an excellent classifier on the side. But: 18.6 GB doesn’t fit in 12 GB VRAM (partial CPU offload, noticeably slower) and it occasionally emits tool calls as XML text (caught by the salvage parser).
4. Use thinking models (qwen3.5, deepseek-r1) with care: they “think” before every answer (30–50s). For router/summarize calls the platform disables thinking automatically; in chat, the latency remains. deepseek-r1 also cannot use tools — unsuitable as an agent model.
5. mistral-nemo fails: calculates incorrectly (17+25=32). Do not use for factual tasks.
6. Models > 12 GB (gpt-oss:20b, qwen3-coder) run on the GPU only with CPU offload — it works, but costs noticeable speed.

The context depth test changes the picture

The needle test (above) only tells you whether a model finds a fact in a large context. The depth test (probe deep) measures whether it actually understands the context — and that reshuffles the recommendation:

Model	Context	Deep score	Retrieval	Multi-hop	Aggregation
nemotron-cascade-2	128k	100%	3/3	4/4	6/6
glm-4.7-flash:q4	128k	77%	3/3	4/4	3/6
gpt-oss:20b	128k	62%	3/3	2/4	3/6
qwen3.5:9b	128k	54%	3/3	4/4	0/6
gemma4:e4b / gemma4:12b	128k	38%	3/3	2/4	0/6
qwen3-coder	256k	38%	3/3	2/4	0/6
mistral	32k	31%	2/3	2/4	0/6
gemma4:26b-a4b	256k	Timeout*	—	—	—

*gemma4:26b: the depth test at large context can’t be run on the 12 GB card (>60 min per run due to RAM offload). Capability 5/5, but unsuitable for large contexts on this hardware.

Two key findings:

1. Retrieval ≠ understanding (the RULER effect): almost all models find facts (3/3), but when aggregating across the whole document, most collapse to 0/6. A pure needle test would have waved them all through as “128k ✓”.
2. The previous all-rounder gemma4:26b is out for large context — not due to incapability, but because the depth test doesn’t run in a reasonable time on the 12 GB card. For large-context work, nemotron-cascade-2 (100%) wins.

Recommended task mapping

The full derivation and step-by-step guide live in the Model Guide. Short version of the mapping for this lab setup:

Category	Model (fallback)	Reason
router	gemma4:e4b (mistral)	fastest reliable classification
chat	qwen3.5:9b (gpt-oss:20b)	5/5, 98 tok/s, good all-rounder
code	qwen3-coder (gemma4:26b)	code specialist, 5/5
tool_calling	gpt-oss:20b (qwen3.5:9b)	strong tool chaining, deep 62%
summarize	nemotron-cascade-2 (glm-4.7-flash)	highest deep score (100%) for document understanding

Notes:

• Pool effect: the agent calculates using the conservative context minimum across all endpoints. As long as the slow CPU instance (11436) is in the pool, min(nvidia, cpu) applies for a model. For large-context work, remove the CPU instance from the config or accept it as a failover backup.
• Server topology (docker-compose on the ai-server): ollama-nvidia port 11434 (RTX 3080 Ti, KV cache q8_0), ollama-cpu port 11436 (deliberately CPU-only, 262k context), AMD instance on 11435 prepared but disabled.