Model Guide

This guide explains how to pick the right models for your own tasks from the measurement data and map them in the config. It uses the three built-in tests (probe, probe deep) as the decision basis — the concrete numbers come from the lab setup (RTX 3080 Ti / 12 GB VRAM + 64 GB RAM); on your own hardware, please measure yourself.

1. The three axes of every task

No model is best at everything. Every task sits somewhere on three axes:

Axis	Question	Measured by
Context size	How much text does the model need to hold in mind at once?	probe (needle ladder) + probe deep (real usage)
Difficulty	Just retrieve, or connect, infer, aggregate?	probe (levels 1–5) + probe deep (tiers)
Speed	Does the answer need to arrive in seconds, or can it take a while?	probe (tok/s, warm time)

The key insight from the tests: “supports 256k context” does not mean “uses 256k context.” Almost all models pass the needle test (finding a single fact), but when it comes to real comprehension (probe deep), most fall apart — especially with aggregation across the whole document. That’s why both tests are needed.

2. What the tests deliver

ollama-agent probe run --endpoint <name>        # capability (level 1–5), needle context, tok/s, warm time, GPU/CPU
ollama-agent probe deep run --endpoint <name>   # real context comprehension score (0–100 %)
ollama-agent probe show                         # view capability matrix
ollama-agent probe deep                         # view comprehension scores (starts nothing)

• Level 1–5 (probe): 1 simple answer, 2 follow instruction, 3 logical inference, 4 tool call, 5 tool chain. A model below 4/5 is not suitable as an agent model (cannot chain tools).
• Needle context (probe): the largest context size at which it can find a single fact. An upper bound, not comprehension.
• Deep score (probe deep): weighted percentage of how well the model actually uses the context (retrieval ×1, multi-hop ×2, aggregation ×3). This is the key metric for tasks with large context.
• Tok/s + warm time: speed. Warm time = typical response time for a short question with the model already loaded.

3. Task profiles (the “levels in between”)

Map your real-world usage to one of these five profiles:

Profile	Context	Difficulty	Speed	What to select on
A — Router/classification	tiny	trivial (choose 1 category)	critical	smallest model with fast, stable JSON output
B — Fast chat	small (<8k)	low–medium	important	high tok/s + level 5/5 + short warm time
C — Tool tasks	medium (8–32k)	medium (tools, multi-hop)	doesn’t matter	level 5/5 + solid deep score
D — Code	medium–large	high (precision)	comfortably fast	best code model with 5/5
E — Document comprehension	large (64k+)	highest (aggregation)	doesn’t matter	highest deep score, speed secondary

4. Model recommendation per profile (lab data)

A — Router / classification → gemma4:e4b (fallback mistral)

Classification is a mini-task with schema-enforced JSON output. What matters here is speed and reliability, not context. gemma4:e4b: 5/5, 0.8 s warm — more reliable than mistral (only 1/5 on real tasks, but usable as a fallback for pure classification at 0.4 s).

B — Fast chat → qwen3.5:9b (fallback gpt-oss:20b)

qwen3.5:9b: 5/5, 98 tok/s, 3.6 s warm, deep 54 % — the best trade-off between speed and capability for short dialogs. gemma4:e4b is even faster (0.8 s), but weaker at connecting information.

C — Tool tasks → gpt-oss:20b (fallback qwen3.5:9b)

gpt-oss:20b: 5/5, deep 62 %, 2.2 s warm — strong tool chaining and decent context usage at good speed.

D — Code → qwen3-coder:latest (fallback gemma4:26b-a4b)

qwen3-coder: 5/5, 256k needle, logic level under 1 s — the specialized code model. Note: sometimes emits tool calls as XML text, which the salvage parser catches.

E — Document comprehension/aggregation → nemotron-cascade-2 (fallback glm-4.7-flash:q4)

nemotron-cascade-2: the clear winner for large context — 100 % deep score, the only model to also complete aggregation fully (6/6). Slower (28 tok/s, 4.8 s warm), but for this profile speed doesn’t matter. glm-4.7-flash:q4 (77 %) is the solid runner-up.

Not recommended

• llama3.1:8b (2/5): fails at logic and tool chaining — only for trivial outputs.
• gemma4:26b-a4b for large context: capability 5/5, but the deep test is not feasible on the 12 GB card (>60 min per run due to RAM offload). Good for short contexts, unsuitable for large ones on this hardware.
• gemma4:e4b/gemma4:12b for large context: fast and 5/5, but deep only 38 % (aggregation 0/6) — top for profile A/B, not for E.

5. Mapping it in the config

The profiles are mapped in models.tasks to task categories. Each category has a preference list (first available model wins) and a description that helps the router with classification.

"models": {
  "default": "qwen3.5:9b",
  "default_category": "chat",
  "tasks": {
    "router":       { "models": ["gemma4:e4b", "mistral:latest"], "options": { "temperature": 0 } },
    "chat":         { "models": ["qwen3.5:9b", "gpt-oss:20b"],
                      "description": "kurze allgemeine Fragen, Gespräch, Wissen" },
    "code":         { "models": ["qwen3-coder:latest", "gemma4:26b-a4b-it-qat"],
                      "description": "Programmcode schreiben, korrigieren oder erklären" },
    "tool_calling": { "models": ["gpt-oss:20b", "qwen3.5:9b"], "options": { "temperature": 0 },
                      "description": "Aufgaben mit Werkzeugen oder mehreren Schritten" },
    "summarize":    { "models": ["nemotron-cascade-2:latest", "glm-4.7-flash:q4_K_M"],
                      "description": "einen langen Text zusammenfassen oder daraus Informationen ableiten" }
  }
}

Why this mapping:

• summarize means “read and condense a long text” = profile E → the model with the highest deep score (nemotron), speed doesn’t matter.
• chat and tool_calling = profile B/C → fast, capable models.
• code = profile D → the specialized code model.
• router = profile A → the fastest reliable model.

Word the description fields clearly — they noticeably affect the router’s accuracy.

6. Procedure for your own server

1. Measure capability: ollama-agent probe run --endpoint <name> — separates the wheat from the chaff (below 4/5, wrong answers). Skip variant models with --exclude uncensored,heretic.
2. Measure comprehension (only for the 5/5 models you’re considering for large context): ollama-agent probe deep run --endpoint <name>. On weaker hardware, set --ctx 65536, otherwise 256k tests will hit the time/memory limit.
3. Assign profiles: For each of the five profiles from section 3, pick the best model following the section 4 logic — for A/B/C/D by speed+capability, for E by deep score.
4. Write the config (section 5) and verify with ollama-agent models that the chosen models are available on a healthy endpoint.
5. Use a small set of models consistently: every model switch costs 30–60 s of load time. A compact set (router + 1 all-rounder + 1 code + 1 deep model) beats ten rarely used ones.

7. Hardware note

The deep score is a server property, not a pure model property: on a card with little VRAM, Ollama offloads the KV cache to RAM at large context sizes — this works, but is slow and can fail on time/memory at very large contexts (256k). The agent therefore calculates using the conservative minimum of the measured context sizes across all endpoints the pool can route to. Anyone who wants to use large contexts in production needs either more VRAM or must accept longer response times (for which timeouts are generously set to 15 min).