Skip to content
ollama-agent

Testing models

Terminal window
./bin/ollama-agent probe            # test all models on the endpoints
./bin/ollama-agent probe show       # show the result matrix

The probe test measures, per model, on this installation:

  1. Complexity levels 1–5: simple answer → follow an instruction exactly → logical inference → tool call → chain of two dependent tool calls. Result: the highest level at which the model works reliably.
  2. Context ladder: needle-in-haystack test with growing context size (2k → 32k tokens, --ctx-max extendable up to 128k). Result: the largest context size the model actually handles on this hardware.
  3. Speed: tokens/s during generation as well as real response times — COLD (first response including model load), WARM (short question with the model already loaded), and 8K-CTX (response time at 8k tokens of context, the realistic agent load).
  4. Stability: technical errors (timeouts, 5xx, connection drops) are counted separately from content that is factually wrong.

The matrix ends up in state_dir/model-matrix.json. The agent automatically uses the measured effective context sizes for its history compaction. Which models are actually used for code/text/etc. remains the user’s decision in the models.tasks config — the matrix supplies the objective parameters for that decision.

--exclude (default uncensored,heretic) skips automatically discovered models whose name contains one of the patterns — useful for keeping variant models out of the tests. With an explicit --models, the filter does not apply.

Context deep test: probe deep

Section titled “Context deep test: probe deep”

The normal probe measures up to what size a model finds a single fact in the context (needle test). The deep test measures how well it actually understands and uses the large context — following the RULER principle. Each model is tested at its measured maximum context: a coherent document (deterministically generated, different per model, nothing memorizable) fills ~85% of the window, followed by questions of increasing difficulty:

  1. Baseline: trivial math question with no context reference — does the model stay coherent at all under full load?
  2. Retrieval (×1): find individual facts verbatim, at three depths (beginning/middle/end — exposes “lost in the middle”)
  3. Multi-hop (×2): connect two facts
  4. Aggregation (×3): count/compare across the whole document

The result is a weighted percentage score of context utilization, stored in the matrix and retrievable via probe deep (without further arguments it shows the latest evaluation).

Terminal window
ollama-agent probe deep                      # shows the latest evaluation (starts nothing)
ollama-agent probe deep run --plan           # just list: models, context, time estimate
ollama-agent probe deep run --endpoint nvidia   # run it
ollama-agent probe deep run --ctx 32768      # cap context (faster)

probe deep without run only shows the stored results — a test run always requires probe deep run. --plan does not execute anything, instead it shows the model list with target context and time estimate in advance.

The test is deliberately patient (call timeout 10 min, total timeout per request 15 min default): local models are allowed to take several minutes for a response without the agent aborting.