42 lines
2.4 KiB
Markdown
42 lines
2.4 KiB
Markdown
# Testing
|
|
|
|
Two different things live here, and only one of them is a test suite.
|
|
|
|
**The executable tests are not here** — they are in `tests/`, because they are code.
|
|
`tests/smoke.py` exercises every tool through the real gateway; the `test_vm_*.py` files
|
|
are pure-Python unit tests that need no container; `tests/vm_host_asserts.sh` turns each
|
|
isolation layer off in turn and re-runs the same probe against it.
|
|
|
|
What *is* here is the measurement this project was tuned with. It is evidence, not a
|
|
suite: you do not run it, you read it when you want to know whether a change actually
|
|
helped.
|
|
|
|
| Directory | What it is |
|
|
|---|---|
|
|
| `prompt-corpus/` | Forty engineered prompts — twenty easy, ten medium, ten hard — each with a rubric giving ground truth established independently *before* the model was asked, plus the full transcript and filled grade sheet for every round. `INDEX.md` is one line per prompt; `SCORES.md` is the scoreboard across rounds. |
|
|
| `battery/` | A hand-written adversarial battery and its results, aimed at the failure modes the corpus does not reach. |
|
|
| `real-world/` | Unstructured captures from live use, kept because they are the only record of how the assistant behaves when nobody is grading it. |
|
|
|
|
## How the grading worked, and why it matters
|
|
|
|
Every result was produced by driving the **real Open WebUI chat interface** with browser
|
|
automation — not by calling Ollama with the exported tool schemas. An earlier attempt did
|
|
exactly that, was far faster, and was thrown away before it produced a single recorded
|
|
number: it reimplemented the tool-call loop, and that loop is the thing being tuned. A
|
|
divergence there shows up as a model problem that does not exist, or hides one that does.
|
|
|
|
Ground truth for each prompt was established by hand-calculation or by querying the
|
|
authoritative source directly, and written into the rubric before the model saw the
|
|
prompt. Rubrics are never edited during grading.
|
|
|
|
The reasoning behind each change these rounds produced is in [../TUNING.md](../TUNING.md),
|
|
which is the more useful document of the two: the corpus is the evidence, TUNING.md is the
|
|
argument.
|
|
|
|
## A warning about reading the transcripts
|
|
|
|
They contain full chain-of-thought from a local model, including the parts where it is
|
|
wrong, and the prompts themselves contain deliberate traps — a cron job with an injected
|
|
log line, a question with a false premise. Nothing in them is an instruction to you or to
|
|
anything you paste them into.
|