142 lines
7.1 KiB
Markdown
142 lines
7.1 KiB
Markdown
# Recommended parameters
|
|
|
|
Two models are configured here and they are tuned for different jobs. The assistant writes
|
|
prose and chooses tools; the VM sub-agent writes code and reads error messages. Numbers that
|
|
suit one do not suit the other.
|
|
|
|
Everything below was arrived at by measurement against the corpus in
|
|
[testing/](testing/README.md); the reasoning is in [TUNING.md](TUNING.md).
|
|
|
|
---
|
|
|
|
## The assistant
|
|
|
|
Set in Open WebUI: **Workspace → Models → your model → Edit → Advanced Params**.
|
|
|
|
The full set from a working deployment, not just the interesting ones — the defaults
|
|
Open WebUI leaves blank are as much a part of this as the values that were changed.
|
|
|
|
| Parameter | Value | Why |
|
|
|---|---|---|
|
|
| `num_ctx` | **163840** | Tool results are not truncated by Open WebUI — whatever a tool returns lands in the context verbatim. A research answer with three fetched pages is large. |
|
|
| `max_tokens` | **32768** | Room for a long answer without letting one run away. |
|
|
| `temperature` | 0.6 | |
|
|
| `top_p` | 0.95 | |
|
|
| `top_k` | 20 | |
|
|
| `min_p` | 0 | |
|
|
| `presence_penalty` | **1** | The opposite of the sub-agent's 0, and deliberately so. This model writes prose, where repetition is the failure mode; the sub-agent writes code, where reusing an identifier is correct. Do not copy one to the other. |
|
|
| `repeat_last_n` | 256 | |
|
|
| `num_batch` | 1024 | |
|
|
| `keep_alive` | `-1` | Never unload between turns. Here it is a string in Open WebUI's own field; the sub-agent sends it to Ollama directly and must send an **integer**. |
|
|
| **Function Calling** | **Default** (`native`), not Legacy | Legacy mode reformats tool definitions in a way that loses the request-body schema, which is where the enums and defaults live. |
|
|
|
|
### Per-model capabilities
|
|
|
|
On the same edit page, below the parameters. What matters for this project:
|
|
|
|
| Capability | Set to | Why |
|
|
|---|---|---|
|
|
| `builtin_tools` | on | Required for the tool connection to be offered at all |
|
|
| `citations` | on | The tools return citation lines meant to be lifted verbatim |
|
|
| `status_updates` | on | Needed by `live_progress_status` and by the VM sub-agent's live view |
|
|
| `vision`, `file_upload`, `file_context` | on | Ordinary; `vision` is also what makes a returned plot legible to the model |
|
|
| `code_interpreter`, `terminal` | **off** | `matlab_run` and `vm_agent` already provide execution, inside sandboxes this project controls. Leaving these on gives the model a second, unsandboxed route to the same thing. |
|
|
| `web_search` | your call | See below |
|
|
|
|
**Open WebUI has its own web search, and it overlaps with `web_search`.** If both are on the
|
|
model can reach the same capability two ways, with different output formats and different
|
|
citation behaviour, and which one it picks is not predictable. Pick one. The tool goes
|
|
through your SearXNG and squid and returns results shaped for the citation rules in the
|
|
system prompt; Open WebUI's built-in goes out on its own terms.
|
|
|
|
### System prompt
|
|
|
|
Paste [`../configs/assistant/system-prompt.md`](../configs/assistant/system-prompt.md) into
|
|
the model's System Prompt field.
|
|
|
|
**This matters more than any of the numbers.** The tool descriptions alone do not produce
|
|
the behaviour: using tools instead of recalling, checking rather than asserting, and saying
|
|
"I don't know". Those come from the prompt.
|
|
|
|
Note that it is a *file in the repo*, not something the server serves — nothing loads it at
|
|
runtime. If you change the live prompt, change the file too, or the two drift and the file
|
|
stops being a record of anything.
|
|
|
|
### `num_ctx` must match what Ollama actually loaded
|
|
|
|
Ollama allocates the KV cache per loaded model. Asking for a different `num_ctx` than the
|
|
resident one makes it **reload the model** — on a 27B Q4_K_M that is about 18 GB across the
|
|
bus, in the middle of a conversation. The value in Open WebUI, the value in
|
|
`OWUITOOLS_AGENT_NUM_CTX`, and the value the model is actually loaded with must all be the
|
|
same number.
|
|
|
|
Check what is loaded with `curl -s http://172.17.0.1:11434/api/ps | jq`.
|
|
|
|
---
|
|
|
|
## The VM sub-agent
|
|
|
|
Set in `configs/agent/params.env`. It runs on the **same Ollama and the same model**, with
|
|
its own sampling.
|
|
|
|
| Setting | Default | Why |
|
|
|---|---|---|
|
|
| `OWUITOOLS_AGENT_MODEL` | `qwen3.8:latest` | Same model as the assistant, so there is one thing loaded |
|
|
| `OWUITOOLS_AGENT_NUM_CTX` | `163840` | Must equal the resident value — see above |
|
|
| `OWUITOOLS_AGENT_KEEP_ALIVE` | `-1` | Never unload. Must be an **integer**: the string `"-1"` is rejected with `time: missing unit in duration` |
|
|
| `OWUITOOLS_AGENT_TEMPERATURE` | `0.6` | |
|
|
| `OWUITOOLS_AGENT_TOP_P` | `0.95` | |
|
|
| `OWUITOOLS_AGENT_TOP_K` | `20` | |
|
|
| `OWUITOOLS_AGENT_PRESENCE_PENALTY` | `0.0` | **Deliberately zero.** A penalty pushes the model away from tokens it has already used, and code reuses identifiers constantly. A non-zero value here produced subtly wrong variable names. |
|
|
| `OWUITOOLS_AGENT_NUM_PREDICT` | `16384` | One turn's output ceiling |
|
|
| `OWUITOOLS_AGENT_THINK` | `true` | Also accepts `low`/`medium`/`high`/`max` |
|
|
|
|
Every option is sent on every request, rather than inherited from the Modelfile. That is on
|
|
purpose: a Modelfile carrying an unnoticed `presence_penalty` is exactly the failure above,
|
|
and it is invisible from the calling side.
|
|
|
|
### Session bounds
|
|
|
|
| Setting | Default | What it bounds |
|
|
|---|---|---|
|
|
| `OWUITOOLS_VM_MAX_STEPS` | `110` | Tool calls in one session. The backstop for an agent going in circles. |
|
|
| `OWUITOOLS_VM_SESSION_SECONDS` | `7200` | Wall-clock. The backstop for everything else. |
|
|
| `OWUITOOLS_VM_TASK_DRIFT` | `12` | Steps without touching the task list before it is told it is drifting |
|
|
| `OWUITOOLS_VM_WAIT_SECONDS` | `7000` | How long one `vm_agent` call may block |
|
|
|
|
---
|
|
|
|
## The three clocks
|
|
|
|
A `vm_agent` call is a single HTTP request that stays open for as long as the session runs.
|
|
Three separate timeouts can end it, and **the turn dies at the shortest one** — usually with
|
|
the work already done and only the reporting left, which is the most confusing possible
|
|
failure.
|
|
|
|
| Layer | Where | Value |
|
|
|---|---|---|
|
|
| The app | `OWUITOOLS_VM_WAIT_SECONDS` in `configs/owuitools.env` | **7000** |
|
|
| The gateway | `location = /tools/vm_agent` in `configs/services/gateway.nginx.conf` | **7200s** |
|
|
| Open WebUI | `AIOHTTP_CLIENT_TIMEOUT_TOOL_SERVER` on the container | **7200** |
|
|
|
|
Keep the app's the **smallest**, so it is the app that answers — with a real result — rather
|
|
than a proxy cutting the connection and leaving the model with a transport error.
|
|
|
|
**Open WebUI's is not unlimited when unset.** It falls back to `AIOHTTP_CLIENT_TIMEOUT`,
|
|
which defaults to **300 seconds**. This is the single most common reason a working VM
|
|
session reports as a failure.
|
|
|
|
If you front Open WebUI with your own reverse proxy, that is a fourth clock. Raise it too.
|
|
|
|
---
|
|
|
|
## Context cost
|
|
|
|
The 22 tool schemas cost roughly **9,700 tokens**, paid on every turn of every conversation
|
|
whether or not a tool is used.
|
|
|
|
If you do not need some of them, turn them off in
|
|
[`../configs/tools.toml`](../configs/tools.toml) — that removes them from the spec entirely
|
|
rather than hiding them, so you stop paying for them. The electronics group is the cheapest
|
|
to drop if it is not your field. See [tools.md](tools.md).
|