Initial Commit

docs/setup/contact.md

@@ -0,0 +1,42 @@
+# `OWUITOOLS_CONTACT` — needed by `wikipedia_lookup` and `search_papers`
+
+Two minutes. No account, no key.
+
+## Why
+
+Wikimedia **requires** a descriptive User-Agent with a way to contact you, and returns
+**HTTP 403** without one. Their policy also rules out generic strings like `curl/8.5.0`,
+which work today and may not tomorrow. OpenAlex and Crossref read the same string to decide
+whether you get their "polite pool", which is noticeably faster.
+
+The tool server builds the header from this one value:
+
+```
+owuitools/1.0 (+<your Open WebUI URL>; <OWUITOOLS_CONTACT>)
+```
+
+## Steps
+
+1. Open `configs/owuitools.env`.
+2. Set it to an address someone could actually reach you at:
+
+   ```
+   OWUITOOLS_CONTACT=you@example.com
+   ```
+
+3. `docker compose up -d api`
+
+## Checking it
+
+```sh
+./tests/try-tool.sh wikipedia_lookup '{"topic": "Ampere"}' | head -5
+```
+
+A 403 from Wikimedia comes back as prose saying the request was refused. Anything else
+means the header was accepted.
+
+## If you leave it unset
+
+It defaults to `owuitools@localhost`, which is a valid string but not a real contact.
+Wikipedia lookups may be refused, and paper searches will be slower. Nothing else is
+affected.

docs/setup/gitea.md

@@ -0,0 +1,108 @@
+# Gitea — so the VM sub-agent can publish what it built
+
+Fifteen minutes. **Optional**: without it a session still runs, still keeps its files in
+the project folder on the array, and simply returns no link.
+
+You need a Gitea instance first — see
+[../deployment/gitea.compose.yml](../deployment/gitea.compose.yml) if you do not have one.
+
+## What it is for
+
+When a session finishes, vmd commits the project folder and pushes it to a repository named
+after the project, then cuts a release. The user gets one link. The agent never touches
+git: the token stays on the host, the commit runs with hooks disabled under a non-root
+user, and a scratch filter keeps test scripts out of what gets published.
+
+## Steps
+
+1. **Create an organisation** to hold the repositories. Signed in as an admin:
+   *+ → New Organisation*. Call it `vmlab` or whatever you like.
+
+   Set its visibility to **Limited** if you want the repositories visible to signed-in
+   users only. The repositories themselves are created public *within* that org, so the
+   org's visibility is what controls who can see them.
+
+2. **Create a bot account.** *Site Administration → Identity & Access → Users → Create
+   User*. Name it `vmagent`. Give it a long random password you will not need again, and
+   untick "require password change".
+
+3. **Add the bot to the organisation** as a member with **write** access to repositories.
+
+4. **Generate a token as the bot.** Sign in as it, then *Settings → Applications →
+   Generate New Token*. It needs:
+   - `write:repository`
+   - `write:organization`
+
+   Copy it now; Gitea shows it once.
+
+5. **Put it on the host**, in `/etc/owuivm/vmd.env`:
+
+   ```
+   OWUITOOLS_GITEA_URL=http://192.168.1.10:7200
+   OWUITOOLS_GITEA_PUBLIC_URL=https://git.example.com
+   OWUITOOLS_GITEA_ORG=vmlab
+   OWUITOOLS_GITEA_BOT=vmagent
+   OWUITOOLS_GITEA_TOKEN=<the token>
+   ```
+
+   `configs/vmd.env` is the template this file is created from; edit the live copy, since
+   `install.sh` never overwrites it once it exists.
+
+6. **Let the daemon reach it.** `vm/host/owuivmd.service` runs with
+   `IPAddressDeny=any` and an allowlist. Gitea's address has to be on it:
+
+   ```ini
+   IPAddressAllow=192.168.1.10/32
+   IPAddressAllow=172.28.0.0/16
+   ```
+
+   **Both lines.** The first is the LAN address you push to. The second is the Docker
+   network Gitea's container is on, and it is not redundant: systemd's filter is a BPF
+   program on the cgroup's packets, and by the time it sees them Docker's DNAT has already
+   rewritten the destination to the container's own address. Measured — with only the
+   `/32`, the connection times out.
+
+7. `sudo ./vm/host/install.sh && sudo systemctl restart owuivmd`
+
+## Checking it
+
+This asks Gitea, as the daemon, using the credentials it will actually use:
+
+```sh
+sudo -u owuivmd python3 - <<'PY'
+import json, urllib.request
+env = {}
+for line in open("/etc/owuivm/vmd.env"):
+    line = line.strip()
+    if line and not line.startswith("#") and "=" in line:
+        key, value = line.split("=", 1)
+        env[key] = value
+token = env.get("OWUITOOLS_GITEA_TOKEN", "")
+if not token:
+    raise SystemExit("no OWUITOOLS_GITEA_TOKEN in /etc/owuivm/vmd.env")
+url = env.get("OWUITOOLS_GITEA_URL", "").rstrip("/")
+org = env.get("OWUITOOLS_GITEA_ORG", "")
+request = urllib.request.Request(f"{url}/api/v1/orgs/{org}",
+                                 headers={"Authorization": f"token {token}"})
+answer = json.load(urllib.request.urlopen(request, timeout=15))
+print(f"OK: reached {url} and the token can see the '{answer['username']}' organisation")
+PY
+```
+
+Running it as `owuivmd` is the point — it proves the daemon can read the file *and* that
+the address filter in its unit permits the connection, which is the step most likely to be
+wrong. A timeout here almost always means step 6 was missed or only half done.
+
+Otherwise, run a session and see whether the result carries a link.
+
+## Two things worth knowing
+
+**The token is never given to the machine or to the agent container.** vmd does the clone,
+commit and push on the host and supplies the credential at push time through `GIT_ASKPASS`.
+A guest that reads its own project folder finds no credential in it.
+
+**Only the release archive downloads.** On a Gitea release page, uploaded assets are served
+`inline` — a browser opens them rather than saving them, even a `.zip`. Only the
+auto-generated *Source code (zip)* is served as an attachment. So what a user actually
+downloads is built from the commit, which is why the scratch filter operates on the commit
+rather than on the asset list.

docs/setup/github-token.md

@@ -0,0 +1,34 @@
+# `OWUITOOLS_GITHUB_TOKEN` — lifts the rate limit on `lookup_package`
+
+Five minutes. Free. **Optional** — the tool works without it.
+
+## Why
+
+`lookup_package` checks GitHub releases as one of its sources. Unauthenticated GitHub
+allows **60 requests per hour per IP**, and that is for the whole stack, not per user — a
+single conversation that asks about a handful of packages can exhaust it. A token with no
+scopes at all raises it to **5,000 per hour**.
+
+## Steps
+
+1. Go to <https://github.com/settings/tokens> → **Generate new token (classic)**.
+2. Give it a name like `owuitools`.
+3. **Select no scopes.** None are needed: it only reads public data, and a token with no
+   scopes cannot do anything else if it leaks.
+4. Set an expiry you are willing to renew.
+5. Copy the token — GitHub shows it once.
+6. Put it in `configs/owuitools.env`:
+
+   ```
+   OWUITOOLS_GITHUB_TOKEN=ghp_...
+   ```
+
+7. `docker compose up -d api`
+
+## Checking it
+
+```sh
+./tests/try-tool.sh lookup_package '{"name": "ripgrep", "ecosystem": "github"}' | head -5
+```
+
+If you were rate-limited, the tool says so in words rather than failing.

docs/setup/nvd.md

@@ -0,0 +1,31 @@
+# `OWUITOOLS_NVD_KEY` — lifts the rate limit on `lookup_cve`
+
+Ten minutes, most of it waiting for an email. Free. **Optional**, and it matters less than
+the other two.
+
+## Why
+
+`lookup_cve` asks CIRCL first, MITRE second, and only goes to NVD for CVSS scoring and CPE
+detail. CIRCL has no hard limit, so most lookups never touch NVD at all. Keyless NVD is
+documented at **5 requests per 30 seconds**, which is only a problem if you lean on the
+score breakdown.
+
+## Steps
+
+1. Go to <https://nvd.nist.gov/developers/request-an-api-key>.
+2. Fill in the form. The key arrives by email, usually within minutes.
+3. Put it in `configs/owuitools.env`:
+
+   ```
+   OWUITOOLS_NVD_KEY=...
+   ```
+
+4. `docker compose up -d api`
+
+## Checking it
+
+```sh
+./tests/try-tool.sh lookup_cve '{"cve_id": "CVE-2014-0160"}' | head -10
+```
+
+You should see a CVSS vector. Without a key you still get the description and references.

docs/setup/octave.md

@@ -0,0 +1,45 @@
+# The Octave sandbox — needed by `matlab_run`
+
+**Nothing to set up** beyond building the image, which `docker compose up -d --build` does.
+This page explains what it is, because the tool is named for MATLAB and does not contain
+any.
+
+## What it is
+
+`matlab_run` executes code in **GNU Octave**, in a container with `network_mode: "none"` —
+no interfaces, no DNS, no route anywhere. Jobs reach it through a file drop on a shared
+tmpfs, not over a socket, so there is nothing listening to reach.
+
+The tool reports three distinct verdicts, and the distinction is the point:
+
+- it ran, and here is the output;
+- it is **not valid MATLAB** (the linter caught a construct Octave accepts and MATLAB does
+  not, such as `endfor` or `!=`);
+- it failed for a reason that is not the code's fault.
+
+The second one exists because the assistant is often asked about MATLAB by people who do
+not have it. Silently accepting Octave-only syntax would teach them something untrue.
+
+## Limits
+
+Set in `configs/owuitools.env`:
+
+| Setting | Default | What it bounds |
+|---|---|---|
+| `OWUITOOLS_OCTAVE_TIMEOUT` | 25 s | one run |
+| `OWUITOOLS_OCTAVE_MAX_TIMEOUT` | 60 s | the most a caller may ask for |
+
+## Checking it
+
+```sh
+./tests/try-tool.sh matlab_run '{"code": "x = linspace(0,1,5);\ndisp(sum(x))"}'
+```
+
+Expect `MATLAB run OK`. Then try `for i=1:2\n printf(\"%d\",i);\nendfor` and expect
+`NOT VALID MATLAB`.
+
+## Plots
+
+A figure comes back as a `data:image/png` URI. Open WebUI hands those to the *model* as a
+vision input and never shows them to you — so the `inline_tool_images` filter exists to put
+the picture in the chat as well. See [../filters.md](../filters.md).

docs/setup/searxng.md

@@ -0,0 +1,45 @@
+# SearXNG — needed by `web_search` and `find_datasheet`
+
+**Nothing to set up.** SearXNG is a container in this project's own `docker-compose.yml`
+and starts with the rest of the stack. This page is for when it is not returning results.
+
+## What it is
+
+A metasearch front end. It queries several engines and merges the results; owuitools never
+talks to a search engine directly. It has no API key and no account.
+
+It lives **inside the guarded network namespace**, sharing it with the tool server, so it
+is not reachable from your LAN or from the internet, and it can only reach the outside
+world through squid. That is deliberate: it takes a query string from a language model, and
+anything that takes model-controlled input belongs behind the same filter the rest of the
+suite is behind.
+
+## Its configuration
+
+`configs/services/searxng.settings.yml`. The parts worth knowing about:
+
+- **`server.secret_key`** — generate your own with `openssl rand -hex 32`. It signs
+  SearXNG's own session data. The shipped file has a placeholder.
+- **`server.limiter: false`** — its bot-detection is off, because the only client is the
+  tool server on a private network, and the limiter rejects programmatic queries.
+- **The engine list.** Which engines are on is a *measurement*, not a preference: some
+  block datacentre IPs, some rate-limit hard, some serve CAPTCHAs. The comments in the file
+  say what was observed and when. **Those observations were made from one egress IP and may
+  not hold for yours** — if results are thin, this list is the first thing to change.
+
+After editing it: `docker compose restart searxng`.
+
+## Checking it
+
+```sh
+./tests/try-tool.sh web_search '{"query": "systemd socket activation"}' | head -20
+```
+
+## If it returns nothing
+
+1. `docker compose logs searxng --tail 50` — engine failures are logged individually, so
+   you can see which ones are refusing.
+2. Check squid is up: SearXNG's only route out is `owuitools-egress`.
+3. Try a different engine set. An engine that blocks your IP fails silently from the
+   model's point of view: the search simply comes back empty, which reads as "nothing was
+   written about this" rather than "the search was refused".

docs/setup/stackexchange.md

@@ -0,0 +1,29 @@
+# `OWUITOOLS_STACKEXCHANGE_KEY` — lifts the rate limit on `search_stackexchange`
+
+Two minutes. Free, instant, no approval. **Optional** — the tool works without it.
+
+## Why
+
+Keyless Stack Exchange allows **300 requests per day per IP**, shared by everyone using
+your assistant. A key raises it to **10,000 per day**. It is not a secret in the usual
+sense — it identifies an application, not a user, and grants no access to anything.
+
+## Steps
+
+1. Go to <https://stackapps.com/apps/oauth/register>.
+2. Fill in any name and description. For OAuth Domain, `localhost` is fine.
+3. Register. You are shown a **key** immediately.
+4. Copy the value labelled **Key** (not "Client Id", not "Client Secret").
+5. Put it in `configs/owuitools.env`:
+
+   ```
+   OWUITOOLS_STACKEXCHANGE_KEY=...
+   ```
+
+6. `docker compose up -d api`
+
+## Checking it
+
+```sh
+./tests/try-tool.sh search_stackexchange '{"query": "bash trap EXIT"}' | head -5
+```

docs/setup/vm-agent.md

@@ -0,0 +1,114 @@
+# `vm_agent` — the VM sub-agent
+
+This is the one tool with real setup. Budget an hour, most of it unattended.
+
+It is also removed automatically when the VM job volume is not mounted, so on a host
+without the VM stack the `vm_agent = true` line in `configs/tools.toml` does nothing and
+the assistant is never told the tool exists.
+
+This page is the checklist. [../vm-agent/overview.md](../vm-agent/overview.md) explains what
+it is and how the isolation works; [../vm-agent/base-images.md](../vm-agent/base-images.md)
+covers media and base images in detail.
+
+## What it does
+
+Hands a task to a second model that has a real, throwaway Ubuntu or Windows machine in
+front of it. That model writes the script, runs it, reads the actual error, fixes it, and
+comes back with evidence. The machine is destroyed afterwards; the project folder survives.
+
+## What the host needs
+
+| Requirement | Why | Check |
+|---|---|---|
+| KVM | the machine is a real VM | `ls -l /dev/kvm` |
+| A ZFS pool | the project quota and the overlay ceiling are ZFS quotas — they are what stops a runaway guest filling `/` and taking Docker, Ollama and Open WebUI down with it | `zfs list` |
+| ~50 GB free for base images | Ubuntu settles around 9 GB, Windows around 40 GB | `df -h /var/lib` |
+| `qemu-system-x86`, `qemu-utils`, `virtiofsd`, `ovmf`, `swtpm-tools`, `socat` | | `dpkg -l` |
+| A VNC viewer (`remmina`) | the maintenance console | |
+
+## Steps
+
+1. **Install the host side.**
+
+   ```sh
+   sudo ./vm/host/install.sh
+   ```
+
+   Creates three service users, the systemd units, the nftables guard, the polkit rule and
+   the directory layout. Idempotent — re-run it after changing anything in `vm/host/`.
+
+   It works out who the operator is from `sudo`, and that person is the only account polkit
+   will let unfreeze a base image. Override with `OWUIVM_OPERATOR` in `configs/host.env`.
+
+2. **Get a base image.** Two ways, and the first is much faster.
+
+   **Import a prebuilt one** — a coffee rather than an afternoon, and for Windows it avoids
+   clicking through an installer in a VNC console entirely:
+
+   ```sh
+   sudo ./vm/images/import.sh ubuntu      # ~9.6 GB
+   sudo ./vm/images/import.sh windows     # ~41 GB
+   ```
+
+   Checksum-verified, installed frozen and owned correctly, and resumable if the download
+   drops. See [../vm-agent/base-images.md](../vm-agent/base-images.md) for what is in them,
+   how to host your own, and the note on Windows licensing.
+
+   **Or build your own**, which is the right path if you want to change what is in the
+   image. Steps 3 and 4 below are that route; skip them if you imported.
+
+3. **Get the installation media.**
+
+   ```sh
+   sudo ./vm/images/fetch.sh ubuntu     # downloads and verifies against Canonical's checksums
+   sudo ./vm/images/fetch.sh virtio     # only needed for Windows
+   ```
+
+   Windows media you supply yourself. You can now also do this from the maintenance app —
+   **Insert disc…** on a machine that has no disk yet.
+
+4. **Build the Ubuntu base image.**
+
+   ```sh
+   sudo ./vm/images/ubuntu/build.sh     # ~25 minutes, unattended
+   ```
+
+   Windows has no unattended path: install it through the maintenance app's console. See
+   [../vm-agent/base-images.md](../vm-agent/base-images.md).
+
+5. **Set up Gitea**, if you want sessions to publish. See [gitea.md](gitea.md). Skippable.
+
+6. **Install the live-progress filter** so you can watch a session. See
+   [../filters.md](../filters.md) — `vm_agent_progress`, with its `api_key` valve set to
+   your `OWUITOOLS_AUTH_TOKEN`.
+
+7. **Raise the three timeouts.** A session runs for minutes to hours and dies at the
+   shortest clock. See [../advanced-params.md](../advanced-params.md).
+
+## Checking it
+
+```sh
+sudo ./tests/vm_host_asserts.sh     # the isolation suite
+
+# A session by hand. `ready` is not optional: every verb that touches the
+# machine is refused until the guest agent has answered, and this is what
+# waits for it.
+sudo -u owuivmd vmctl open --os ubuntu --project scratch
+sudo -u owuivmd vmctl ready
+sudo -u owuivmd vmctl plan "check the machine came up"
+sudo -u owuivmd vmctl run 'uname -a'
+sudo -u owuivmd vmctl close
+```
+
+Then, from a chat: *"use the VM to write a bash script that prints the largest ten files
+under /usr and run it"*.
+
+## Where the base images live, and what "frozen" means
+
+`/var/lib/owuivm/base/<id>.qcow2`, and frozen is **literally the owner write bit**. A
+session boots a throwaway overlay backed by the base; nothing it installs survives. vmd
+refuses to start a session at all while a base image is writable, so an image left
+unfrozen stops the agent rather than quietly letting it change the original.
+
+Freezing and unfreezing is one button in the **Tool Server VM Maintenance** app on the
+desktop.