Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,72 @@
+# When a VM session goes wrong
+
+## First: read the session log, not the chat
+
+Every session writes its own transcript, and it outlives the machine:
+
+```sh
+sudo vmctl logs                 # every session, newest first
+sudo vmctl log                  # the most recent one, rendered
+sudo vmctl log <chat-id>        # the session for a conversation
+sudo vmctl log <project>        # by project name
+```
+
+It records every verb with a timestamp, what the model said each turn, and a census at the
+end — *"what it actually did: 34× run, 12× screenshot… it never called click"*.
+
+**Two records cannot answer "what did it do", and both look as though they can.** The tool
+result used to count only `run`. And Open WebUI's status history is *sampled* every two
+seconds, so fast verbs are usually overwritten before the filter sees them — measured, 456
+samples caught `launch_app` nine times and not one of the screenshots that demonstrably
+happened. Read the log.
+
+## Common symptoms
+
+| Symptom | Cause |
+|---|---|
+| The call fails after about five minutes | `AIOHTTP_CLIENT_TIMEOUT_TOOL_SERVER` is unset on Open WebUI. It defaults to **300**, not to unlimited. See [../advanced-params.md](../advanced-params.md). |
+| No live view; the chat sits empty | The `vm_agent_progress` filter is not installed or not enabled, or `ENABLE_FORWARD_USER_INFO_HEADERS` is off. See [../filters.md](../filters.md). |
+| "The machine has been destroyed" seconds in | A second `vm_agent` call in one conversation reading the first one's finished state. Fixed; if you see it again the api container is stale — `docker compose up -d --build api`. |
+| Sessions refuse to start | A base image is unfrozen. Freeze it in the maintenance app. `sudo stat -c '%n %a' /var/lib/owuivm/base/*.qcow2` — 444 is frozen. |
+| The agent says it has no internet | It used `ping`. `ping` can **never** work in the guest — `net.ipv4.ping_group_range` denies the unprivileged ICMP sockets the user-mode network stack needs — while HTTPS works fine. `check_internet` is the tool for this. |
+| Git fails with exit 128, "dubious ownership" | The agent tried to run git inside the machine, where `.git` is owned by another uid over the shared filesystem. It should use `git_commit`, which runs on the host. `run` refuses mutating git now and says so. |
+| Nothing was published | No Gitea token in `/etc/owuivm/vmd.env`. The session still ran and kept its files. See [../setup/gitea.md](../setup/gitea.md). |
+| The published release contains test scripts | It should not — a scratch filter excludes them at commit time, and `git_commit` reports what it held back. If a real deliverable was held back, its name matched a scratch pattern; the list is in `vm/vmd/gitea.py`. |
+
+## Checking the isolation still holds
+
+```sh
+sudo ./tests/vm_host_asserts.sh
+```
+
+This is the one to run after changing anything in `vm/host/`. It turns each layer off in
+turn and re-runs the same probe against the others, so it fails if a control has quietly
+started depending on another one.
+
+## Driving a session by hand
+
+Skip the model entirely when you want to know whether the machinery works:
+
+```sh
+sudo -u owuivmd vmctl open --os ubuntu --project scratch
+sudo -u owuivmd vmctl ready                       # waits for the guest agent
+sudo -u owuivmd vmctl plan "check the machine came up"
+sudo -u owuivmd vmctl run  'uname -a'
+sudo -u owuivmd vmctl shot /tmp/screen.png
+sudo -u owuivmd vmctl close
+```
+
+If that works and sessions do not, the problem is in the agent loop, not the sandbox.
+
+## Cleaning up after a crash
+
+Orphans are reaped when the next session starts, but to do it now:
+
+```sh
+sudo systemctl stop owuivm.service
+sudo ls -la /Pool/vmrun/          # overlays; should be empty between sessions
+sudo ls -la /run/owuivm/
+```
+
+An overlay left behind is harmless — it is deleted when the next session starts — but it
+does hold disk.