73 lines
3.8 KiB
Markdown
73 lines
3.8 KiB
Markdown
# When a VM session goes wrong
|
||
|
||
## First: read the session log, not the chat
|
||
|
||
Every session writes its own transcript, and it outlives the machine:
|
||
|
||
```sh
|
||
sudo vmctl logs # every session, newest first
|
||
sudo vmctl log # the most recent one, rendered
|
||
sudo vmctl log <chat-id> # the session for a conversation
|
||
sudo vmctl log <project> # by project name
|
||
```
|
||
|
||
It records every verb with a timestamp, what the model said each turn, and a census at the
|
||
end — *"what it actually did: 34× run, 12× screenshot… it never called click"*.
|
||
|
||
**Two records cannot answer "what did it do", and both look as though they can.** The tool
|
||
result used to count only `run`. And Open WebUI's status history is *sampled* every two
|
||
seconds, so fast verbs are usually overwritten before the filter sees them — measured, 456
|
||
samples caught `launch_app` nine times and not one of the screenshots that demonstrably
|
||
happened. Read the log.
|
||
|
||
## Common symptoms
|
||
|
||
| Symptom | Cause |
|
||
|---|---|
|
||
| The call fails after about five minutes | `AIOHTTP_CLIENT_TIMEOUT_TOOL_SERVER` is unset on Open WebUI. It defaults to **300**, not to unlimited. See [../advanced-params.md](../advanced-params.md). |
|
||
| No live view; the chat sits empty | The `vm_agent_progress` filter is not installed or not enabled, or `ENABLE_FORWARD_USER_INFO_HEADERS` is off. See [../filters.md](../filters.md). |
|
||
| "The machine has been destroyed" seconds in | A second `vm_agent` call in one conversation reading the first one's finished state. Fixed; if you see it again the api container is stale — `docker compose up -d --build api`. |
|
||
| Sessions refuse to start | A base image is unfrozen. Freeze it in the maintenance app. `sudo stat -c '%n %a' /var/lib/owuivm/base/*.qcow2` — 444 is frozen. |
|
||
| The agent says it has no internet | It used `ping`. `ping` can **never** work in the guest — `net.ipv4.ping_group_range` denies the unprivileged ICMP sockets the user-mode network stack needs — while HTTPS works fine. `check_internet` is the tool for this. |
|
||
| Git fails with exit 128, "dubious ownership" | The agent tried to run git inside the machine, where `.git` is owned by another uid over the shared filesystem. It should use `git_commit`, which runs on the host. `run` refuses mutating git now and says so. |
|
||
| Nothing was published | No Gitea token in `/etc/owuivm/vmd.env`. The session still ran and kept its files. See [../setup/gitea.md](../setup/gitea.md). |
|
||
| The published release contains test scripts | It should not — a scratch filter excludes them at commit time, and `git_commit` reports what it held back. If a real deliverable was held back, its name matched a scratch pattern; the list is in `vm/vmd/gitea.py`. |
|
||
|
||
## Checking the isolation still holds
|
||
|
||
```sh
|
||
sudo ./tests/vm_host_asserts.sh
|
||
```
|
||
|
||
This is the one to run after changing anything in `vm/host/`. It turns each layer off in
|
||
turn and re-runs the same probe against the others, so it fails if a control has quietly
|
||
started depending on another one.
|
||
|
||
## Driving a session by hand
|
||
|
||
Skip the model entirely when you want to know whether the machinery works:
|
||
|
||
```sh
|
||
sudo -u owuivmd vmctl open --os ubuntu --project scratch
|
||
sudo -u owuivmd vmctl ready # waits for the guest agent
|
||
sudo -u owuivmd vmctl plan "check the machine came up"
|
||
sudo -u owuivmd vmctl run 'uname -a'
|
||
sudo -u owuivmd vmctl shot /tmp/screen.png
|
||
sudo -u owuivmd vmctl close
|
||
```
|
||
|
||
If that works and sessions do not, the problem is in the agent loop, not the sandbox.
|
||
|
||
## Cleaning up after a crash
|
||
|
||
Orphans are reaped when the next session starts, but to do it now:
|
||
|
||
```sh
|
||
sudo systemctl stop owuivm.service
|
||
sudo ls -la /Pool/vmrun/ # overlays; should be empty between sessions
|
||
sudo ls -la /run/owuivm/
|
||
```
|
||
|
||
An overlay left behind is harmless — it is deleted when the next session starts — but it
|
||
does hold disk.
|