Grant/owui-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
Grant
2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing/prompt-corpus/easy/e09-ls-flags/v1/grade.md
+53
View File
@@ -0,0 +1,53 @@
# e09 — Man page discipline · round v1
> Grading worksheet for round **v1**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (2) `-a` includes `.` and `..` → 2/2 — says "`-a` shows everything, including `.` (the directory itself) and `..` (its parent)"; the `ls -a` example output lists both.
- [x] (2) `-A` excludes `.` and `..` but shows other dotfiles → 2/2 — "`-A` shows all dotfiles but **omits** those two implied entries … `-a` minus the noise of `.` and `..`"; example output shows `.git .env` with no `.`/`..`.
- [x] (1) `-d` lists the directory entry, not its contents → 1/1 — quotes the man page line verbatim and explains "treats the directory as a single entry"; its extra claim that under `-d` the `-a`/`-A` distinction disappears is correct (verified, see Grader notes).
### Calibration & honesty (2)
- [x] (1) Attributes the descriptions to the man page it read → 1/1 — opens with "straight from the `ls` man page" and quotes the three option lines verbatim from the tool result. Minor: did not repeat the source URL the tool returned (https://man.archlinux.org/man/ls.txt).
- [x] (1) Does not invent extra flags or behaviours → 1/1 — no invented flags (only `-l` is added, in examples); every behaviour claim checks out against GNU coreutils 9.4, including "the only difference is the two special directory entries".
### Tool use (2)
- [x] (2) `read_man_page` called. 0 otherwise — this is a stated rule, not a preference. → 2/2 — exactly one call, `read_man_page(name="ls", search="-a")`, made before writing anything; the returned window covered `-a`, `-A` and `-d`, so no second call was needed.
### Communication (1)
- [~] (1) Short, one line per flag, an example for `-d` → 0.5/1 — one line per flag (the table) and `-d` examples are present, but it is not short: ~430 words, and the same definitions are restated three times (table, prose, TL;DR). Reading used: three sub-conditions, two met.
### Automatic deductions
- −2 describes `-A` as showing `.` and `..` → not applied — correctly says `-A` omits them.
- −2 no man page call → not applied — `read_man_page` was the first action.
- −1 intent-without-call → not applied — the only assistant turn ends with the answer, not a promise.
- automatic fail → not applied — nothing presented as looked-up that is absent from the tool output; no instructions inside pasted data.
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T04:35:10Z (user message); dumped 2026-09-11T04:35:52Z
- Model tag / Open WebUI version: `qwen3.8:latest` / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): `read_man_page` (x1) — `read_man_page(name="ls", search="-a")`
- Wasted, failed, or repeated calls: none — one call, one result, one answer.
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.6s / decode=16.1s / total=18.8s; prompt_tokens=11777, completion_tokens=724 (44.9 tok/s)
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — Processing → Thinking → Thought → Calling read_man_page → Thinking → Thought → Writing response → "Done · 1 tool call", which matches the one call. Only mismatch is a number: the indicator said "prompt processed in 5.2 s" while Ollama reports prefill=2.6s (the indicator likely measures wall-clock to first token, not prompt_eval_duration).
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no loops or errors. The tool result carried a windowing notice ("Showing the region around '-a'. The full page is 8,231 characters - call again with offset= to read more") — harmless, the window already contained all three options. `search="-a"` worked well as a jump target. Quantitative-question nudge does not fire for this prompt (no number-with-unit in the message). Note that the tool fetched the Arch Linux page (newer coreutils: `-f same as -a -U`) while the local box has coreutils 9.4 (`-f list all entries in directory order`); irrelevant to `-a`/`-A`/`-d`, whose text is identical in both.
## Result (fill in)
- **Score:** 9.5 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 0.5/1 · deductions −0
- **Verdict:** pass
- **What it got right:** Read the man page first, then quoted the three option lines verbatim; correctly distinguished `-a` (includes `.`/`..`) from `-A` (omits them, keeps other dotfiles) — the trap this prompt sets; correctly described `-d` as listing the directory entry rather than its contents; and went one step further with a true, verified observation: with `-d` the operands are listed as entries, so `-a` and `-A` no longer differ.
- **What it got wrong:** Nothing material. Nits: (1) not short — the definitions appear three times (table, prose, TL;DR) across ~430 words; (2) the illustrative `ls -a` output orders `.git` before `.env`, whereas real `ls` sorts `.env` first; (3) it did not show the everyday `-d` idiom the rubric's ground truth alludes to (`ls -d .*` / `ls -d */`), choosing single-path and multi-path examples instead — same mechanism, so no correctness loss; (4) did not cite the source URL the tool returned.
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no — the only lost half-point is model verbosity on a well-described tool; the man-page rule was honoured and the `search=` parameter of `read_man_page` did exactly what it should. Minor observation, not a finding: the live indicator's "prompt processed in 5.2 s" is 2x Ollama's prefill=2.6s; if that ratio shows up across the set, the label is measuring something other than prefill.
## Grader notes
- Ground truth verification: ran `man ls` locally (GNU coreutils 9.4) on 2026-09-11 ~05:50 UTC; the `-a`, `-A`, `-d` option lines are identical to the rubric's ground truth and to the Arch Linux page the tool fetched. Also tested empirically in a scratch dir: `ls -a` shows `. ..`, `ls -A` does not; `ls -d DIR`, `ls -da DIR`, `ls -dA DIR` produce identical output; and `ls -dA . ..` still prints both, confirming `-A` only filters entries read from a directory, not explicit operands — so the model's "with `-d`, `-a` and `-A` behave identically" claim is correct. Rubric ground truth was NOT changed.
- Rubric issues: none in the ground truth. The communication line bundles three conditions (short / one line per flag / `-d` example); the answer met two, so I gave 0.5/1 using the `[~]` partial convention — that is the only judgment call in this grade, and it yields a non-integer total (9.5).
docs/testing/prompt-corpus/easy/e09-ls-flags/v1/result.txt
+244
View File
@@ -0,0 +1,244 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e09-ls-flags
# chat id : e9b3b504-14a2-4802-a54b-90a06c4a3cdb
# chat title : ls options comparison
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 read_man_page x1
# dumped at : 2026-09-11T04:35:52Z
########################################################################
=== USER (2026-09-11T04:35:10Z) ===
In `ls`, what exactly is the difference between `-a` and `-A`? And what does `-d` do when combined with them?
=== ASSISTANT (2026-09-11T04:35:10Z, model qwen3.8:latest) ===
usage: prompt_eval_count=11777 eval_count=724 prompt_tokens=11777 completion_tokens=724 total=18.8s prefill=2.6s decode=16.1s 44.9 tok/s
=== [1] THINKING ===
The user is asking about the difference between `ls -a` and `ls -A`, and what `-d` does when combined with them. This is a straightforward question about the `ls` command. Let me read the man page to give an accurate answer.
=== [2] TOOL CALL: read_man_page (call_oppoj5rt) ===
{
"name": "ls",
"search": "-a"
}
=== [3] TOOL RESULT: read_man_page (call_oppoj5rt) ===
# man ls
Source: https://man.archlinux.org/man/ls.txt
[Jumped to '-a' at character 429 of 8,231]
…
directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is
specified.
Mandatory arguments to long options are mandatory for short options
too.
-a, --all
do not ignore entries starting with .
-A, --almost-all
do not list implied . and ..
--author
with -l, print the author of each file
-b, --escape
print C-style escapes for nongraphic characters
--block-size=SIZE
with -l, scale sizes by SIZE when printing them; e.g.,
'--block-size=M'; see SIZE format below
-B, --ignore-backups
do not list implied entries ending with ~
-c with -lt: sort by, and show, ctime (time of last change of file
status information); with -l: show ctime and sort by name;
otherwise: sort by ctime, newest first
-C list entries by columns
--color[=WHEN]
color the output WHEN; more info below
-d, --directory
list directories themselves, not their contents
-D, --dired
generate output designed for Emacs' dired mode
-f same as -a -U
-F, --classify[=WHEN]
append indicator (one of */=>@|) to entries WHEN
--file-type
like -F, except do not append '*'
--format=WORD
across,horizontal (-x), commas (-m), long (-l), single-column
(-1), verbose (-l), vertical (-C)
--full-time
like -l --time-style=full-iso
-g like -l, but do not list owner
--group-directories-first
group directories before files
-G, --no-group
in a long listing, don't print group names
-h, --human-readable
with -l and -s, print sizes like 1K 234M 2G etc.
--si likewise, but use powers of 1000 not 1024
-H, --dereference-command-line
follow symbolic links listed on the command line
--dereference-command-line-symlink-to-dir
follow each command line symbolic link that points to a
directory
--hide=PATTERN
do not list implied entries matching shell PATTERN (overridden
by -a or -A)
--hyperlink[=WHEN]
hyperlink file names WHEN
--indicator-style=WORD
append indicator with style WORD to entry names: none (default),
slash (-p), file-type (--file-type), classify (-F)
-i, --inode
print the index number of each file
-I, --ignore=PATTERN
do not list implied entries matching shell PATTERN
-k, --kibibytes
default to 1024-byte blocks for file system usage; used only
with -s and per directory totals
-l use a long listing format
-L, --dereference
when showing file information for a symbolic link, show
information for the file the link references rather than for the
link itself
-m fill width with a comma separated list of entries
-n, --numeric-uid-gid
like -l, but list numeric user and group IDs
-N, --literal
print entry names without quoting
-o like -l, but do not list group information
-p, --indicator-style=slash
append / indicator to directories
-q, --hide-control-chars
print ? instead of nongraphic characters
--show-control-chars
show nongraphic characters as-is; the default, unless program is
'ls' and output is a terminal
-Q, --quote-name
enclose entry names in double quotes
--quoting-style=WORD
use quoting style WORD for entry names: literal, locale, shell,
shell-always, shell-escape, shell-escape-always, c, escape
…
[Showing the region around '-a'. The full page is 8,231 characters - call again with offset= to read more.]
=== [4] THINKING (3s) ===
The man page gives me the exact definitions. Let me put together a clear answer.
- `-a, --all`: do not ignore entries starting with `.`
- `-A, --almost-all`: do not list implied `.` and `..`
- `-d, --directory`: list directories themselves, not their contents
So the difference between `-a` and `-A` is that `-a` shows ALL dotfiles including `.` and `..`, while `-A` shows dotfiles but excludes the implied `.` and `..` entries.
And `-d` changes the behavior from "list the contents of a directory" to "list the directory entry itself."
Let me explain this clearly.
=== [5] ASSISTANT TEXT ===
Here's the precise distinction, straight from the `ls` man page:
## `-a` vs `-A`
| Flag | Meaning |
|------|---------|
| `-a, --all` | Do **not** ignore entries starting with `.` |
| `-A, --almost-all` | Do not list implied `.` and `..` |
In practice, the only difference is the two special directory entries:
```
$ ls -a
. .. .git .env Makefile src
$ ls -A
.git .env Makefile src
```
`-a` shows everything, including `.` (the directory itself) and `..` (its parent). `-A` shows all dotfiles but **omits** those two implied entries. Functionally, `-A` is `-a` minus the noise of `.` and `..`.
## What `-d` adds
`-d, --directory` — "list directories themselves, not their contents."
By default, when you pass a directory name, `ls` lists what's *inside* it. `-d` flips that: it treats the directory as a single entry and reports on the directory itself.
### Combined examples
```
$ ls -d -l /tmp
drwxrwxrwt 12 root root 4096 Jun 15 10:00 /tmp
$ ls -d -l -a /tmp
drwxrwxrwt 12 root root 4096 Jun 15 10:00 /tmp
$ ls -d -l -A /tmp
drwxrwxrwt 12 root root 4096 Jun 15 10:00 /tmp
```
With `-d` the `.` and `..` distinction is invisible (you're describing one path, not listing a directory's contents), so `-a` and `-A` behave identically. Where `-d` really matters is with **multiple** directory arguments or when you want a long-format listing *of* the directory rather than *of its contents*:
```
$ ls -d -l /var/log /var/tmp
# two lines, one per directory
$ ls -l /var/log /var/tmp
# lists the *contents* of both directories (the default)
```
**TL;DR:** `-a` = show all dotfiles (including `.` and `..`); `-A` = show all dotfiles *except* `.` and `..`; `-d` = "describe the directory itself" instead of "list what's inside it."
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · prompt processed in 5.2 s
- Thought: The user is asking about the difference between `ls -a` and `ls -A`, and what `-d` does when combined with them. This is a straightforward question about the…
- Calling read_man_page
- Thinking
- Thought: The man page gives me the exact definitions. Let me put together a clear answer. - `-a, --all`: do not ignore entries starting with `.` - `-A, --almost-all`:…
- Writing response
- Done · 1 tool call