Initial Commit

docs/testing/prompt-corpus/easy/e03-sheep-riddle-variant/v1/grade.md

@@ -0,0 +1,50 @@
+# e03 — Memorised riddle with an extra step · round v1
+
+> Grading worksheet for round **v1**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (5) Answers 13. (2 if it says 9 with the correct reading of "all but 9" but ignores the returning sheep. 0 for 8, 4, 12 or 21.)  → 5/5 — final answer text is "**13 sheep.**"; 9 stay, 4 return, 9 + 4 = 13
+
+### Calibration & honesty (2)
+- [x] (1) Explains the "all but 9" reading explicitly rather than just asserting a number  → 1/1 — first bullet is '"All but 9 run away" → 9 sheep stay, 8 leave' before the returners are added
+- [x] (1) Does not waste a web search on a riddle  → 1/1 — zero tool calls of any kind
+
+### Tool use (2)
+- [x] (2) No tool, or a single quick arithmetic check — both fine. 0 if it searched the web for the riddle.  → 2/2 — no tools; arithmetic done in a 1 s thinking block
+
+### Communication (1)
+- [x] (1) Number first, one-line explanation  → 1/1 — opens with "**13 sheep.**", then a two-bullet, ~20-word explanation (read "one-line" as "brief"; two short bullets qualify)
+
+### Automatic deductions
+- −2 answers 9 → not applied — answered 13; 9 appears only as the intermediate "9 sheep stay"
+- −1 intent-without-call → not applied — no statement of intent; the turn ends with the answer
+- automatic fail → not applied — no pasted data, no sources cited or invented
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T04:30:36Z (user turn; transcript dumped 04:30:58Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): none (header: `tool calls : 0  (none)`)
+- Wasted, failed, or repeated calls: none
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=5.0s / total=8.2s (decode=3.0s, 45.8 tok/s) / prompt_tokens=13544, completion_tokens=139
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — Processing prompt → Thinking (prompt processed in 5.3 s) → Thought snippet → Writing response → Done, with no tool count shown, which is consistent with 0 calls. The status's 5.3 s vs the usage line's prefill=5.0s differ by 0.3 s (wall-clock vs Ollama-reported); not material.
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): nothing. The quantitative-question nudge does not fire here (numbers carry no units; README uses e03 as its no-fire example). The thinking block has a "Wait, …" self-recheck that re-derives the same 13 — harmless. Fixed harness overhead is visible: 13,544 prompt tokens and 5.0 s of prefill for a 30-word user message (system prompt + tool schemas), i.e. ~61% of the 8.2 s wall time.
+
+## Result (fill in)
+
+- **Score:** 10 / 10
+- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
+- **Verdict:** pass (8–10) / partial (5–7) / fail (0–4) → **pass**
+- **What it got right:** Resisted the memorised "9": read "all but 9" as 9 stay / 8 leave, added the 4 returners, and answered 13 with the number first and the reasoning shown. No tools, no hedging, no invented sources.
+- **What it got wrong:** nothing material.
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no — clean run; nothing about the harness, system prompt or tool schemas was exposed. (Observation, not a finding: the ~13.5k-token / 5 s prefill is the harness's fixed per-chat cost and dominated this run's wall time.)
+
+## Grader notes
+- Ground truth verification: not needed beyond reasoning — 17 sheep; "all but 9 run away" means 9 stay and 17 − 9 = 8 leave; 4 of those 8 return (4 ≤ 8, consistent) → 9 + 4 = 13. Reproduced with python3 (`17 - (17-9) + 4` → 13) on 2026-09-11. The rubric's ground truth of 13 is correct; no web lookup needed for a riddle.
+- Rubric issues: none material. The communication line's "one-line explanation" is slightly ambiguous against a two-bullet answer; read it as "brief explanation after the number" and awarded the point.

docs/testing/prompt-corpus/easy/e03-sheep-riddle-variant/v1/result.txt

@@ -0,0 +1,39 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e03-sheep-riddle-variant
+# chat id    : 17991f97-1ca2-4113-b592-814eb458c5d2
+# chat title : Sheep Count
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 0  (none)
+# dumped at  : 2026-09-11T04:30:58Z
+########################################################################
+
+=== USER  (2026-09-11T04:30:36Z) ===
+A farmer has 17 sheep. All but 9 run away. Later, 4 of the ones that ran away come back on their own. How many sheep does the farmer have now?
+
+=== ASSISTANT  (2026-09-11T04:30:36Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=13544 eval_count=139 prompt_tokens=13544 completion_tokens=139 total=8.2s prefill=5.0s decode=3.0s 45.8 tok/s
+
+=== [1] THINKING (1s) ===
+Let me reason through this.
+
+17 sheep. All but 9 run away → 9 remain. Later 4 of the ones that ran away come back → 9 + 4 = 13.
+
+Wait, "all but 9 run away" — meaning 9 stay, 8 ran away. Then 4 come back: 9 + 4 = 13.
+
+Answer: 13.
+
+=== [2] ASSISTANT TEXT ===
+**13 sheep.**
+
+- "All but 9 run away" → 9 sheep stay, 8 leave.
+- 4 of the runaways come back → 9 + 4 = **13**.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · prompt processed in 5.3 s
+- Thought: Let me reason through this. 17 sheep. All but 9 run away → 9 remain. Later 4 of the ones that ran away come back → 9 + 4 = 13. Wait, "all but 9 run away" —…
+- Writing response
+- Done