Initial Commit
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
# e03 — Memorised riddle with an extra step · round v2
|
||||
|
||||
> Grading worksheet for round **v2**. The ground truth, the trap and the
|
||||
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
|
||||
> master and is not edited per round. The transcript being graded is
|
||||
> [`result.txt`](result.txt).
|
||||
|
||||
## Scoring (10 points)
|
||||
|
||||
### Correctness (5)
|
||||
- [x] (5) Answers 13. (2 if it says 9 with the correct reading of "all but 9" but ignores the returning sheep. 0 for 8, 4, 12 or 21.) → 5/5 — **13**, stated first and in bold
|
||||
|
||||
### Calibration & honesty (2)
|
||||
- [x] (1) Explains the "all but 9" reading explicitly rather than just asserting a number → 1/1 — both in thinking and in the visible answer: "'All but 9 run away' → 9 stay, 8 leave"
|
||||
- [x] (1) Does not waste a web search on a riddle → 1/1 — zero tool calls; no tool is weighed in thinking
|
||||
|
||||
### Tool use (2)
|
||||
- [x] (2) No tool, or a single quick arithmetic check — both fine. 0 if it searched the web for the riddle. → 2/2 — no tools, which the rubric explicitly allows
|
||||
|
||||
### Communication (1)
|
||||
- [x] (1) Number first, one-line explanation → 1/1 — "**13 sheep.**" then three short bullets
|
||||
|
||||
### Automatic deductions
|
||||
- −2 answers 9 → not applied — answered 13; 9 appears only as the correctly-identified intermediate
|
||||
- −1 intent-without-call → not applied
|
||||
- automatic fail → not applied
|
||||
|
||||
## Harness observations (fill in after the run)
|
||||
|
||||
- Run date/time (UTC): 2026-09-11T19:15:14Z (dumped 19:15:44Z)
|
||||
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
|
||||
- Tool calls, in order (copy from the `result.txt` header): 0 (none)
|
||||
- Wasted, failed, or repeated calls: none
|
||||
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=5.4s / total=9.4s (decode=3.9s, 45.4 tok/s) / prompt_eval_count=12689, completion_tokens=177
|
||||
- Ended a turn with a statement of intent and no tool call? (y/n): n
|
||||
- Live status indicator matched what actually happened? (y/n): y — `Processing prompt → Thinking · first token after 5.6 s → Thought: … → Writing response → Done · prefill 5.4 s for 12,689 tokens · 177 generated`. The final line's numbers now agree with the `usage:` line to the token, because with no tool call there is only one generation round for the two counters to disagree about
|
||||
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no. This run is the cleanest available measurement of the **fixed per-chat prompt cost** after the trimming changes: 12,689 tokens with zero tool calls, against v1 e01's 13,541 under the same conditions — roughly 850 tokens, ~6%, removed by the system-prompt and tool-description rewrites. The quantitative nudge does not fire (bare integers, no units), so this is the model's unassisted behaviour. Thinking was 2 s and went straight down the correct path with no false start
|
||||
|
||||
## Result (fill in)
|
||||
|
||||
- **Score:** 10 / 10
|
||||
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
|
||||
- **Verdict:** pass
|
||||
- **What it got right:** did not answer the riddle it remembers. Parsed "all but 9" correctly (9 stay, 8 leave), carried the second step (4 of the 8 return), answered 13, number first, working shown, no tools wasted on a riddle
|
||||
- **What it got wrong:** nothing
|
||||
- **Change from the previous round:** **10 → 10 (0).** Identical behaviour and identical reasoning path; the v2 answer is marginally tighter. Nothing in Changes 45–53 touches this prompt's failure modes, and nothing regressed — in particular the decoding-penalty change (presence_penalty 0 → 1.0) did not disturb a short, structured, arithmetic answer, which is the main thing this prompt is now also passively testing
|
||||
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no — clean run. Doubles as a quiet regression check on the Wave-1 sampling change: no repetition, no language mixing, no padding
|
||||
|
||||
## Grader notes
|
||||
|
||||
- Ground truth verification: trivial and re-derived by hand — 17 total, "all but 9 run away" leaves 9 with the farmer and 8 gone, 4 of the 8 return, 9 + 4 = 13. The rubric's ground truth is correct. (The distractor answers the rubric lists map to the predictable misreadings: 9 = ignoring the second step, 8 = reading "all but 9" as "9 ran away", 12 = 8+4, 21 = 17+4)
|
||||
- Rubric issues: none
|
||||
@@ -0,0 +1,40 @@
|
||||
########################################################################
|
||||
# owuitools prompt test result
|
||||
# prompt dir : easy/e03-sheep-riddle-variant (round v2)
|
||||
# chat id : 47ebacfd-683c-49b4-809d-8f64c8d5f815
|
||||
# chat title : Sheep Count
|
||||
# user : the operator
|
||||
# model : qwen3.8:latest
|
||||
# turns : 2 messages on selected branch
|
||||
# tool calls : 0 (none)
|
||||
# dumped at : 2026-09-11T19:15:44Z
|
||||
########################################################################
|
||||
|
||||
=== USER (2026-09-11T19:15:14Z) ===
|
||||
A farmer has 17 sheep. All but 9 run away. Later, 4 of the ones that ran away come back on their own. How many sheep does the farmer have now?
|
||||
|
||||
=== ASSISTANT (2026-09-11T19:15:15Z, model qwen3.8:latest) ===
|
||||
usage: prompt_eval_count=12689 eval_count=177 prompt_tokens=12689 completion_tokens=177 total=9.4s prefill=5.4s decode=3.9s 45.4 tok/s
|
||||
|
||||
=== [1] THINKING (2s) ===
|
||||
Let me think about this carefully.
|
||||
|
||||
1. The farmer has 17 sheep.
|
||||
2. "All but 9 run away." This means all sheep except 9 run away. So 9 sheep remain, and 17 - 9 = 8 sheep ran away.
|
||||
3. "Later, 4 of the ones that ran away come back on their own." So 4 of the 8 that ran away come back.
|
||||
|
||||
So now the farmer has: 9 (that stayed) + 4 (that came back) = 13 sheep.
|
||||
|
||||
=== [2] ASSISTANT TEXT ===
|
||||
**13 sheep.**
|
||||
|
||||
- Start: 17 sheep.
|
||||
- "All but 9 run away" → 9 stay, 8 leave.
|
||||
- 4 of those 8 wander back → 9 + 4 = **13**.
|
||||
|
||||
=== STATUS HISTORY (live progress indicator) ===
|
||||
- Processing prompt
|
||||
- Thinking · first token after 5.6 s
|
||||
- Thought: Let me think about this carefully. 1. The farmer has 17 sheep. 2. "All but 9 run away." … farmer has: 9 (that stayed) + 4 (that came back) = 13 sheep.
|
||||
- Writing response
|
||||
- Done · prefill 5.4 s for 12,689 tokens · 177 generated
|
||||
Reference in New Issue
Block a user