Initial Commit
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
# e03 — Memorised riddle with an extra step · round v4
|
||||
|
||||
> Grading worksheet for round **v4**. The ground truth, the trap and the
|
||||
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
|
||||
> master and is not edited per round. The transcript being graded is
|
||||
> [`result.txt`](result.txt).
|
||||
|
||||
## Scoring (10 points)
|
||||
|
||||
### Correctness (5)
|
||||
- [x] (5) Answers 13. (2 if it says 9 with the correct reading of "all but 9" but ignores the returning sheep. 0 for 8, 4, 12 or 21.) → 5/5 — "**13 sheep.**", first line, bold
|
||||
|
||||
### Calibration & honesty (2)
|
||||
- [x] (1) Explains the "all but 9" reading explicitly rather than just asserting a number → 1/1 — "'all but 9 run away' — that means **9 stay**, and the other **8** run away (17 − 9 = 8)" in the visible answer, and again in both thinking blocks
|
||||
- [x] (1) Does not waste a web search on a riddle → 1/1 — no search; the one call is a scratch arithmetic check
|
||||
|
||||
### Tool use (2)
|
||||
- [x] (2) No tool, or a single quick arithmetic check — both fine. 0 if it searched the web for the riddle. → 2/2 — one `matlab_run` (`purpose: "scratch"`, five lines with comments) confirming `ran away=8, now=13`. The rubric explicitly allows this. The model had already reached 13 in thinking before calling, and says why it called anyway: "the guidance says prefer tools over mental math even when easy. One quick matlab scratch call settles it"
|
||||
|
||||
### Communication (1)
|
||||
- [x] (1) Number first, one-line explanation → 1/1 — number first; explanation is three short lines rather than one. Same shape as v2 and v3
|
||||
|
||||
### Automatic deductions
|
||||
- −2 answers 9 → not applied — 13; 9 appears only as the correctly identified intermediate
|
||||
- −1 intent-without-call → not applied — "Let me just do a quick check" is followed by the call
|
||||
- automatic fail → not applied
|
||||
|
||||
## Harness observations (fill in after the run)
|
||||
|
||||
- Run date/time (UTC): 2026-09-12T04:06:02Z (dumped 04:06:33Z)
|
||||
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
|
||||
- Tool calls, in order (copy from the `result.txt` header): 1 — `matlab_run` x1 (scratch)
|
||||
- Wasted, failed, or repeated calls: none failed. The one call is redundant by the model's own account ("it's trivial though") but the rubric does not charge it
|
||||
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.3s / total=5.3s (decode=3.9s, 39.4 tok/s) / prompt_eval_count=9756, completion_tokens=154
|
||||
- Ended a turn with a statement of intent and no tool call? (y/n): n
|
||||
- Live status indicator matched what actually happened? (y/n): y — `Calling matlab_run`, `Done · 1 tool call · prefill 5.0 s for 12,773 tokens · 460 generated`
|
||||
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): the first three rounds made no tool call on this prompt; this one does, and the thinking shows why in full — the model recognises the riddle, gets 13, and then argues with itself for four sentences about whether the system prompt's arithmetic rule applies to `17 − 9 + 4` ("The instructions say use calculators for what they cover; this is basic arithmetic. I could just do it… Actually the guidance says prefer tools over mental math even when easy"). It is the "free accuracy, even when it looks easy" sentence in the system prompt doing exactly what it says, on a prompt where nothing needed checking. Harmless here; worth knowing that the rule pulls a call on trivial sums. No nudge (no units); no loop; the 12,773-token fixed cost is the Change 55 schema growth seen on every prompt
|
||||
|
||||
### Change 55 checks (this round's changes — fill in even when they did not fire)
|
||||
|
||||
- Did a tool result carry a `Sources:` trailer? (y/n): n — `matlab_run`
|
||||
- Did the answer end with a URL — written by the model, or appended by the `answer_sources` filter? (model / filter / none): none — none expected
|
||||
- Any of the 55.2 tool fixes exercised? Which, and did it fire correctly?: none — the Octave ran clean
|
||||
|
||||
## Result (fill in)
|
||||
|
||||
- **Score:** 10 / 10
|
||||
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
|
||||
- **Verdict:** pass
|
||||
- **What it got right:** read the question rather than the riddle; "all but 9" parsed correctly in both thinking and answer; the returning sheep carried; 13; no search
|
||||
- **What it got wrong:** nothing
|
||||
- **Change from the previous round:** **10 → 10 (0).** Fourth identical score. The one behavioural change is the scratch `matlab_run` call that v1–v3 did not make; the rubric allows it in so many words, and the transcript records the model choosing it because of the system prompt's arithmetic rule, not because it doubted the answer. As a no-search control the prompt still shows nothing odd under the sampling settings
|
||||
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): marginal — the arithmetic rule now fires on a three-term sum inside a riddle. One call at 0.44 s is not a cost worth tuning against, but if the round shows the same reflex on other reasoning prompts it is a sign the "even when it looks easy" sentence is read as "always"
|
||||
|
||||
## Grader notes
|
||||
|
||||
- Ground truth verification: 17 − 9 = 8 leave, 9 stay, 4 of the 8 return, 9 + 4 = 13. The Octave printed the same
|
||||
- Rubric issues: none
|
||||
@@ -0,0 +1,71 @@
|
||||
########################################################################
|
||||
# owuitools prompt test result
|
||||
# prompt dir : easy/e03-sheep-riddle-variant (round v4)
|
||||
# chat id : e740562e-cb74-4798-8917-996fa6f517ff
|
||||
# chat title : Sheep Count
|
||||
# user : the operator
|
||||
# model : qwen3.8:latest
|
||||
# turns : 2 messages on selected branch
|
||||
# tool calls : 1 matlab_run x1
|
||||
# dumped at : 2026-09-12T04:06:33Z
|
||||
########################################################################
|
||||
|
||||
=== USER (2026-09-12T04:06:02Z) ===
|
||||
A farmer has 17 sheep. All but 9 run away. Later, 4 of the ones that ran away come back on their own. How many sheep does the farmer have now?
|
||||
|
||||
=== ASSISTANT (2026-09-12T04:06:02Z, model qwen3.8:latest) ===
|
||||
usage: prompt_eval_count=9756 eval_count=154 prompt_tokens=9756 completion_tokens=154 total=5.3s prefill=1.3s decode=3.9s 39.4 tok/s
|
||||
|
||||
=== [1] THINKING ===
|
||||
Let me work through this carefully.
|
||||
|
||||
Start: 17 sheep.
|
||||
"All but 9 run away." — So 9 remain, and 17 − 9 = 8 ran away.
|
||||
"Later, 4 of the ones that ran away come back on their own." — 4 return.
|
||||
|
||||
Now: 9 (that stayed) + 4 (returned) = 13 sheep.
|
||||
|
||||
This is a reading-comprehension trap, not a computation, but let me verify the arithmetic with a tool as is my habit — it's trivial though. The instructions say use calculators for what they cover; this is basic arithmetic. I could just do it. It's simple enough, but let me be safe and quick. Actually the guidance says prefer tools over mental math even when easy. One quick matlab scratch call settles it.
|
||||
|
||||
Let me just do a quick check.
|
||||
|
||||
=== [2] TOOL CALL: matlab_run (call_hbbtl4co) ===
|
||||
{
|
||||
"purpose": "scratch",
|
||||
"code": "start = 17;\nstayed = 9; % \"all but 9 run away\" -> 9 remain\nran_away = start - stayed; % 8 ran away\nreturned = 4; % 4 come back\nnow = stayed + returned;\nfprintf('ran away=%d, now=%d\\n', ran_away, now);"
|
||||
}
|
||||
|
||||
=== [3] TOOL RESULT: matlab_run (call_hbbtl4co) ===
|
||||
# Scratch run OK (0.44s)
|
||||
## Output
|
||||
```
|
||||
ran away=8, now=13
|
||||
```
|
||||
|
||||
=== [4] THINKING (2s) ===
|
||||
The key trap: "All but 9 run away" means 9 stay (not that 9 are gone). Then 4 of the 8 who ran away return.
|
||||
|
||||
Start: 17
|
||||
Stayed: 9
|
||||
Ran away: 8
|
||||
Returned: 4
|
||||
Now: 9 + 4 = 13
|
||||
|
||||
=== [5] ASSISTANT TEXT ===
|
||||
**13 sheep.**
|
||||
|
||||
The trick is in "all but 9 run away" — that means **9 stay**, and the other **8** run away (17 − 9 = 8).
|
||||
|
||||
Then 4 of those 8 come back on their own.
|
||||
|
||||
So: 9 (that never left) + 4 (returned) = **13**.
|
||||
|
||||
=== STATUS HISTORY (live progress indicator) ===
|
||||
- Processing prompt
|
||||
- Thinking · first token after 5.2 s
|
||||
- Thought: Let me work through this carefully. Start: 17 sheep. "All but 9 run away." — So 9 … matlab scratch call settles it. Let me just do a quick check.
|
||||
- Calling matlab_run
|
||||
- Thinking
|
||||
- Thought: The key trap: "All but 9 run away" means 9 stay (not that 9 are gone). Then 4 of the 8 … Start: 17 Stayed: 9 Ran away: 8 Returned: 4 Now: 9 + 4 = 13
|
||||
- Writing response
|
||||
- Done · 1 tool call · prefill 5.0 s for 12,773 tokens · 460 generated
|
||||
Reference in New Issue
Block a user