# e03 — Memorised riddle with an extra step | | | |---|---| | **Difficulty** | easy | | **Domain** | logic / reading comprehension | | **Probes** | whether the model answers the riddle it remembers or the question that was asked | | **Tools expected** | none (a single `matlab_run` is harmless) | | **Trap** | The classic riddle's answer is 9 and it is heavily memorised. This version adds a second step after the punchline, so the memorised answer is wrong. | ## Ground truth "All but 9 run away" → 9 remain. 4 return → **13**. ## Scoring (10 points) ### Correctness (5) - [ ] (5) Answers 13. (2 if it says 9 with the correct reading of "all but 9" but ignores the returning sheep. 0 for 8, 4, 12 or 21.) ### Calibration & honesty (2) - [ ] (1) Explains the "all but 9" reading explicitly rather than just asserting a number - [ ] (1) Does not waste a web search on a riddle ### Tool use (2) - [ ] (2) No tool, or a single quick arithmetic check — both fine. 0 if it searched the web for the riddle. ### Communication (1) - [ ] (1) Number first, one-line explanation ### Automatic deductions - −2 answers 9 - −1 intent-without-call - automatic fail ## Grading One worksheet per round, alongside that round's transcript: - `v1/grade.md` · `v1/result.txt` — first round (baseline) - `v2/grade.md` · `v2/result.txt` — second round Copy the scoring skeleton above into a new round's `grade.md` to add one.