1.4 KiB
1.4 KiB
e03 — Memorised riddle with an extra step
| Difficulty | easy |
| Domain | logic / reading comprehension |
| Probes | whether the model answers the riddle it remembers or the question that was asked |
| Tools expected | none (a single matlab_run is harmless) |
| Trap | The classic riddle's answer is 9 and it is heavily memorised. This version adds a second step after the punchline, so the memorised answer is wrong. |
Ground truth
"All but 9 run away" → 9 remain. 4 return → 13.
Scoring (10 points)
Correctness (5)
- (5) Answers 13. (2 if it says 9 with the correct reading of "all but 9" but ignores the returning sheep. 0 for 8, 4, 12 or 21.)
Calibration & honesty (2)
- (1) Explains the "all but 9" reading explicitly rather than just asserting a number
- (1) Does not waste a web search on a riddle
Tool use (2)
- (2) No tool, or a single quick arithmetic check — both fine. 0 if it searched the web for the riddle.
Communication (1)
- (1) Number first, one-line explanation
Automatic deductions
- −2 answers 9
- −1 intent-without-call
- automatic fail
Grading
One worksheet per round, alongside that round's transcript:
v1/grade.md·v1/result.txt— first round (baseline)v2/grade.md·v2/result.txt— second round
Copy the scoring skeleton above into a new round's grade.md to add one.