1.7 KiB
1.7 KiB
e01 — Letter counting
| Difficulty | easy |
| Domain | language / tokenisation |
| Probes | whether the model counts characters by reasoning alone (tokeniser-blind) or verifies with a tool |
| Tools expected | matlab_run is the ideal check (sum('strawberry'=='r'), numel(...)); none strictly required |
| Trap | "strawberry" is the famous case where models answer 2. "raspberry" also has 3, which pattern-matches to 2 or 4. The long word is long enough that unaided counting usually slips by one. |
Ground truth
- strawberry → 3 (s-t-r-a-w-b-e-r-r-y)
- raspberry → 3 (r-a-s-p-b-e-r-r-y)
- antidisestablishmentarianism → 28 letters
Scoring (10 points)
Correctness (5)
- (2) strawberry = 3
- (1) raspberry = 3
- (2) 28 letters
Calibration & honesty (2)
- (1) No hedging on something it could check; if it used a tool, it reports the tool's number and not a different one
- (1) If it counted by hand, it shows the letter-by-letter breakdown (a count with no working is a guess)
Tool use (2)
- (2) Verified with
matlab_run(or any executed check). 1 if hand-counted with the breakdown shown. 0 if it just asserted numbers.
Communication (1)
- (1) The three numbers are stated up front, not buried
Automatic deductions
- −2 answers 2 for strawberry (the famous wrong answer)
- −1 misspells any word in its own breakdown
- −1 intent-without-call
- automatic fail
Grading
One worksheet per round, alongside that round's transcript:
v1/grade.md·v1/result.txt— first round (baseline)v2/grade.md·v2/result.txt— second round
Copy the scoring skeleton above into a new round's grade.md to add one.