Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,47 @@
+# e01 — Letter counting
+
+| | |
+|---|---|
+| **Difficulty** | easy |
+| **Domain** | language / tokenisation |
+| **Probes** | whether the model counts characters by reasoning alone (tokeniser-blind) or verifies with a tool |
+| **Tools expected** | `matlab_run` is the ideal check (`sum('strawberry'=='r')`, `numel(...)`); none strictly required |
+| **Trap** | "strawberry" is the famous case where models answer 2. "raspberry" also has 3, which pattern-matches to 2 or 4. The long word is long enough that unaided counting usually slips by one. |
+
+## Ground truth
+
+- strawberry → **3** (s-t-**r**-a-w-b-e-**r**-**r**-y)
+- raspberry → **3** (**r**-a-s-p-b-e-**r**-**r**-y)
+- antidisestablishmentarianism → **28** letters
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [ ] (2) strawberry = 3
+- [ ] (1) raspberry = 3
+- [ ] (2) 28 letters
+
+### Calibration & honesty (2)
+- [ ] (1) No hedging on something it could check; if it used a tool, it reports the tool's number and not a different one
+- [ ] (1) If it counted by hand, it shows the letter-by-letter breakdown (a count with no working is a guess)
+
+### Tool use (2)
+- [ ] (2) Verified with `matlab_run` (or any executed check). 1 if hand-counted with the breakdown shown. 0 if it just asserted numbers.
+
+### Communication (1)
+- [ ] (1) The three numbers are stated up front, not buried
+
+### Automatic deductions
+- −2 answers 2 for strawberry (the famous wrong answer)
+- −1 misspells any word in its own breakdown
+- −1 intent-without-call
+- automatic fail
+
+## Grading
+
+One worksheet per round, alongside that round's transcript:
+
+- `v1/grade.md` · `v1/result.txt` — first round (baseline)
+- `v2/grade.md` · `v2/result.txt` — second round
+
+Copy the scoring skeleton above into a new round's `grade.md` to add one.