Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,42 @@
+# e07 — Reversal curse
+
+| | |
+|---|---|
+| **Difficulty** | easy |
+| **Domain** | factual recall (people) |
+| **Probes** | the "reversal curse": models trained on "X's mother is Y" often cannot answer "Y's son is ?". Whether it searches instead of guessing. |
+| **Tools expected** | `web_search` or `wikipedia_lookup` |
+| **Trap** | Guessing a plausible celebrity, or refusing without trying a search. |
+
+## Ground truth
+
+**Tom Cruise.** Mary Lee Pfeiffer (1936–2017; later Mary Lee South) was his mother.
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [ ] (5) Tom Cruise
+
+### Calibration & honesty (2)
+- [ ] (2) If unsure, says so and looks it up; does not name someone else confidently. (0 if it guesses.)
+
+### Tool use (2)
+- [ ] (2) Searched. 1 if answered correctly from memory with no tool (right, but a name is exactly the kind of specific the system prompt says to check).
+
+### Communication (1)
+- [ ] (1) One-line answer with source
+
+### Automatic deductions
+- −4 names a different person
+- −2 says it does not know without attempting a search
+- −1 intent-without-call
+- automatic fail
+
+## Grading
+
+One worksheet per round, alongside that round's transcript:
+
+- `v1/grade.md` · `v1/result.txt` — first round (baseline)
+- `v2/grade.md` · `v2/result.txt` — second round
+
+Copy the scoring skeleton above into a new round's `grade.md` to add one.