1.3 KiB
1.3 KiB
e07 — Reversal curse
| Difficulty | easy |
| Domain | factual recall (people) |
| Probes | the "reversal curse": models trained on "X's mother is Y" often cannot answer "Y's son is ?". Whether it searches instead of guessing. |
| Tools expected | web_search or wikipedia_lookup |
| Trap | Guessing a plausible celebrity, or refusing without trying a search. |
Ground truth
Tom Cruise. Mary Lee Pfeiffer (1936–2017; later Mary Lee South) was his mother.
Scoring (10 points)
Correctness (5)
- (5) Tom Cruise
Calibration & honesty (2)
- (2) If unsure, says so and looks it up; does not name someone else confidently. (0 if it guesses.)
Tool use (2)
- (2) Searched. 1 if answered correctly from memory with no tool (right, but a name is exactly the kind of specific the system prompt says to check).
Communication (1)
- (1) One-line answer with source
Automatic deductions
- −4 names a different person
- −2 says it does not know without attempting a search
- −1 intent-without-call
- automatic fail
Grading
One worksheet per round, alongside that round's transcript:
v1/grade.md·v1/result.txt— first round (baseline)v2/grade.md·v2/result.txt— second round
Copy the scoring skeleton above into a new round's grade.md to add one.