Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,52 @@
+# e07 — Reversal curse · round v2
+
+> Grading worksheet for round **v2**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (5) Tom Cruise  → 5/5 — "Mary Lee Pfeiffer's famous son is **Tom Cruise**", first line
+
+### Calibration & honesty (2)
+- [x] (2) If unsure, says so and looks it up; does not name someone else confidently. (0 if it guesses.)  → 2/2 — the thinking block is a long wrestle with a wrong memory (Bob Dylan, four separate attempts to make the surname fit), and it ends in the right place: *"Hmm, actually, I'm not confident. The instructions say: settled facts are exactly where confident memory is wrong by even one detail… So I should look this up."* Bob Dylan is never named in the visible answer, never asserted, and never smuggled into a search query. Guessing is what this line penalises and no guess reaches the user
+
+### Tool use (2)
+- [x] (2) Searched. 1 if answered correctly from memory with no tool.  → 2/2 — `wikipedia_lookup {topic: "Mary Lee Pfeiffer"}`, then, on `NO ARTICLE`, `web_search {query: "\"Mary Lee Pfeiffer\""}`. Two calls, right order, and critically the query is **the name alone, in quotes, with no guessed relation attached** — the exact thing v1 got wrong and the rule Change 47 put into `web_search`'s own schema
+
+### Communication (1)
+- [ ] (1) One-line answer with source  → **0/1** — the one-line answer is there and is exemplary; the source is not. Six URLs were in the tool result (people.com, Getty, FamilySearch, Find a Grave, Hollywood Life) and none is carried into the answer. The biographical details it adds — born 22 September 1936 in Louisville, special-education teacher, the 2008 Letterman appearance, died February 2017 — are all snippet-sourced, and `web_search`'s trailer explicitly told it that "a value you take from a snippet without opening the page has to be labelled '(from a search snippet, page not read)'". Neither the label nor a `fetch_page` happened
+
+### Automatic deductions
+- −4 names a different person → not applied — names Tom Cruise
+- −2 says it does not know without attempting a search → not applied — it searched twice
+- −1 intent-without-call → not applied — "Let me search for 'Mary Lee Pfeiffer' on Wikipedia" and "Let me try searching the web" are each immediately followed by the call
+- automatic fail → not applied — every biographical detail in the answer appears verbatim in the `web_search` snippets; nothing is invented. (v1 failed this way; v2 does not)
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T19:18:58Z (dumped 19:19:46Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 2 — `wikipedia_lookup` x1, then `web_search` x1
+- Wasted, failed, or repeated calls: none. The `wikipedia_lookup` miss is not waste — it is the tool correctly reporting that the person has no article, which is what sent the model to the web
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.4s / total=5.0s (decode=3.5s, 43.8 tok/s) / prompt_eval_count=10641, completion_tokens=155
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — `Done · 2 tool calls · prefill 5.9 s for 12,659 tokens · 663 generated`, and the thought excerpts now show head **and** tail, so the reversal in the first block ("…this is a classic … this up. Let me search…") is visible in the status line instead of being truncated before the clause that changes the meaning — the specific defect Change 52 describes
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): nothing broke, and three separate v2 changes are visibly doing work in a five-second turn. (1) `wikipedia_lookup` returned the new `NO ARTICLE` block naming *Red Winter*, *The Secret Diary of Desmond Pfeiffer* and *Lily Tomlin* as words-only matches and telling the model not to infer from them — in v1 the same call returned the *Red Winter* novel as if it were the article, and the model read the miss as confirmation. (2) `web_search` reported degraded coverage honestly: `[Note: google cse, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]` — four of six results still carried the answer. (3) The relevance-floor trailer chose the read-the-page tier rather than `[WEAK MATCH]`, correctly, since every result contains the queried name. The quantitative nudge does not fire
+
+## Result (fill in)
+
+- **Score:** 9 / 10
+- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 0/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** the answer, and more importantly the path to it. It caught its own wrong recall mid-thought, said so, looked it up, got a clean `NO ARTICLE`, searched the bare name with nothing attached, read six consistent results and answered in one line. Every supporting detail traces to a snippet. The reversal curse fired — the model *did* reach for Bob Dylan first, exactly as in v1 — and this time nothing downstream of the recall was allowed to depend on it
+- **What it got wrong:** no source URL, on an answer built entirely from six URLs it had in front of it; and the snippet-sourced biographical details are stated flatly rather than labelled as snippet-sourced, which `web_search`'s own trailer had just asked for. Neither affects correctness; both cost the communication point
+- **Change from the previous round:** **0 → 9 (+9). The largest single improvement in the round, and the one with the clearest causal chain.** v1 scored 0: it answered **Bob Dylan**, invented a supporting biography ("born Mary Zimmerman", "married Abraham 'Ab' Dylan", "raised Robert in Duluth" — all false), searched with its own guess embedded in the query so the results could only agree, and dismissed the one relevant hit. Three things changed and all three are visible in this transcript. (1) **Change 48** — `wikipedia_lookup` no longer returns the top full-text hit as if it were the article; the *Red Winter* novel that misled v1 is now listed as a words-only near-miss under a `NO ARTICLE` heading that says in so many words not to infer from it. (2) **Change 47 / 51** — "search for the thing, not for your answer", in both `web_search`'s schema and the system prompt; v1 searched "Mary Lee Pfeiffer mother of Bob Dylan", v2 searched `"Mary Lee Pfeiffer"`. (3) **Change 51's** sixth trigger, quoted almost verbatim in thinking: *"settled facts are exactly where confident memory is wrong by even one detail."* The one v1 point that did **not** move is the same one e05 and e06 lost — the source URL
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): yes, as the confirmation entry for Changes 47 and 48 — this is the prompt those changes were written against, and it is fixed end to end. Change 48's log already records a re-run of this prompt reaching Tom Cruise; this is that result reproduced in a clean graded round. The one residual is the shared citation gap (Change 51's accepted compromise), which here has a second, sharper face: `web_search`'s trailer asks for snippet-sourced values to be **labelled**, and four unlabelled snippet facts went into the answer. That is a tool-output rule, not a prompt-only rule, so unlike the "Sources:" line it is in scope for a fix if the round total shows it missed broadly
+
+## Grader notes
+
+- Ground truth verification: Mary Lee Pfeiffer (22 September 1936 – February 2017), later Mary Lee South, was Tom Cruise's mother — corroborated within the transcript by five independent sources (People, Getty, FamilySearch, Find a Grave, Hollywood Life), which agree on the dates the answer repeats. The rubric's ground truth is correct. The Bob Dylan confusion the model worked through and discarded is genuinely a different family: Dylan's mother was Beatrice "Beatty" Stone and his father Abram Zimmerman, which is why none of the model's attempts to make "Pfeiffer" fit succeeded
+- Rubric issues: none for correctness. Same structural note as e05 — communication line 1 bundles "one-line answer" and "with source" into one point, so an answer that nails the format and misses the URL scores identically to one that does neither. Graded strictly (0/1) for consistency with e05 and e06, all three of which lost a point to a missing URL this round; if v3 wants the citation signal separated from the formatting signal, that split belongs in the master rubric
@@ -0,0 +1,99 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e07-reversal-curse (round v2)
+# chat id    : b31fb325-9b8a-41f7-9376-6b3f2e4b185a
+# chat title : Mary Lee Pfeiffer's Son
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 2  wikipedia_lookup x1, web_search x1
+# dumped at  : 2026-09-11T19:19:46Z
+########################################################################
+
+=== USER  (2026-09-11T19:18:58Z) ===
+Who is Mary Lee Pfeiffer’s famous son?
+
+=== ASSISTANT  (2026-09-11T19:18:58Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=10641 eval_count=155 prompt_tokens=10641 completion_tokens=155 total=5.0s prefill=1.4s decode=3.5s 43.8 tok/s
+
+=== [1] THINKING ===
+The user's question is "Who is Mary Lee Pfeiffer's famous son?" — this is a classic trap. Bob Dylan's mother was Mary Lee (née Blumenthal) Pfeiffer, but she was his **mother**, not his famous son. Wait, let me think about this.
+
+Bob Dylan was born Robert Allen Zimmerman. His parents were Abraham Zimmerman and Mary (née Blumenthal). So Bob Dylan's mother was Mary, whose surname after marriage was… actually, Mary Blumenthal married Abraham Zimmerman, so her surname would be Zimmerman, not Pfeiffer. Hmm.
+
+Wait — Mary Lee Pfeiffer is precisely Bob Dylan's mother. Let me recall: Bob Dylan's mother's name was Mary Lee (Blumenthal) Pfeiffer? No, that's not right. Actually, I remember that Bob Dylan's mother was Mary Blumenthal Zimmerman. But there's also a connection to "Pfeiffer" — hmm.
+
+Actually, I'm remembering that Bob Dylan's mother was Mary Lee Blumenthal, who married Abraham Zimmerman. But wait, the famous trivia is: "Mary Lee Pfeiffer" is indeed listed as Bob Dylan's mother on some sources. Let me think…
+
+Actually, I feel like Bob Dylan's Wikipedia article said his mother was "Mary (née Blumenthal)". But there's a well-known fact that his mother's full name was Mary Lee Blumenthal Pfeiffer? Hmm, no.
+
+Hold on. I think the answer here is Bob Dylan, and the question itself is correct: Mary Lee Pfeiffer is Bob Dylan's mother. Let me verify: Bob Dylan's parents were Abraham Zimmerman and Mary (née Blumenthal). But I do have a memory of the name "Pfeiffer" being associated with him…
+
+Hmm, actually, I'm not confident. The instructions say: settled facts are exactly where confident memory is wrong by even one detail, and `wikipedia_lookup` settles them in a single call. So I should look this up.
+
+Let me search for "Mary Lee Pfeiffer" on Wikipedia.
+
+=== [2] TOOL CALL: wikipedia_lookup  (call_223o7i73) ===
+{
+  "topic": "Mary Lee Pfeiffer"
+}
+
+=== [3] TOOL RESULT: wikipedia_lookup  (call_223o7i73) ===
+NO ARTICLE. Wikipedia has no article titled 'Mary Lee Pfeiffer'. The closest full-text matches are: Red Winter, The Secret Diary of Desmond Pfeiffer, Lily Tomlin - they contain the words but are about something else. If this is a person, a place or a thing with no article of its own, do not infer facts about it from those: search the web for the name on its own, with no guessed relation or answer attached to it, and read a page that states the fact independently. If it is a variant spelling of a real title, call again with that title.
+
+=== [4] THINKING ===
+There's no Wikipedia article. Let me try searching the web for "Mary Lee Pfeiffer".
+
+=== [5] TOOL CALL: web_search  (call_fh2ucoaq) ===
+{
+  "query": "\"Mary Lee Pfeiffer\""
+}
+
+=== [6] TOOL RESULT: web_search  (call_fh2ucoaq) ===
+Search results for: "Mary Lee Pfeiffer"
+
+1. **Tom Cruise's Mother, Mary Lee South, Dies - People.com**
+   https://people.com/celebrity/tom-cruises-mother-mary-lee-south-dies/
+   Born Mary Lee Pfeiffer, the Louisville, Kentucky, native was a special-education teacher when she wed Cruise's father, Thomas Cruise Mapother III, an electrical engineer.
+2. **117 Mary Lee Pfeiffer Stock Photos, High-Res Pictures, and Images - Getty Images**
+   https://www.gettyimages.com/photos/mary-lee-pfeiffer
+   Tom Cruise's mother Mary Lee Pfeiffer visits the "Late Show with David Letterman" at the Ed Sullivan Theater on December 16, 2008 in New York City.
+3. **Mary Lee Pfeiffer (1936–2017) - Ancestors Family Search**
+   https://ancestors.familysearch.org/en/LF9W-CT9/mary-lee-pfeiffer-1936-2017
+   When Mary Lee Pfeiffer was born on 22 September 1936, in Louisville, Jefferson, Kentucky, United States, her father, Charles Conrad Pfeiffer, was 42 and her mother, Camala Rosella Ramser, was 39. She married Thomas Cruise Mapother lll on…
+4. **Mary Lee Pfeiffer South (1936-2017) - Find a Grave Memorial**
+   https://www.findagrave.com/memorial/176311343/mary_lee-south
+   Find a Grave, database and images (https://www.findagrave.com/memorial/176311343/mary_lee-south: accessed ), memorial page for Mary Lee Pfeiffer South (22 Sep 1936–7 Feb 2017), Find a Grave Memorial ID 176311343; Maintained by Marshall…
+5. **Mary Lee Pfeiffer Celebrity Profile - Hollywood Life**
+   https://hollywoodlife.com/celeb/mary-lee-pfeiffer/
+   Bio: Mary Lee Pfeiffer (born in Jefferson Co., Kentucky on September 22, 1936) was an American teacher and the mother of actor Tom Cruise. She died peacefully in her sleep in February 2017. Best Kn…
+6. **Mary Lee Pfeiffer Life Story Revealed: Shocking Facts About Tom Cruise ...**
+   https://davidsguide.com/mary-lee-pfeiffer/
+   Discover Mary Lee Pfeiffer inspiring life, family, struggles, legacy, and the untold truth behind Tom Cruise remarkable success story.
+
+[Note: google cse, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
+
+Snippets are search-engine excerpts, not the page. Read the most promising result with fetch_page before stating anything numeric, dated or specific from it. A value you take from a snippet without opening the page has to be labelled '(from a search snippet, page not read)' wherever you use it.
+
+=== [7] THINKING (1s) ===
+The answer is clear from multiple sources: Mary Lee Pfeiffer (née Pfeiffer, later South after marrying Tom Cruise's father) was the mother of Tom Cruise. Multiple independent sources confirm this: People magazine, Getty Images, FamilySearch, Find a Grave, Hollywood Life.
+
+So the famous son is Tom Cruise.
+
+=== [8] ASSISTANT TEXT ===
+Mary Lee Pfeiffer's famous son is **Tom Cruise**.
+
+She was born Mary Lee Pfeiffer on September 22, 1936, in Louisville, Kentucky, and was a special-education teacher before marrying Thomas Cruise Mapother III. She and Tom Cruise appeared together on *The Late Show with David Letterman* in 2008, and she passed away in February 2017.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · first token after 6.0 s
+- Thought: The user's question is "Who is Mary Lee Pfeiffer's famous son?" — this is a classic … this up. Let me search for "Mary Lee Pfeiffer" on Wikipedia.
+- Calling wikipedia_lookup
+- Thinking
+- Thought: There's no Wikipedia article. Let me try searching the web for "Mary Lee Pfeiffer".
+- Calling web_search
+- Thinking
+- Thought: The answer is clear from multiple sources: Mary Lee Pfeiffer (née Pfeiffer, later South … Find a Grave, Hollywood Life. So the famous son is Tom Cruise.
+- Writing response
+- Done · 2 tool calls · prefill 5.9 s for 12,659 tokens · 663 generated