Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,60 @@
+# e07 — Reversal curse · round v3
+
+> Grading worksheet for round **v3**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (5) Tom Cruise  → 5/5 — "Mary Lee Pfeiffer's famous son is **Tom Cruise**", first sentence
+
+### Calibration & honesty (2)
+- [x] (2) If unsure, says so and looks it up; does not name someone else confidently. (0 if it guesses.)  → 2/2 — the reversal curse fires again in thinking block [1] ("Mary Lee Pfeiffer, who is Bob Dylan's mother") and, as in v2, nothing is allowed to rest on it: "rather than answering from memory, I should verify". The wrong name never reaches the answer, never enters a query, and is not mentioned again after the first tool result
+
+### Tool use (2)
+- [x] (2) Searched. 1 if answered correctly from memory with no tool.  → 2/2 — `wikipedia_lookup` → `NO ARTICLE` → `web_search` with the bare name and nothing attached. Two calls, both necessary, in the right order
+
+### Communication (1)
+- [ ] (1) One-line answer with source  → **0/1** — the answer is direct but carries no URL, with five in the tool output, one of which (People) states the fact in its title. Third round losing this point
+
+### Automatic deductions
+- −4 names a different person → not applied — Bob Dylan appears only in thinking, as a recalled belief that was then checked and discarded
+- −2 says it does not know without attempting a search → not applied
+- −1 intent-without-call → not applied
+- automatic fail → not applied — every biographical detail in the answer is in a snippet
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T23:02:34Z (dumped 23:10:24Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 2 — `wikipedia_lookup` {topic: "Mary Lee Pfeiffer"} → `NO ARTICLE`; `web_search` {query: "Mary Lee Pfeiffer", max_results: 5}
+- Wasted, failed, or repeated calls: none. The `NO ARTICLE` result is a correct negative, not a failure, and it did its job: it names *Red Winter* (the novel that misled v1) as a words-only match and the model took nothing from it
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.3s / total=4.1s (decode=2.7s, 41.7 tok/s) / prompt_eval_count=10162, completion_tokens=113
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — `Done · 2 tool calls · prefill 5.9 s for 12,659 tokens · 236 generated`
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no faults. Nudge does not fire. One factual slip in the supporting detail that the rubric does not score: "after marrying Thomas Cruise Mapother III… she took the surname South" — the snippets say she was born Pfeiffer, married Mapother, and died as Mary Lee South; the South surname came from a later marriage, not from Mapother. The model joined two snippet facts with a wrong connective. Nothing in the tool output says it; nothing in the tool output contradicts it either. Recorded because it is the same shape as the v2 finding about unlabelled snippet facts — snippet-sourced details stated flatly, this time with an inference glued in
+
+### Change 54 checks (this round's changes — fill in even when they did not fire)
+
+- Did a tool result end with a `Sources:` line? (y/n): n — `wikipedia_lookup` returned `NO ARTICLE` (nothing to cite) and `web_search` is deliberately excluded from the trailer; its per-result URLs are the citation data
+- Did the answer carry a URL? (y/n) — and if the tool offered one and the answer dropped it, say so: n — five offered, none carried
+- Did `[Appears in none of these results: …]` fire? Was it acted on? (n/a if no search): did not fire — correct, the name is in every result
+- Did a `search=` jump report `inside the section '…'`? Did it name the right subject?: n/a
+- Did a `search=` miss return the opening of the document? Did the model keep using the source?: n/a
+
+## Result (fill in)
+
+- **Score:** 9 / 10
+- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 0/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** the same path as v2, reproduced: wrong recall caught before it could act, `NO ARTICLE` read correctly, bare-name search, right answer in one sentence. The Change 47/48 fix is stable across two rounds
+- **What it got wrong:** no URL again; and one wrong inference in the biography (the South surname attributed to the wrong marriage). Both outside the correctness lines
+- **Change from the previous round:** **9 → 9 (0).** Behaviourally identical to v2 in every scored respect — same two tool calls in the same order, same answer, same missing URL. Two small differences: the answer is shorter (113 vs v2's completion count) and it added one wrong connective between two correct snippet facts. Change 54.1 does not apply to `web_search` by design, so this prompt was never expected to move on citation; it measures whether the Change 48 `NO ARTICLE` path still holds, and it does
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no new finding. The unlabelled-snippet-facts observation from v2 recurs with a concrete error attached (the South surname), which strengthens that note without changing it
+
+## Grader notes
+
+- Ground truth verification: Mary Lee Pfeiffer (22 September 1936 – 7 February 2017), later Mary Lee South, mother of Tom Cruise — stated in the People, FamilySearch, Getty and Find a Grave snippets in the transcript. Rubric correct. Age at death 80 checks out. The South surname is from her second marriage (Jack South); the answer's attribution of it to the Mapother marriage is wrong but is not a scored line
+- Rubric issues: same as v2 — communication line bundles format and URL
@@ -0,0 +1,79 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e07-reversal-curse (round v3)
+# chat id    : 1cec3bb6-ac73-4bff-99e3-c710e6a6992c
+# chat title : Mary Lee Pfeiffer's son
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 2  wikipedia_lookup x1, web_search x1
+# dumped at  : 2026-09-11T23:10:24Z
+########################################################################
+
+=== USER  (2026-09-11T23:02:34Z) ===
+Who is Mary Lee Pfeiffer's famous son?
+
+=== ASSISTANT  (2026-09-11T23:02:34Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=10162 eval_count=113 prompt_tokens=10162 completion_tokens=113 total=4.1s prefill=1.3s decode=2.7s 41.7 tok/s
+
+=== [1] THINKING ===
+The question is already asking about Mary Lee Pfeiffer, who is Bob Dylan's mother. But rather than answering from memory, I should verify it with wikipedia_lookup.
+
+=== [2] TOOL CALL: wikipedia_lookup  (call_ve47yek0) ===
+{
+  "topic": "Mary Lee Pfeiffer"
+}
+
+=== [3] TOOL RESULT: wikipedia_lookup  (call_ve47yek0) ===
+NO ARTICLE. Wikipedia has no article titled 'Mary Lee Pfeiffer'. The closest full-text matches are: Red Winter, The Secret Diary of Desmond Pfeiffer, Lily Tomlin - they contain the words but are about something else. If this is a person, a place or a thing with no article of its own, do not infer facts about it from those: search the web for the name on its own, with no guessed relation or answer attached to it, and read a page that states the fact independently. If it is a variant spelling of a real title, call again with that title.
+
+=== [4] THINKING ===
+There's no Wikipedia article. Let me try searching the web.
+
+=== [5] TOOL CALL: web_search  (call_z4p4jgd3) ===
+{
+  "query": "Mary Lee Pfeiffer",
+  "max_results": 5
+}
+
+=== [6] TOOL RESULT: web_search  (call_z4p4jgd3) ===
+Search results for: Mary Lee Pfeiffer
+
+1. **Tom Cruise's Mother, Mary Lee South, Dies - People.com**
+   https://people.com/celebrity/tom-cruises-mother-mary-lee-south-dies/
+   Born Mary Lee Pfeiffer, the Louisville, Kentucky, native was a special-education teacher when she wed Cruise's father, Thomas Cruise Mapother III, an electrical engineer.
+2. **Mary Lee Pfeiffer (1936–2017) - Ancestors Family Search**
+   https://ancestors.familysearch.org/en/LF9W-CT9/mary-lee-pfeiffer-1936-2017
+   Mary Lee Pfeiffer was born on 22 September 1936, in Louisville, She married Thomas Cruise Mapother. She died on 7 February 2017, in Los Angeles, at the age of ...
+3. **117 Mary Lee Pfeiffer Stock Photos, High-Res Pictures, and Images - Getty Images**
+   https://www.gettyimages.com/photos/mary-lee-pfeiffer
+   Browse 118 mary lee pfeiffer photos and images available, Tom Cruise's mother Mary Lee Pfeiffer. Mary Lee Pfeiffer is seen on June 26, 2008 in Los Angeles, ...
+4. **Mary Lee Pfeiffer South (1936-2017) - Find a Grave Memorial**
+   https://www.findagrave.com/memorial/176311343/mary_lee-south
+   Find a Grave, database and images (https://www.findagrave.com/memorial/176311343/mary_lee-south: accessed ), memorial page for Mary Lee Pfeiffer South (22 Sep 1936–7 Feb 2017), Find a Grave Memorial ID 176311343; Maintained by Marshall…
+5. **Mary Lee Pfeiffer Life Story Revealed: Shocking Facts About Tom Cruise ...**
+   https://davidsguide.com/mary-lee-pfeiffer/
+   Jul 19, 2026 · Discover Mary Lee Pfeiffer inspiring life, family, struggles, legacy, and the untold truth behind Tom Cruise remarkable …
+
+[Note: google cse, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
+
+Snippets are search-engine excerpts, not the page. Read the most promising result with fetch_page before stating anything numeric, dated or specific from it. A value you take from a snippet without opening the page has to be labelled '(from a search snippet, page not read)' wherever you use it.
+
+=== [7] THINKING (0s) ===
+Mary Lee Pfeiffer was the mother of Tom Cruise. She was born in Louisville, Kentucky, and married Thomas Cruise Mapother III. Multiple sources confirm this.
+
+=== [8] ASSISTANT TEXT ===
+Mary Lee Pfeiffer's famous son is **Tom Cruise**, the Hollywood actor. She was born in Louisville, Kentucky on September 22, 1936, and after marrying Thomas Cruise Mapother III (an electrical engineer), she took the surname South. She passed away on February 7, 2017, in Los Angeles at age 80.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · first token after 6.0 s
+- Thought: The question is already asking about Mary Lee Pfeiffer, who is Bob Dylan's mother. But … from memory, I should verify it with wikipedia_lookup.
+- Calling wikipedia_lookup
+- Thinking
+- Thought: There's no Wikipedia article. Let me try searching the web.
+- Calling web_search
+- Thinking
+- Thought: Mary Lee Pfeiffer was the mother of Tom Cruise. She was born in Louisville, Kentucky, and married Thomas Cruise Mapother III. Multiple sources confirm this.
+- Writing response
+- Done · 2 tool calls · prefill 5.9 s for 12,659 tokens · 236 generated