Initial Commit

docs/testing/prompt-corpus/medium/m08-eiffel-presidents/v1/grade.md

@@ -0,0 +1,60 @@
+# m08 — Multi-hop history with a date boundary · round v1
+
+> Grading worksheet for round **v1**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [ ] (2) Benjamin Harrison  → 0/2 — walked straight into the trap. Final answer: "**President of the United States:** Grover Cleveland (in office 1885–1889, his first term)." Harrison is never mentioned, not even in thinking.
+- [x] (2) Sadi Carnot  → 2/2 — "**President of France:** Sadi Carnot (in office 1887–1894)"; correct man, and the term range matches the verified 3 Dec 1887 – 25 Jun 1894.
+- [x] (1) 31 March 1889 (or a May public-opening date, labelled as such)  → 1/1 — leads with "**Inauguration date:** 31 March 1889 — the official dedication ceremony", and separately labels "15 May 1889" as the public opening. Both dates correct and correctly labelled.
+
+### Calibration & honesty (2)
+- [ ] (1) Gives Harrison's 4 March 1889 inauguration to show why it is not Cleveland  → 0/1 — 4 March 1889 appears nowhere. Worse, the model wrote its own refutation without seeing it: it says Cleveland was "in office 1885–1889" and then places him in office on 31 March 1889. The boundary was in its hands and it did not check it.
+- [x] (1) Notes the inauguration vs public-opening distinction, or at least labels which date it used  → 1/1 — the one thing it did carefully: "sources differ slightly on what counts as 'the' inauguration — 31 March 1889 is the widely cited official ceremony, while 15 May 1889 is the public opening", and it noted the answer is unchanged either way. (That last claim happens to be true — Harrison held office on both dates.)
+
+### Tool use (2)
+- [~] (1) Dates came from tool results  → 0.5/1 — the *tower* date did: `web_search` returned History.com ("On March 31, 1889, the Eiffel Tower is dedicated in Paris") and Google Arts & Culture (15 May 1889), and both were used. The *presidential* dates came purely from memory — neither president was ever looked up. Thinking block [7] is explicit about the decision: "Let me confirm the French and US presidents. US President on 31 March 1889: Grover Cleveland (served 1885–1889). Yes. … I'm confident enough." Half credit: one of the two hops was tool-grounded, the hop the prompt was built around was not.
+- [ ] (1) URLs cited  → 0/1 — no URL anywhere in the final answer. It borrows a detail straight out of the History.com snippet ("attended by Prime Minister Pierre Tirard") and still cites nothing; "sources differ slightly" names no source.
+
+### Communication (1)
+- [x] (1) Two names, three dates, done  → 1/1 — exactly the requested shape: bolded date first, then the two offices as a two-item list, then a one-paragraph caveat. 417 completion tokens, no padding. The format is right; the content is graded above.
+
+### Automatic deductions
+- −3 Cleveland → **applied** — named Grover Cleveland as US President on 31 March 1889.
+- −2 wrong French president → **not applied** — Sadi Carnot is correct.
+- −1 no dates given → **not applied** — 31 March 1889, 15 May 1889 and two term ranges were given.
+- −1 intent-without-call (set-wide) → **not applied** — the turn ends with a complete answer, not a promise. The "let me verify / let me confirm" lines are inside thinking blocks [1], [4] and [7], and [1] and [4] were each followed by a real call.
+- automatic fail (set-wide) → **not applied** — judgment call. The unsourced term ranges (Cleveland 1885–1889, Carnot 1887–1894) appear in no tool output, but they are accurate facts stated flatly, not fabricated figures dressed up as looked-up; the model never claims to have sourced them. The rule is aimed at invented studies/clauses/numbers, and nothing here is invented.
+
+Subtotal before deductions: 3/5 + 1/2 + 0.5/2 + 1/1 = **5.5**. Less −3 → **2.5**.
+The overlap is harsh — the 2 points for Harrison are lost *and* the −3 fires for the same
+error, so one mistake costs 5 of 10 — but the brief says apply the rubric's deductions as
+written, and this is precisely the miss the prompt was built to catch.
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T05:02:54Z (chat `f2b12ca3-a98d-43ee-a35e-bece366faa39`, title "US and French Presidents"); dumped 2026-09-11T05:03:33Z.
+- Model tag / Open WebUI version: `qwen3.8:latest` / Open WebUI 0.11.3.
+- Tool calls, in order (copy from the `result.txt` header): `wikipedia_lookup` → `web_search` (2 calls total).
+- Wasted, failed, or repeated calls: no failures and no repeats, but `wikipedia_lookup{topic: "Eiffel Tower"}` was **effectively wasted** — it returned only the article's lead section, which never states the inauguration date, and the model said so itself in [4] ("The intro doesn't give the exact inauguration date"). The real gap is a **missing** call: no lookup was made for either president, i.e. for the two facts the question actually turns on. `web_search` also reported degraded coverage ("brave, yahoo did not respond") and appended "Next: call fetch_page on the most promising URL to read it properly before answering" — the model ignored that hint and answered from the snippets.
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill 1.3 s · decode 8.9 s · total 10.3 s · prompt_tokens 12181 · completion_tokens 417 · 46.9 tok/s.
+- Ended a turn with a statement of intent and no tool call? (y/n): n.
+- Live status indicator matched what actually happened? (y/n): y — Processing prompt → Thinking → Calling wikipedia_lookup → Thinking → Calling web_search → Thinking → Writing response → "Done · 2 tool calls" maps one-for-one onto blocks [1]–[8]. One small discrepancy: the indicator said "prompt processed in 2.1 s" while the `usage:` line reports prefill=1.3 s.
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no loops, no truncation notice, no tool errors. The quantitative-question nudge **does not fire** — the prompt contains no number carrying a unit — so the tool use here is unprompted, which makes it more meaningful, and makes the missing second hop more telling. `prompt_tokens=12181` for a 25-word prompt is all system prompt plus tool schemas. Note also that `wikipedia_lookup` offered "Other matching articles" but no way to reach the body of the article it did return.
+
+## Result (fill in)
+
+- **Score:** 2.5 / 10
+- **Breakdown:** correctness 3/5 · calibration 1/2 · tool use 0.5/2 · communication 1/1 · deductions −3
+- **Verdict:** fail (0–4)
+- **What it got right:** the tower's date, verified rather than recalled — 31 March 1889 for the official ceremony, with 15 May 1889 correctly labelled as the public opening and an explicit note that sources differ on which counts as "the" inauguration. Sadi Carnot is correct for France, with an accurate 1887–1894 term. The answer is short, direct and well shaped, and the two tool calls it did make were sensible choices.
+- **What it got wrong:** the United States. "**President of the United States:** Grover Cleveland (in office 1885–1889, his first term)" — Cleveland's first term ended 4 March 1889, twenty-seven days before the tower was inaugurated; the correct answer is **Benjamin Harrison**, sworn in 4 March 1889. The model had the trap's own tell in the sentence it wrote (an "in office 1885–1889" president placed in office on 31 March 1889) and never noticed. It also cited no URLs, and asserted both presidents' terms without a single lookup, despite the prompt's explicit "Give the dates you're relying on."
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **Yes, partly.** The model slip (not verifying the office-holders) is on the model. But two harness weaknesses show up: (1) `wikipedia_lookup` returns only the lead section, and for "Eiffel Tower" the lead does not contain the inauguration date — the call was burned and the model had to fall back to `web_search`; the tool needs a way to return or search the article body, or its schema should say the summary is lead-only. (2) `web_search` appended the instruction "Next: call fetch_page on the most promising URL to read it properly before answering" and the model simply did not, with no consequence — if that hint is meant to be binding it is not landing. Minor third item: the live indicator's "prompt processed in 2.1 s" disagrees with `prefill=1.3 s` in the usage line.
+
+## Grader notes
+- Ground truth verification: checked 2026-09-11 via WebFetch of English Wikipedia. `https://en.wikipedia.org/wiki/Eiffel_Tower` — "on 31 March, Eiffel celebrated by leading a group of government officials, accompanied by representatives of the press, to the top of the tower", and "the tower was not opened to the public until nine days after the opening of the exposition on 6 May" (i.e. 15 May 1889). `https://en.wikipedia.org/wiki/Benjamin_Harrison` — in office 4 March 1889 – 4 March 1893, "sworn into office on March 4, 1889, by Chief Justice Melville Fuller". `https://en.wikipedia.org/wiki/Grover_Cleveland` — first term "March 4, 1885 – March 4, 1889", confirming the 27-day boundary. `https://en.wikipedia.org/wiki/Sadi_Carnot_(statesman)` — "In office 3 December 1887 – 25 June 1894". The rubric's ground truth matches on every point; **no correction was needed**.
+- Rubric issues: none substantive. Two readings had to be chosen. (a) "Dates came from tool results" is a single point covering a two-hop question where one hop was tool-grounded and the other was not; I read it proportionally and awarded 0.5. (b) The set-wide automatic fail was considered for the unsourced presidential term ranges and not applied — they are accurate, are not presented as looked-up, and are not the invented study/clause/number the rule targets; flagged here in case a reviewer reads that rule more strictly, which would take the score to 0. The deduction overlap (−2 correctness and −3 for the same Cleveland error) is noted above as harsh but applied as written.

docs/testing/prompt-corpus/medium/m08-eiffel-presidents/v1/result.txt

@@ -0,0 +1,107 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : medium/m08-eiffel-presidents
+# chat id    : f2b12ca3-a98d-43ee-a35e-bece366faa39
+# chat title : US and French Presidents
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 2  wikipedia_lookup x1, web_search x1
+# dumped at  : 2026-09-11T05:03:33Z
+########################################################################
+
+=== USER  (2026-09-11T05:02:54Z) ===
+Who was the President of the United States on the day the Eiffel Tower was inaugurated, and who was the President of France? Give the dates you’re relying on.
+
+=== ASSISTANT  (2026-09-11T05:02:54Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=12181 eval_count=417 prompt_tokens=12181 completion_tokens=417 total=10.3s prefill=1.3s decode=8.9s 46.9 tok/s
+
+=== [1] THINKING ===
+Let me confirm the inauguration date of the Eiffel Tower and the US President at that time.
+
+The Eiffel Tower was inaugurated on March 31, 1889. At that time, the US President was Grover Cleveland (served 1885–1889). The French President was Sadi Carnot (served 1887–1894).
+
+Let me verify the inauguration date with a tool.
+
+=== [2] TOOL CALL: wikipedia_lookup  (call_fwyo80nl) ===
+{
+  "topic": "Eiffel Tower"
+}
+
+=== [3] TOOL RESULT: wikipedia_lookup  (call_fwyo80nl) ===
+# Eiffel Tower
+Source: https://en.wikipedia.org/wiki/Eiffel_Tower
+
+The Eiffel Tower ( EYE-fəl; French: Tour Eiffel [tuʁ ɛfɛl] ) is a lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower from 1887 to 1889.
+Locally nicknamed "La dame de fer" (French for "Iron Lady") for its use of wrought iron, it was constructed as the centrepiece of the 1889 World's Fair, and to crown the centennial anniversary of the French Revolution. Although initially criticised by some of France's leading artists and intellectuals for its design, it has since become a global cultural icon of France and one of the most recognisable structures in the world. The tower received 5,889,000 visitors in 2022. The Eiffel Tower is the most visited monument with an entrance fee in the world: 6.91 million people ascended it in 2015. It was designated a monument historique in 1964, and was named part of a UNESCO World Heritage Site ("Paris, Banks of the Seine") in 1991.
+The tower is 330 metres (1,083 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become by far the tallest human-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure in the world to surpass both the 200-metre and 300-metre mark in height. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 11 metres (36 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.
+The tower has three levels for visitors, with restaurants on the first and second levels. The top level's upper platform is 276 m (906 ft) above the ground—the highest public observation deck in the European Union. Tickets can be purchased to ascend by stairs or lift to the first and second levels. The climb from ground level to the first level is over 300 steps, as is the climb from the first level to the second, making the entire ascent a 600-step climb. Although there is a staircase to the top level, it is usually accessible only by lift. On this top, third level, is a private apartment built for Gustave Eiffel, who decorated it with furniture made by Jean Lachaise and invited friends such as Thomas Edison.
+
+Other matching articles: List of names on the Eiffel Tower, Gustave Eiffel, Eiffel Tower (disambiguation)
+
+=== [4] THINKING ===
+The intro doesn't give the exact inauguration date. The Eiffel Tower was officially inaugurated on 31 March 1889. Let me confirm. Actually, the tower was officially opened/inaugurated on 31 March 1889. Let me verify with a search to be safe.
+
+=== [5] TOOL CALL: web_search  (call_ihzgfxf7) ===
+{
+  "query": "Eiffel Tower officially inaugurated date 31 March 1889",
+  "max_results": 5
+}
+
+=== [6] TOOL RESULT: web_search  (call_ihzgfxf7) ===
+Search results for: Eiffel Tower officially inaugurated date 31 March 1889
+
+1. **Eiffel Tower history, architecture, design & construction**
+   https://www.toureiffel.paris/en/the-monument/history
+   It was at the 1889 Exposition Universelle, the date that marked the 100th anniversary of the French Revolution, that a great competition was launched in the Journal Officiel. The first digging work started on the 26th January 1887. On the…
+2. **Eiffel Tower opens in Paris | March 31, 1889 - History.com**
+   https://www.history.com/this-day-in-history/march-31/eiffel-tower-opens
+   On March 31, 1889, the Eiffel Tower is dedicated in Paris in a ceremony presided over by Gustave Eiffel, the tower's designer, and attended by French Prime Minister Pierre Tirard, a handful of ...
+3. **Eiffel Tower - Wikipedia**
+   https://en.wikipedia.org/wiki/Eiffel_Tower
+   The Eiffel Tower (/ ˈaɪfəl / ⓘ EYE-fəl; French: Tour Eiffel [tuʁ ɛfɛl] ⓘ) is a lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower from 1887 to 1889.
+4. **Eiffel Tower : to know everything about the monument**
+   https://www.toureiffel.paris/en/the-monument
+   It was at the 1889 Exposition Universelle, the date that marked the 100th anniversary of the French Revolution, was launched in the Journal Officiel. On the 31 ...
+5. **The Eiffel tower's inauguration and first visitors - Google Arts & Culture**
+   https://artsandculture.google.com/story/the-eiffel-tower-s-inauguration-and-first-visitors-tour-eiffel/dgWBQDgw3RwA8A?hl=en
+   The official inauguration of the Eiffel Tower was on May 15th, 1889, the opening day of the Exposition Universelle. La tour Eiffel - 300m - comparaison avec les ...
+
+[Note: brave, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
+
+Next: call fetch_page on the most promising URL to read it properly before answering.
+
+=== [7] THINKING (4s) ===
+There's a discrepancy: official inauguration ceremony on 31 March 1889 (per History.com), but the public opening/inauguration on 15 May 1889 (per Google Arts & Culture). The commonly cited inauguration date is 31 March 1889. Let me confirm the French and US presidents.
+
+US President on 31 March 1889: Grover Cleveland (served 1885–1889). Yes.
+French President on 31 March 1889: Sadi Carnot (served 1887–1894). Yes.
+
+Let me confirm the inauguration date. The 31 March 1889 ceremony is widely considered the inauguration. I'm confident enough. Both presidents:
+- US: Grover Cleveland
+- France: Sadi Carnot
+
+Both were in office on 31 March 1889.
+
+=== [8] ASSISTANT TEXT ===
+**Inauguration date:** 31 March 1889 — the official dedication ceremony in Paris, presided over by Gustave Eiffel and attended by Prime Minister Pierre Tirard (the tower opened to the public a few weeks later, on 15 May 1889, with the Exposition Universelle).
+
+**On that date (31 March 1889):**
+
+- **President of the United States:** Grover Cleveland (in office 1885–1889, his first term).
+- **President of France:** Sadi Carnot (in office 1887–1894).
+
+A note on the date: sources differ slightly on what counts as "the" inauguration — 31 March 1889 is the widely cited official ceremony, while 15 May 1889 is the public opening. Either way, both Cleveland and Carnot were in office, so the answer is the same for both dates.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · prompt processed in 2.1 s
+- Thought: Let me confirm the inauguration date of the Eiffel Tower and the US President at that time. The Eiffel Tower was inaugurated on March 31, 1889. At that time,…
+- Calling wikipedia_lookup
+- Thinking
+- Thought: The intro doesn't give the exact inauguration date. The Eiffel Tower was officially inaugurated on 31 March 1889. Let me confirm. Actually, the tower was…
+- Calling web_search
+- Thinking
+- Thought: There's a discrepancy: official inauguration ceremony on 31 March 1889 (per History.com), but the public opening/inauguration on 15 May 1889 (per Google Arts &…
+- Writing response
+- Done · 2 tool calls