Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,65 @@
+# m08 — Multi-hop history with a date boundary · round v3
+
+> Grading worksheet for round **v3**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [ ] (2) Benjamin Harrison  → **0/2** — "**US President on 31 March 1889:** **Grover Cleveland**". The trap, walked into, and with the contradiction written into the same sentence: "which ran from 4 March 1885 to 4 March 1889, so he was still in office at the inauguration" — a term that ended on 4 March cannot cover 31 March. The answer also mislabels Cleveland's *first* term as "his non-consecutive second term (22nd and then 24th president)"
+- [x] (2) Sadi Carnot  → 2/2 — "**Sadi Carnot**, who served as President of the French Republic from 3 December 1887 until his assassination in 1894"; the dates are correct (recalled, not retrieved — see tool use)
+- [x] (1) 31 March 1889 (or a May public-opening date, labelled as such)  → 1/1 — 31 March 1889 from the retrieved section, with the ceremony described from it; the 15 May public opening correctly derived from "nine days after the opening of the exposition on 6 May"
+
+### Calibration & honesty (2)
+- [ ] (1) Gives Harrison's 4 March 1889 inauguration to show why it is not Cleveland  → **0/1** — Harrison is never mentioned. The 4 March 1889 date *is* given, as the end of Cleveland's term, and the answer then asserts the opposite of what it implies
+- [x] (1) Notes the inauguration vs public-opening distinction, or at least labels which date it used  → 1/1 — "Inauguration date relied on: … 31 March 1889 … Note the tower wasn't opened to the public until 15 May 1889"
+
+### Tool use (2)
+- [~] (1) Dates came from tool results  → **0.5/1** — 31 March and the 6 May / 15 May opening are from the retrieved Eiffel Tower section. Cleveland's term dates and Carnot's 3 December 1887 are recall; **neither president was looked up**, and thinking block [4] says why: "these are well-established historical facts, so I can answer directly" — the sentence Change 51's sixth trigger was written to overrule
+- [ ] (1) URLs cited  → **0/1** — the answer names the source in prose ("per Wikipedia's 'Inauguration and the 1889 exposition' section") but gives no URL, and the prompt's last sentence is "Give the dates you're relying on"
+
+### Communication (1)
+- [x] (1) Two names, three dates, done  → 1/1 — three bold blocks, about 130 words
+
+### Automatic deductions
+- **−3 Cleveland → APPLIED**
+- −2 wrong French president → not applied
+- −1 no dates given → not applied
+- −1 intent-without-call (set-wide) → not applied — "Let me confirm the date via Wikipedia" is followed by the call. Thinking block [4]'s "a quick check wouldn't hurt" is a consideration declined in the next clause, not an announced call
+- automatic fail (set-wide) → not applied — Cleveland's term dates are correct as dates and are stated as recall, not attributed to the article
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T23:32:50Z (dumped 23:33:35Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 1 — `wikipedia_lookup` {topic: "Eiffel Tower", search: "inaugurat"}
+- Wasted, failed, or repeated calls: none — the one call landed exactly on the inauguration section. The problem is the calls not made: v2 made four (tower, Carnot, Cleveland, and a second tower jump); v3 made one and answered the two office-holder questions from memory
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=3.5s / total=13.0s (decode=9.5s, 42.2 tok/s) / prompt_eval_count=10741, completion_tokens=399
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — `Done · 1 tool call · prefill 5.9 s for 12,684 tokens · 506 generated`
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): the thinking is the whole story. Block [1] opens with "US President at the time: Grover Cleveland (3rd term, 1885–1889)" — a term number that does not exist — and block [4] writes "his second term ran from March 4, 1885 to March 4, 1889. So on March 31, 1889 it was still Cleveland": the arithmetic that refutes the claim is in the sentence that makes it, and the model reads past it. v2's thinking wrote the same sentence and *noticed* ("Wait…"). Nudge does not fire
+
+### Change 54 checks (this round's changes — fill in even when they did not fire)
+
+- Did a tool result end with a `Sources:` line? (y/n): **y** — `Sources: https://en.wikipedia.org/wiki/Eiffel_Tower`
+- Did the answer carry a URL? (y/n) — and if the tool offered one and the answer dropped it, say so: **n — offered and dropped**; the source is named in prose with the section title, which is more attribution than v2 gave and still not a URL
+- Did `[Appears in none of these results: …]` fire? Was it acted on? (n/a if no search): n/a
+- Did a `search=` jump report `inside the section '…'`? Did it name the right subject?: **y** — `[Jumped to 'inaugurat' at character 13,662 of 44,949; this is inside the section 'Construction']`. Accurate: the match is the "=== Inauguration and the 1889 exposition ===" sub-heading, which sits under the "Construction" section in the article's hierarchy, and the window contains the sub-heading itself. The model then cited the *sub*-section by name in its answer, so the section context was read and used
+- Did a `search=` miss return the opening of the document? Did the model keep using the source?: n/a
+
+## Result (fill in)
+
+- **Score:** 2.5 / 10
+- **Breakdown:** correctness 3/5 · calibration 1/2 · tool use 0.5/2 · communication 1/1 · deductions −3
+- **Verdict:** fail
+- **What it got right:** the tower's inauguration date and the public-opening distinction, both from the article, both labelled; Carnot; a compact answer that names its source section
+- **What it got wrong:** the question. Cleveland instead of Harrison, asserted with the disproof in the same sentence, after deciding that the two office-holders were "well-established historical facts" not worth a lookup. Also no URL, and Cleveland's term misnumbered
+- **Change from the previous round:** **8.5 → 2.5 (−6). The full v2 gain given back.** v2's grade credited Change 51's sixth trigger with the Cleveland lookup that let the model catch the boundary; v3 shows the trigger is not reliable — the same model, same prompt, same tool set, and this time the "everyone knows this" exemption won. The one retrieval it did make was better than any of v2's (one call, straight to the section, with 54.3's heading note), and it did not matter, because the fact that decides the answer was never retrieved. Same score as v1, same wrong answer as v1, one tool call more than v1
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes — the clearest evidence in the round that the Change 51 recall trigger is a coin flip on office-holder questions.** Three rounds: v1 no lookup (Cleveland), v2 lookup (Harrison), v3 no lookup (Cleveland). Nothing in the harness changed between v2 and v3 that touches this decision. The tool-side lever, consistent with the log's principle, is to make the *first* lookup carry the fact: a `wikipedia_lookup` on an event that returns a dated section could be followed — or the model could be told in the result — that a "who held office on <date>" question needs the office-holder's article with `search=<year>`. Weaker but cheaper: the quantitative nudge does not fire on this prompt because there is no unit; a date in the question is arguably the quantitative fact it should treat as one. Positive, and worth keeping separate: 54.3's section note fired correctly on a Wikipedia `==` heading and the model used the section name in its attribution
+
+## Grader notes
+
+- Ground truth verification: as in v2 — the retrieved text gives 31 March 1889 for the ceremony and "nine days after the opening of the exposition on 6 May" (= 15 May) for the public opening; Carnot 3 December 1887 – 25 June 1894; Cleveland's first term ended and Harrison's began on 4 March 1889, twenty-seven days before the ceremony. Cleveland was the 22nd (1885–89) and 24th (1893–97) president; the term in question is his first, not his second, and there was no "3rd term"
+- Rubric issues: none. The split tool-use line again produced the informative half-point; v2's suggestion of returning the lead plus the first body section on intro-only calls would not have helped here, since the missing lookups were never made
@@ -0,0 +1,82 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : medium/m08-eiffel-presidents (round v3)
+# chat id    : 046d9f50-abd6-4971-ada2-47729ab4aa2a
+# chat title : US and French Presidents
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 1  wikipedia_lookup x1
+# dumped at  : 2026-09-11T23:33:35Z
+########################################################################
+
+=== USER  (2026-09-11T23:32:50Z) ===
+Who was the President of the United States on the day the Eiffel Tower was inaugurated, and who was the President of France? Give the dates you're relying on.
+
+=== ASSISTANT  (2026-09-11T23:32:50Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=10741 eval_count=399 prompt_tokens=10741 completion_tokens=399 total=13.0s prefill=3.5s decode=9.5s 42.2 tok/s
+
+=== [1] THINKING ===
+Eiffel Tower inauguration: March 31, 1889 (official opening ceremony). US President at the time: Grover Cleveland (3rd term, 1885–1889). French President: Sadi Carnot. Let me confirm the date via Wikipedia.
+
+=== [2] TOOL CALL: wikipedia_lookup  (call_xulw2esn) ===
+{
+  "topic": "Eiffel Tower",
+  "search": "inaugurat"
+}
+
+=== [3] TOOL RESULT: wikipedia_lookup  (call_xulw2esn) ===
+# Eiffel Tower
+Source: https://en.wikipedia.org/wiki/Eiffel_Tower
+(Wikipedia, full article, 44,949 characters)
+
+[Jumped to 'inaugurat' at character 13,662 of 44,949; this is inside the section 'Construction']
+
+…
+sandboxes on the scaffold. Although construction involved 300 on-site employees, due to Eiffel's safety precautions and the use of movable gangways, guardrails and screens, only one person died.
+
+=== Inauguration and the 1889 exposition ===
+
+The main structural work was completed at the end of March 1889 and, on 31 March, Eiffel celebrated by leading a group of government officials, accompanied by representatives of the press, to the top of the tower. Because the lifts were not yet in operation, the ascent was made by foot, and took over an hour, with Eiffel stopping frequently to explain various features. Most of the party chose to stop at the lower levels, but a few, including the structural engineer, Émile Nouguier, the head of construction, Jean Compagnon, the President of the City Council, and reporters from Le Figaro and Le Monde Illustré, completed the ascent. At 2:35 pm, Eiffel hoisted a large Tricolour to the accompaniment of a 25-gun salute fired at the first level.
+There was still work to be done, particularly on the lifts and facilities, and the tower was not opened to the public until nine days after the opening of the exposition on 6 May; even then, the lifts had not been completed. The tower was an instant success with the public, and nearly 30,000 visitors made the 1,710-step climb to the top before the lifts entered service on 26 May. Tickets cost 2 francs for the first level, 3 for the second, and 5 for the top, with half-price admission on Sundays, and by the end of the exhibition there had been 1,896,987 visitors.
+After dark, the tower was lit by hundreds of gas lamps, and a beacon sent out three beams of red, white and blue light. Two searchlights mounted on a circular rail were used to illuminate various buildings of the exposition. The daily opening and closing of the exposition were announced by a cannon at the top.
+
+On the second level, the French newspaper Le Figaro had an office and a printing press, where a special souvenir edition, Le Figaro de la Tour, was made.
+At the top, there was a post office where visitors could send letters and postcards as a memento of their visit. Graffitists were also catered for: sheets of paper were mounted on the walls each day for visitors to record their impressions of the tower. Gustave Eiffel described the collection of responses as "truly curious".
+
+Famous visitors to the tower included the Prince of Wales, Sarah Bernhardt, "Buffalo Bill" Cody (his Wild West show was an attraction at the exposition) and Thomas Edison. Eiffel invited Edison to his private apartment at the top of the tower, where Edison presented him with one of his phonographs, a new invention and one of the many highlights of the exposition. Edison signed the guestbook with this message on September 10, 1889: To M Eiffel the Engineer the brave builder of so gigantic and original specimen of modern Engineering from one who has the greatest respect and admiration for all Engineers including the Great Engineer the Bon Dieu, Thomas Edison.
+Eiffel made use of his apartment at the top of the tower to carry out meteorological observations, and also used the tower to perform experiments on the action of air resistance on falling bodies.
+
+=== Subsequent events ===
+
+Originally commissioned as a temporary structure for the 1889 Exposition Universelle, Eiffel was slated for dismantling in 1909, at which point ownership would have reverted to the City of Paris. While the original design contest necessitated that the tower be easily disassembled, the structure was spared from scheduled demolition due to its demonstrated utility in early 20th-century scientific and technological advancements, specifically radio telegraphy. Following the expiration of the original concession, the tower was permitted to remain, subsequently becoming an integral part of the International Time Service in 1910.
+For the 1900 Exposition Universelle, the lifts in the east and west legs were replaced by lifts running as far as the second level constructed by the French firm Fives-Lille. These had a compensating mechanism to keep the floor level as the angle of ascent changed at the first level, and were driven by a similar hydraulic mechanism as the Otis lifts, although this was situated at the base of the tower. Hydraulic pressure was provided by pressurised accumulators located near this mechanism. At the same time the lift in the north pillar was removed and replaced by a staircase to the first level. The layout of both first and second levels was modified. The original lift in the south pillar was removed 13 years later.
+
+On 19 October 1901, Alberto Santos-Dumont, flying his No.6 airship, won a 100,000-franc prize offered by Henri Deutsch de la Meurthe for the first person to make a flight from St. Cloud to the Eiffel Tower and back in less than half an hour.
+From 1910, the astronomical clocks of the Paris Observatory sent the time to sea daily through the Eiffel Tower within a radius of 5 000 km. The development of wireless telegraphy allowed unifying Universal Time. On 9 March 1911, France adopted Greenwich Mean Time by law. However, the law did not refer to Greenwich Prime Meridian, but to the local mean t
+…
+
+[Showing the region around 'inaugurat'. The full document is 44,949 characters - call wikipedia_lookup with topic='Eiffel Tower', full_article=true and offset= to read more.]
+
+---
+Sources: https://en.wikipedia.org/wiki/Eiffel_Tower
+
+=== [4] THINKING (4s) ===
+The inauguration ceremony was held on March 31, 1889. The US President at the time: Grover Cleveland (his second term was from 1885 to 1889). The French President: Sadi Carnot (in office from 1887 to 1894). Let me quickly verify — Cleveland served as the 22nd and 24th President, and his second term ran from March 4, 1885 to March 4, 1889. So on March 31, 1889 it was still Cleveland. Sadi Carnot took office on December 3, 1887. Both are correct. I'm confident in these facts, but a quick check wouldn't hurt — these are well-established historical facts, so I can answer directly.
+
+=== [5] ASSISTANT TEXT ===
+**Inauguration date relied on:** The Eiffel Tower was inaugurated on **31 March 1889**, when Gustave Eiffel led government officials and press representatives to the top and raised the French tricolour with a 25-gun salute (per Wikipedia's "Inauguration and the 1889 exposition" section). Note the tower wasn't opened to the public until 15 May 1889.
+
+**US President on 31 March 1889:** **Grover Cleveland** — he was serving his non-consecutive second term (22nd and then 24th president), which ran from 4 March 1885 to 4 March 1889, so he was still in office at the inauguration.
+
+**French President on 31 March 1889:** **Sadi Carnot**, who served as President of the French Republic from 3 December 1887 until his assassination in 1894.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · first token after 6.0 s
+- Thought: Eiffel Tower inauguration: March 31, 1889 (official opening ceremony). US President at … President: Sadi Carnot. Let me confirm the date via Wikipedia.
+- Calling wikipedia_lookup
+- Thinking
+- Thought: The inauguration ceremony was held on March 31, 1889. The US President at the time: … well-established historical facts, so I can answer directly.
+- Writing response
+- Done · 1 tool call · prefill 5.9 s for 12,684 tokens · 506 generated