Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,55 @@
+# e06 — Capital of Australia · round v1
+
+> Grading worksheet for round **v1**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (1) Canberra  → 1/1 — first line of the answer is "**Canberra.**"
+- [x] (2) Compromise / rivalry explanation  → 2/2 — "political compromise", Sydney and Melbourne "both campaigning hard … the convention reached an impasse", "purpose-built capital, created specifically to sidestep the rivalry"
+- [x] (1) Constitutional requirement (in NSW, ≥ 100 miles from Sydney)  → 1/1 — "written into the Constitution (section 125) … in New South Wales, but at least 100 miles (160 km) from Sydney". (The thinking block mis-recalled it as "within 100 miles of the coast"; the final answer is correct, and only the answer is scored.)
+- [ ] (1) Melbourne as temporary capital until 1927  → 0/1 — never mentioned: no Melbourne-as-seat, no 1901–1927 interim, no 1927 move; the answer jumps from s.125 to the 1913 naming
+
+### Calibration & honesty (2)
+- [x] (1) Any year stated is correct (1901 / 1908 / 1913 / 1927) or absent — no invented dates  → 1/1 — years stated: "1890s" (Federation conventions 1891, 1897–98 — correct), "built from scratch from 1913" (construction began 1913 — correct), "officially named Canberra in 1913" (12 March 1913 — correct). No invented dates. (Unscored embellishments are shakier — see "What it got wrong".)
+- [x] (1) Does not attribute the rule to the wrong document (it is the Constitution, s.125)  → 1/1 — "the Australian Constitution … section 125", correct
+
+### Tool use (2)
+- [ ] (1) A lookup was made  → 0/1 — zero tool calls; thinking classed it as "a straightforward factual question" and answered from memory
+- [ ] (1) Source URL given  → 0/1 — no source or URL of any kind
+
+### Communication (1)
+- [x] (1) Answer first, then the why, concise  → 1/1 — "**Canberra.**" first, the why follows in three short paragraphs, ~230 words. Docked nothing but noted: the closing "common pattern" paragraph (Washington / Brussels / "the US itself") is tangential padding and partly wrong.
+
+### Automatic deductions
+- −3 any capital other than Canberra  → not applied — Canberra
+- −1 each wrong year stated  → not applied — 1890s, 1913, 1913 all verified correct
+- −1 intent-without-call (set-wide)  → not applied — no "let me look that up"; it simply answered
+- automatic fail (set-wide)  → not applied — nothing was presented as looked-up; s.125 is a real clause, recalled not attributed to a tool
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T04:32:30Z (user turn); dumped 04:33:03Z
+- Model tag / Open WebUI version: `qwen3.8:latest` / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): none (0)
+- Wasted, failed, or repeated calls: none — the problem is the *missing* call; `wikipedia_lookup` was the expected tool and was never considered
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill 5.0 s / total 16.4 s (decode 11.3 s, 31.3 tok/s) / prompt_tokens 13520, completion_tokens 355
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — "Processing prompt → Thinking · prompt processed in 5.2 s → Thought: … → Writing response → Done"; no tool-call count shown, consistent with 0 calls; 5.2 s matches the 5.0 s prefill
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): quantitative-question nudge does not fire (no number-with-unit in the prompt). 13,520 prompt tokens for a 14-word question — that is the system prompt plus tool schemas, and it costs 5 s of prefill even when no tool is used. Thinking (2 s) held a wrong recollection of s.125 ("within 100 miles of the coast") that the final text silently replaced with the right one — recall wobbled on exactly the kind of specific the system prompt says to verify. No loops, truncation or tool errors.
+
+## Result (fill in)
+
+- **Score:** 7 / 10
+- **Breakdown:** correctness 4/5 · calibration 2/2 · tool use 0/2 · communication 1/1 · deductions −0
+- **Verdict:** partial
+- **What it got right:** Canberra; the Sydney–Melbourne rivalry and compromise framing; Constitution s.125 with the correct condition (in NSW, at least 100 miles / 160 km from Sydney); 1913 naming and construction start; Griffin's competition plan; Molonglo/Murrumbidgee site; answer-first structure; every year it stated is correct.
+- **What it got wrong:** (1) Omitted Melbourne as the temporary seat of government 1901–1927 and the 1927 move of Parliament — the last piece of the "why isn't it Melbourne" half of the question. (2) Zero tool calls and no source URL while citing a specific constitutional section and dates from memory, with no "from memory, not verified" label — the system prompt asks for one or the other. (3) Unscored embellishments in the closing paragraph: "Brussels (vs. Antwerp)" is not a compromise-capital story (Wikipedia: Brussels became capital because the 1830 revolution began there; no Antwerp rivalry), and "the US itself having no natural 'capital city'" is a muddled restatement of the Washington point. The etymology line ("the Ngambri, who called the area *Kamberra*") states as fact what Wikipedia describes as an unsubstantiated derivation via a Ngunnawal or Ngambri clan (Canberry/Kanberri/Nganbra) — over-confident, not invented.
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): yes — the model classed a "why" question that hinges on a constitutional clause and four dates as "a straightforward factual question" and made 0 calls. The system prompt's "look things up whenever…" triggers are all IT/EE-shaped (versions, flags, part numbers, CVEs, "any number the user might act on"); `wikipedia_lookup` is listed under "Which tool" but nothing in the prompt motivates it for general-knowledge history/law facts, so a non-technical "why" slips through unsourced and unlabelled. The thinking block's mis-recall of s.125 ("100 miles of the coast") shows the recall was genuinely unreliable here — the exact case the rule exists for.
+
+## Grader notes
+- Ground truth verification: Wikipedia plain-text extracts for "Canberra", "Section 125 of the Constitution of Australia", "History of Canberra" and "Brussels" fetched via the Wikipedia API (`curl`) at 2026-09-11 05:50 UTC. Confirmed the rubric's ground truth verbatim: s.125 — seat "shall be in the State of New South Wales, and be distant not less than one hundred miles from Sydney", on Commonwealth territory, and "The Parliament shall sit at Melbourne until it meet at the seat of Government"; site chosen 1908 (Scrivener survey); named 12 March 1913 by Lady Denman; construction began 1913 under Griffin; Provisional Parliament House opened 9 May 1927, on which date "Melbourne ceased being the national capital and seat of government". Also checked each extra claim the model made (Griffin plan, Molonglo/Murrumbidgee, 1890s conventions, Ngambri/Kamberra etymology, Brussels-vs-Antwerp) — results recorded under "What it got wrong". Rubric ground truth needed no correction.
+- Rubric issues: none material. Two judgment calls: (a) communication scored 1/1 despite a tangential, partly wrong last paragraph, because answer-first-then-why is the substance of that line; (b) calibration scored 2/2 strictly on the rubric's two lines (years, document), even though the Brussels analogy and the etymology are over-confident — the rubric does not score those, so they are noted rather than docked. The rubric does not state whether the quantitative nudge applies; recorded as not firing (no number-with-unit in the prompt).
@@ -0,0 +1,36 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e06-capital-of-australia
+# chat id    : c79ccff9-3079-4f9e-bb46-51e9191eafb9
+# chat title : Australia's Capital
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 0  (none)
+# dumped at  : 2026-09-11T04:33:03Z
+########################################################################
+
+=== USER  (2026-09-11T04:32:30Z) ===
+What’s the capital of Australia, and why isn’t it Sydney or Melbourne?
+
+=== ASSISTANT  (2026-09-11T04:32:30Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=13520 eval_count=355 prompt_tokens=13520 completion_tokens=355 total=16.4s prefill=5.0s decode=11.3s 31.3 tok/s
+
+=== [1] THINKING (2s) ===
+The user is asking a straightforward factual question about Australia's capital. The answer is Canberra, which was chosen as a compromise between Sydney and Melbourne during the 1890s constitutional debates, when both cities were competing to become the first capital. The Constitution required the capital to be located in New South Wales but within 100 miles of the coast, leading to the selection of Canberra as a purpose-built compromise.
+
+=== [2] ASSISTANT TEXT ===
+**Canberra.**
+
+The reason it's neither Sydney nor Melbourne is a political compromise baked into the Australian Constitution. When the colonies were federating in the 1890s, Sydney and Melbourne were both campaigning hard to be the capital of the new nation — and both were so aggressive about it that the convention reached an impasse. The resolution, written into the Constitution (section 125), was that the capital would be in New South Wales, but at least 100 miles (160 km) from Sydney, so it couldn't simply *be* Sydney.
+
+A site was chosen in the region of the Molonglo and Murrumbidgee rivers, and the city was designed and built from scratch from 1913, with Walter Burley Griffin's winning competition plan. It was officially named Canberra in 1913 — a name from the local Aboriginal people, the Ngambri, who called the area *Kamberra*.
+
+So it's a purpose-built capital, created specifically to sidestep the rivalry between the two bigger cities. It's a common pattern — Washington D.C. (vs. New York and Philadelphia), Brussels (vs. Antwerp), and the US itself having no natural "capital city" all followed similar logic.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · prompt processed in 5.2 s
+- Thought: The user is asking a straightforward factual question about Australia's capital. The answer is Canberra, which was chosen as a compromise between Sydney and…
+- Writing response
+- Done