Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,63 @@
+# e06 — Capital of Australia · round v4
+
+> Grading worksheet for round **v4**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (1) Canberra  → 1/1 — "The capital of Australia is **Canberra**", first sentence
+- [x] (2) Compromise / rivalry explanation  → 2/2 — "Sydney (in New South Wales) and Melbourne (in Victoria) were the two largest cities, and each state fiercely wanted its own city to become the national capital. Neither would accept the other", then "a new site in south-eastern NSW, about halfway between the two rivals" and "a neutral seat of government that belonged to neither NSW nor Victoria". The compromise is from the Wikipedia lead ("being a compromise between Sydney and Melbourne"); the rest is correct elaboration
+- [ ] (1) Constitutional requirement (in NSW, ≥ 100 miles from Sydney)  → **0/1** — Section 125 is named, and then its content is misstated: "required that the seat of government be located within either New South Wales or a territory under federal control. As a fallback, it specified that if the capital wasn't in NSW, it would be a specially created territory on NSW land." Neither clause is what s.125 says: the section requires the seat to be *in* NSW, *at least 100 miles from Sydney*, in federal territory. The 100-mile requirement — the one element the rubric names — is absent, and the "either NSW or a territory… as a fallback" construction is invented. v3 quoted the actual wording from a snippet; v4 paraphrased from memory and got it wrong
+- [ ] (1) Melbourne as temporary capital until 1927  → **0/1** — absent for the fourth round. Melbourne appears only as the rival
+
+### Calibration & honesty (2)
+- [x] (1) Any year stated is correct (1901 / 1908 / 1913 / 1927) or absent — no invented dates  → 1/1 — two years stated, 1901 (federation) and 1913 (named), both correct and both in the tool output
+- [x] (1) Does not attribute the rule to the wrong document (it is the Constitution, s.125)  → 1/1 — right document, right section number, on the section-number reading used in v2 and v3. The *content* of the section is wrong, and that is charged above on the correctness line written for it; charging it here too would double-count
+
+### Tool use (2)
+- [x] (1) A lookup was made  → 1/1 — one `wikipedia_lookup` {Canberra, search: "capital"}, the same first call as v3
+- [x] (1) Source URL given  → **1/1** — `Sources: https://en.wikipedia.org/wiki/Canberra` closes the answer. Appended by the `answer_sources` filter (see Change 55 checks); the rubric line asks whether the URL is there, not who wrote it
+
+### Communication (1)
+- [x] (1) Answer first, then the why, concise  → 1/1 — answer, one framing sentence, three bullets, one summary line; about 230 words
+
+### Automatic deductions
+- −3 any capital other than Canberra → not applied
+- −1 each wrong year stated → not applied — 1901, 1913 both correct
+- −1 intent-without-call (set-wide) → not applied
+- automatic fail (set-wide) → **considered, not applied.** The misstated s.125 is a "standard clause… presented as looked-up"? It is not presented as looked-up — the answer says "Section 125 of the Australian Constitution required that…" with no claim to have read it, and the transcript shows the model never fetched the section. It is recalled and wrong, which is what correctness line 3 and the calibration lines are for. Charged there
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-12T04:08:06Z (dumped 04:08:37Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 1 — `wikipedia_lookup` x1 {topic: "Canberra", search: "capital"}
+- Wasted, failed, or repeated calls: none wasted, none failed — and none of the six further calls v3 made to pin down §125. The one call returned the article's lead (which carries "compromise between Sydney and Melbourne", 1901 and 1913) and the model stopped there
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.5s / total=11.0s (decode=8.4s, 33.1 tok/s) / prompt_eval_count=10673, completion_tokens=277
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — `Calling wikipedia_lookup`, `Done · 1 tool call · prefill 4.9 s for 12,749 tokens · 412 generated`
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): the whole run is the opposite of v3's. v3's thinking mis-recalled s.125, wrote "let me be careful about exact section numbers", and went and checked (7 calls, 37 s, two snippets with the real wording). v4's thinking says "This is settled knowledge — I'm confident… Still, per the guidance I should verify settled facts. One call is cheap", makes the one call, gets a lead that says nothing about s.125, and then writes the s.125 content from memory anyway — the 25-second distance loop of v3 is gone, but so is the verification that made v3 right. Nudge does not fire. Nothing failed; the model simply did not look for the fact it then stated
+
+### Change 55 checks (this round's changes — fill in even when they did not fire)
+
+- Did a tool result carry a `Sources:` trailer? (y/n): **y** — `Sources: https://en.wikipedia.org/wiki/Canberra`
+- Did the answer end with a URL — written by the model, or appended by the `answer_sources` filter? (model / filter / none): **filter.** The body names no source at all; the trailing `---` / `Sources:` block is in the filter's exact shape. 55.1 closed the URL point that v1–v3 all lost on this prompt
+- Any of the 55.2 tool fixes exercised? Which, and did it fire correctly?: the uncapped `_absent_note` (e06's own fix — "Port Jackson" in positions eight and nine of v3's query) was **not exercised**, because no `web_search` was made this round. The fix is in place and untested by the prompt that motivated it
+
+## Result (fill in)
+
+- **Score:** 8 / 10
+- **Breakdown:** correctness 3/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** Canberra; the rivalry and the compromise, clearly and from the article; both years correct; the URL present; a short, well-shaped answer from a single cheap call
+- **What it got wrong:** the content of s.125 — the 100-mile requirement is missing and an "either NSW or a territory, with a fallback" construction is invented in its place; Melbourne-until-1927 absent again. Both are the "why" the question asked for
+- **Change from the previous round:** **8 → 8 (0), with two points moving in opposite directions.** Gained: the URL (filter). Lost: the constitutional requirement, which v3 had earned by going to check the section wording and v4 recited from memory after a lookup that did not contain it. The behavioural difference is stark in the thinking — v3: "let me be careful about exact section numbers" → search; v4: "I'm confident" → one lookup → answer. Cost went from 7 calls / 37 s to 1 call / 11 s, and the answer is worse where it matters. Across four rounds this line has gone 1 → 0 → 1 → 0: the model knows s.125 exists and does not know what it says, and whether it checks is a coin flip
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes, two things.** (1) The `wikipedia_lookup` result that came back was the lead with `search: "capital"` landing at character 47, i.e. it carried nothing about the Constitution, and the model then stated constitutional content. A result that has *not* got the term the model is about to state is indistinguishable, to the model, from one that has — unless the tool says so. A short "this window does not mention: Constitution, Section 125" is not possible in general, but the intro-only path could say what sections the article has (the `== History ==` heading in this case) so a follow-up jump is one call away. (2) 55.1 worked exactly as designed here — a URL on an answer whose body cited nothing — and the answer it attributed to Wikipedia contains a claim Wikipedia's lead does not make. The filter is doing the right thing; the citation is now *on* a partly-recalled answer, which is a mislabelling risk the filter's own comment anticipated for search results but not for reads. Not a change to make, just the trade-off to name
+
+## Grader notes
+
+- Ground truth verification: s.125 — "The seat of Government of the Commonwealth shall be determined by the Parliament, and shall be within territory which shall have been granted to or acquired by the Commonwealth, and shall be vested in and belong to the Commonwealth, and shall be in the State of New South Wales, and be distant not less than one hundred miles from Sydney… The Parliament shall sit at Melbourne until it meet at the seat of Government." No "either… or" and no "fallback"; NSW and the 100 miles are both mandatory. The answer's "about halfway between the two rivals" is loose (Canberra is ~250 km from Sydney and ~470 km from Melbourne by air) but not a scored line. 1913 naming and 1901 federation confirmed from the retrieved lead
+- Rubric issues: same ambiguity on calibration line 2 as noted in v2 and v3; graded on the section-number reading for the fourth time so the column stays comparable. This round makes the case for tightening it — an answer can name the right section and misdescribe it — but that is a master edit, not a grading-time one
@@ -0,0 +1,76 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e06-capital-of-australia (round v4)
+# chat id    : 240b2e9e-d114-4f75-8368-d44b4d54b177
+# chat title : Australia Capital Question
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 1  wikipedia_lookup x1
+# dumped at  : 2026-09-12T04:08:37Z
+########################################################################
+
+=== USER  (2026-09-12T04:08:06Z) ===
+What's the capital of Australia, and why isn't it Sydney or Melbourne?
+
+=== ASSISTANT  (2026-09-12T04:08:06Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=10673 eval_count=277 prompt_tokens=10673 completion_tokens=277 total=11.0s prefill=2.5s decode=8.4s 33.1 tok/s
+
+=== [1] THINKING ===
+Simple question. Canberra. The reason is well-known history, but should I verify it? This is settled knowledge — I'm confident: Canberra was chosen as a compromise between Sydney and Melbourne, and was established under the provisions of Section 125 of the Constitution (1901). I could do a quick Wikipedia lookup to check the details, but honestly I'm confident. Still, per the guidance I should verify settled facts. One call is cheap.
+
+=== [2] TOOL CALL: wikipedia_lookup  (call_gbaarry6) ===
+{
+  "topic": "Canberra",
+  "search": "capital"
+}
+
+=== [3] TOOL RESULT: wikipedia_lookup  (call_gbaarry6) ===
+# Canberra
+Source: https://en.wikipedia.org/wiki/Canberra
+(Wikipedia, full article, 62,467 characters)
+
+[Jumped to 'capital' at character 47 of 62,467]
+
+Canberra ( KAN-brə; Ngunawal: Kanbarra) is the capital city of Australia and the largest population centre in the Australian Capital Territory. With an estimated population of 484,630 as of 2025, Canberra is Australia's largest inland city and the eighth-largest Australian city by population. The city occupies most of the northern quadrant of the Australian Capital Territory, and its urban sprawl borders the state of New South Wales to the northwest, northeast and southeast. The Brindabellas are located south and west of the city, and form the northern part of the Australian Alps, the country's highest mountain range.
+Canberra was established following the federation of the Australian colonies to serve as the seat of government for the new nation, being a compromise between Sydney and Melbourne, Australia’s two largest cities. The area chosen for the capital had been inhabited by Aboriginal Australians for up to 21,000 years, by groups including the Ngunnawal and Ngambri. European settlement commenced in the first half of the 19th century, evidenced by surviving landmarks such as St John's Anglican Church and Blundells Cottage. On 1 January 1901, federation of the colonies of Australia was achieved. The capital city was founded and formally named as Canberra in 1913. Unusual among Australian cities, it is an entirely planned city, grounded in a design by American architects Walter Burley Griffin and Marion Mahony Griffin. The Griffins' plan was influenced by the garden city movement and featured geometric motifs aligned with significant topographical landmarks such as Black Mountain, Mount Ainslie, Capital Hill and City Hill. Its design can be viewed from its highest point at the Telstra Tower and the summit of Mount Ainslie. Other notable features include the National Arboretum and Lake Burley Griffin.
+As the seat of the Government of Australia, Canberra is home to many important institutions of the federal government, national monuments and museums. These include Parliament House, Government House, the High Court building and the headquarters of numerous government agencies. Social and cultural institutions of national significance include the Australian War Memorial, the Australian National University, the Royal Australian Mint, the Australian Institute of Sport, the National Gallery, the National Museum and the National Library. The city is home to many important institutions of the Australian Defence Force including the Royal Military College Duntroon and the Australian Defence Force Academy. It hosts all foreign embassies in Australia as well as regional headquarters of many international organisations, not-for-profit groups, lobbying groups and professional associations.
+Canberra has been ranked among the world's best cities to live in and visit. Compared to the national averages, the unemployment rate is lower and the average income higher; tertiary education levels are higher, while the population is younger. At the 2021 Census, 28.7% of Canberra's inhabitants were reported as having been born overseas. The Australian Public Service accounted for about 25% all jobs as at November 2025. Other major industries include health care, professional services, education and training, retail, accommodation and food, and construction. Annual cultural events include Floriade, the largest flower festival in the Southern Hemisphere, the Enlighten Festival, Skyfire, the National Multicultural Festival and Summernats. Canberra's main sporting venues are Canberra Stadium and Manuka Oval. The city is served with domestic and international flights at Canberra Airport, while interstate train and coach services depart from Canberra railway station and the Jolimont Centre respectively. City Interchange and Alinga Street station form the hub of Canberra's bus and light rail transport network.
+
+== Name ==
+The word "Canberra" is derived from the Ngunnawal language of a local Ngunnawal or Ngambri clan who resided in the area and were referred to by the early British colonists as either the Canberry, Kanberri or Nganbra tribe. Joshua John Moore, the first European land-owner in the region, named his operator "Canberry" in 1823 after these people. "Canberry Creek" and "Canberry" first appeared on regional maps from 1830, while the derivative name "Canberra" started to appear from around 1857. Other early recorded variants of the spelling include "Canbury" (potentially influenced by the settlement of the same name in England), "Canburry" and "Kembery".
+Numerous local commentators, including the Ngunnawal elder Don Bell, have speculated upon possible meanings of "Canberra" over the years. These include "meeting place", "woman's breasts" and "the hollow between a woman's breasts". References to breasts or the space between them are thought to derive from Black Mountain and Mount Ainslie, two large hills with similar elevations situated immediately to the northwest and northeast, respectively, of what is now the city centre.
+Alternative proposals for the name of the city during 
+…
+
+[Showing the region around 'capital'. The full document is 62,467 characters - call wikipedia_lookup with topic='Canberra', full_article=true and offset= to read more.]
+
+---
+Sources: https://en.wikipedia.org/wiki/Canberra
+
+=== [4] THINKING (0s) ===
+I have enough information. I'll answer concisely.
+
+=== [5] ASSISTANT TEXT ===
+The capital of Australia is **Canberra**.
+
+It isn't Sydney or Melbourne because of a political compromise made at federation in 1901:
+
+- **Sydney vs. Melbourne rivalry** — At the time of federation, Sydney (in New South Wales) and Melbourne (in Victoria) were the two largest cities, and each state fiercely wanted its own city to become the national capital. Neither would accept the other.
+- **A constitutional solution** — Section 125 of the Australian Constitution required that the seat of government be located within either New South Wales or a territory under federal control. As a fallback, it specified that if the capital wasn't in NSW, it would be a specially created territory on NSW land.
+- **A purpose-built compromise** — When neither city won the argument, the federal government selected a new site in south-eastern NSW, about halfway between the two rivals. The area was formally named Canberra in 1913 (from the local Ngunnawal name for the region), and it was designed from scratch by American architect Walter Burley Griffin after his plan won an international competition.
+
+So Canberra is unusual in being an entirely planned city — built specifically to be a neutral seat of government that belonged to neither NSW nor Victoria.
+
+---
+Sources: https://en.wikipedia.org/wiki/Canberra
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · first token after 5.1 s
+- Thought: Simple question. Canberra. The reason is well-known history, but should I verify it? … the guidance I should verify settled facts. One call is cheap.
+- Calling wikipedia_lookup
+- Thinking
+- Thought: I have enough information. I'll answer concisely.
+- Writing response
+- Done · 1 tool call · prefill 4.9 s for 12,749 tokens · 412 generated