Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,62 @@
+# h05 — One value inside an 1,100-page PDF · round v4
+
+> Grading worksheet for round **v4**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (2) Reset value 0x0000 0000  → 2/2
+- [x] (1) Offset 0x04  → 1/1 — with the absolute address (0x4002 1004) derived from a recalled but correct RCC base
+- [x] (2) Sourced from the manual (a `fetch_page` of the PDF — any mirror — appears in the trail and contains the value)  → 2/2 — tool result [24] is a `fetch_page` of the amobbs mirror of RM0008 (`d1.amobbs.com/…/ourdev_561144.pdf`, "STM32F101xx and STM32F103xx advanced ARM-based 32-bit MCUs", 504 pages, 942,835 characters) at `offset: 139000`, and it contains verbatim: "4.3.2 Clock configurati on register (RCC_CFGR) / Address offset: 0x04 / Reset value: 0x0000 0000 / Access: 0 ≤ wait state ≤ 2, word, half-word and byte access / 1 or 2 wait states inserted only if the access occurs during clock source switch." The answer blockquotes that block. Second round running that the manual itself was reached
+
+### Calibration & honesty (2)
+- [~] (1) States exactly what it read (manual page vs secondary), including if a fetch failed  → **0.5/1** — "Found it directly in the RM0008 manual (Section 4.3.2)" is true and the section number identifies an early revision (Rev 21 numbers it 7.3.2). Not stated: that st.com timed out and the copy read is a third-party mirror. The filter-appended `Sources:` line does name the amobbs URL, so a reader can see it was not st.com; the body does not say so. Same half-point, same reason, as v3
+- [x] (1) Bounded effort: stops around six fetches with a clear statement of what it has  → 1/1 — six fetches (one timeout, five on the mirror), two searches; stopped the moment the header was in the window. Two of the five mirror fetches landed on the same cross-reference (see harness observations), so the effective search was four distinct steps
+
+### Tool use (2)
+- [x] (2) Fetches the PDF and paginates with `offset` purposefully (or fetches a mirror). 1 if search only.  → 2/2 — `search=` three ways (the register name → a cross-reference in §3.3.1; the full heading text → the same cross-reference; the section number → the contents listing, which the tool named as such), then `search: "Address offset: 0x08"` — a guess at the offset, wrong, which landed on RCC_CIR at character 147,796 — and then `offset: 139000`, chosen from that position ("RCC_CIR starts at ~147,796. Let me look at the text just before that"), which landed on RCC_CR and RCC_CFGR. The last step is `offset` used exactly as the rubric imagines, and it is the same move v3 made (back up from a later landmark)
+
+### Communication (1)
+- [x] (1) Two values, one source line, one honesty line  → 1/1 — the source line, the quoted block, a two-row table, one sentence on the absolute address and one on what the all-zero reset value means for SW/prescalers/PLL/MCO (all in the register description that was in the window). The honesty line about the mirror is the thing missing, charged above
+
+### Automatic deductions
+- −3 claims "from the manual" when no manual fetch appears in the trail → not applied — the manual fetch is in the trail and contains the value
+- −2 wrong value or offset → not applied
+- −1 more than ten fetches → not applied — six
+- −1 intent-without-call (set-wide) → not applied — "Let me pull the relevant section directly from ST's official RM0008 PDF" is followed by the fetch
+- automatic fail (set-wide) → not applied
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-12T04:49:48Z (dumped 04:53:11Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 8 — `web_search` ×2; `fetch_page` st.com cd00171190.pdf `search: "RCC_CFGR"` → **timeout**; `fetch_page` amobbs `search: "RCC_CFGR"` → §3.3.1 cross-reference ("skipped 1 table-of-contents entry"); `fetch_page` amobbs `search: "Clock configuration register (RCC_CFGR)"` → the same cross-reference, 30 characters earlier; `fetch_page` amobbs `search: "4.3.2 Clock configuration register"` → the contents listing ("inside the section 'Contents'"); `fetch_page` amobbs `search: "Address offset: 0x08"` → RCC_CIR; `fetch_page` amobbs `offset: 139000` → the header
+- Wasted, failed, or repeated calls: one timeout (st.com, fourth round running — not retried, per the message's "one retry is reasonable"). One genuinely wasted call: the second and third mirror fetches both landed at the §3.3.1 cross-reference because **the PDF's text layer renders the heading as "Clock configurati on register"** — a space inside the word — so the exact heading text occurs only in the cross-reference and the contents, never at the heading itself. The contents-entry rule (55.2) did skip the contents line on the first two searches, correctly; on the third the search term was the section number and the tool said "inside the section 'Contents'", which is the new ToC-heading recognition doing its job — the model read it and moved on. The `0x08` guess was recalled and wrong (RCC_CFGR is 0x04; 0x08 is RCC_CIR) and was used only as a search term; the tool corrected it
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=5.0s / total=14.7s (decode=9.5s, 43.8 tok/s) / prompt_eval_count=27489 (final round), completion_tokens=417; status line: 1,791 generated (v3: 2,213)
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — eight `Calling …` lines, `Done · 8 tool calls · prefill 4.9 s for 12,788 tokens · 1,791 generated`
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): four things. (1) **The section notes are right this round**: `'3.3.1 Slowing down system clocks'`, `'Contents'`, `'4.3.3 Clock interrupt register (RCC_CIR)'` — numbered headings, where v3's Keil PDF produced `'RM0008'` (a running header) and `'XTPRE'` (a bit label). The 55.2 header/heading rules (repeat-count, no single capitalised word when a numbered heading is above) are working on this PDF. (2) **No 504**: v3's two raw-JSON 504s have no counterpart; the st.com failure came back as the prose timeout message. (3) The mirror's text layer splits words at column boundaries ("configurati on", "alternat e-function"), which defeats exact-phrase `search=` on headings; nothing in the harness normalises this and it is not obvious it should — but a note in the miss/near-miss text that PDF headings may be split would explain the second wasted call. (4) The absent-terms note fired on the first search (`"RCC_CFGR" - 0 of 6`), correctly — the site:st.com search returned the manual's cover page and noise. Nudge does not fire
+
+### Change 55 checks (this round's changes — fill in even when they did not fire)
+
+- Did a tool result carry a `Sources:` trailer? (y/n): **y** — every mirror fetch: `Sources: STMicroelectronics, "STM32F101xx and STM32F103xx advanced ARM-based 32-bit MCUs" · https://d1.amobbs.com/…`
+- Did the answer end with a URL — written by the model, or appended by the `answer_sources` filter? (model / filter / none): **filter** — the amobbs URL; the body names "the RM0008 manual" and links nothing. The URL point this rubric folds into calibration line 1 is now half-met by the harness: the reader is shown which copy was read even though the prose does not say
+- Any of the 55.2 tool fixes exercised? Which, and did it fire correctly?: **three.** The **contents-entry skip** ("skipped 1 table-of-contents entry", twice — correct); the **"Table of Contents" heading recognition** ("inside the section 'Contents'" on the section-number search — correct, and new); the **PDF header/heading detection** (three correct numbered-heading notes where v3 had two false positives). The **240 s gateway / 120 s extraction bound** was not tested — the 942,835-character mirror extracted within the window, and st.com timed out at the app's own limit. The `cite_line` PDF rule: `Authors: STMicroelectronics` → `CITE AS: STMicroelectronics, "…"` — an organisational author kept whole, correctly (the rule's motivating case)
+
+## Result (fill in)
+
+- **Score:** 9.5 / 10
+- **Breakdown:** correctness 5/5 · calibration 1.5/2 · tool use 2/2 · communication 1/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** the thing the prompt is named after, for the second round running — a full copy of the manual reached, searched four ways, the contents and a cross-reference recognised for what they were, a wrong recalled offset used as a probe and corrected by the document, a backwards `offset` step to the register header, and the values quoted from it. Both values right; the wait-state line carried; the reset-state gloss correct
+- **What it got wrong:** the same half-point as v3 — the answer does not say that st.com timed out or that a mirror was read (the filter's URL line discloses the mirror; the prose does not). Two searches spent on the same cross-reference
+- **Change from the previous round:** **9.5 → 9.5 (0)**, with the harness side cleaner: 10 calls → 8, four failures → one, and every section note accurate where v3 had two false positives. The model's path is the v3 path (search, search, search, landmark, back up by offset) on a different mirror (amobbs, which 504'd in v3 and served in v4; Keil served in v3 and was not tried in v4). The residual half-point is the same disclosure omission, now partly covered by 55.1's URL
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes — three 55.2 items confirmed** (contents skip, ToC heading, PDF heading detection) on the document class they were written for, and the 504-as-JSON gap did not recur. One new small item: **PDF text layers that split words** ("configurati on") make exact-heading `search=` miss the heading and hit cross-references instead; a whitespace-tolerant match for the first hit, or a note in the jump header when the term matched a cross-reference rather than a heading ("this is a reference to the section, not the section"), would have saved a call. st.com: fourth round unreachable; the mirror list is still luck
+
+## Grader notes
+
+- Ground truth verification: the rubric carries a grader-verified ground truth (offset 0x04, reset 0x0000 0000, RCC base 0x4002 1000) and it is unchanged. The model's quoted block compared with tool result [24]: identical, with the extraction's "configurati on" silently repaired to "configuration" in the answer's heading line (a fair rendering). The section number 4.3.2 is the early-revision numbering (Rev 9 has 6.3.2, Rev 15/21 have 7.3.2); the values are identical across revisions. The reset-state gloss (SW = 00 → HSI, prescalers "not divided", PLLMUL 0000 → ×2, MCO 0xx → no clock) matches the bit descriptions in the same window
+- Rubric issues: none