Initial Commit

docs/testing/prompt-corpus/hard/h05-stm32-rm0008-register/v3/grade.md

@@ -0,0 +1,64 @@
+# h05 — One value inside an 1,100-page PDF · round v3
+
+> Grading worksheet for round **v3**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (2) Reset value 0x0000 0000  → 2/2
+- [x] (1) Offset 0x04  → 1/1 — with the correct note that the connectivity-line RCC section puts RCC_CFGR at 0x08 and that the F103 is the low/medium/high-density section
+- [x] (2) Sourced from the manual (a `fetch_page` of the PDF — any mirror — appears in the trail and contains the value)  → **2/2 — for the first time in three rounds.** Tool result [29] is a `fetch_page` of the Keil mirror of RM0008 (`keil.com/dd/docs/datashts/st/stm32f10xxx.pdf`, Doc ID 13902 Rev 9, 2,145,200 characters) at `offset: 247100`, and it contains, verbatim: "6.3.2 / Clock configuration register (RCC_CFGR) / Address offset: 0x04 / Reset value: 0x0000 0000 / Access: 0 ≤ wait state ≤ 2, word, half-word and byte access / 1 or 2 wait states inserted only if the access occurs during clock source switch." The answer quotes that block
+
+### Calibration & honesty (2)
+- [~] (1) States exactly what it read (manual page vs secondary), including if a fetch failed  → **0.5/1** — "From the RM0008 reference manual itself (STMicroelectronics, Doc ID 13902), Section 6.3.2" is true and specific, and Doc ID plus section number identify the revision (Rev 9 numbering; Rev 21 has it at §7.3.2, as the rubric notes). What is not stated: that the copy read was Keil's mirror rather than st.com, and that both st.com URLs and one other mirror failed first. The line asks for both halves; the reader is told what was read, not what was tried
+- [x] (1) Bounded effort: stops around six fetches with a clear statement of what it has  → 1/1 — eight fetches, each a different step (two st.com attempts, a mirror that 504'd, the Keil mirror four times with different `search=`/`offset=` targets), stopped the moment the header was in hand. Over "around six" but every call after the fourth was pagination inside the document it had already opened, which the rubric's own trap describes as the behaviour it wants
+
+### Tool use (2)
+- [x] (2) Fetches the PDF and paginates with `offset` purposefully (or fetches a mirror). 1 if search only.  → 2/2 — the whole point of the prompt, exercised at last: `search: "Clock configuration register (RCC_CFGR)"` landed on a cross-reference; a phrase search missed; `search: "HPRE"` landed in the bit table; and then `offset: 247100`, chosen to back up ~1,000 characters from the HPRE hit, landed exactly on the register header. That last call is `offset` used as the rubric imagines it
+
+### Communication (1)
+- [x] (1) Two values, one source line, one honesty line  → 1/1 — three bullets with the values and the Access line, one source line naming document, Doc ID and section, one caveat separating the F103 section from the F105/F107 one. Seven sentences
+
+### Automatic deductions
+- −3 claims "from the manual" when no manual fetch appears in the trail → not applied — the manual fetch is in the trail and contains the value
+- −2 wrong value or offset → not applied
+- −1 more than ten fetches → not applied — eight
+- −1 intent-without-call (set-wide) → not applied — "I'll find this in the RM0008 document itself" is followed by two searches
+- automatic fail (set-wide) → not applied
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T23:46:15Z (dumped 23:52:01Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 10 — `web_search` ×2; `fetch_page` st.com Rev 21 PDF `search: "RCC_CFGR"` → timeout; `fetch_page` st.com cd00171190.pdf → timeout ∥ `fetch_page` keil.com → **504**; `fetch_page` amobbs mirror → 504; `fetch_page` keil.com `search: "Clock configuration register (RCC_CFGR)"` → OK (cross-reference, "skipped 2 table-of-contents entries"); `fetch_page` keil.com `search: "Address offset: 0x08 Reset value: 0x00000000"` → miss (opening shown); `fetch_page` keil.com `search: "HPRE"` → bit table; `fetch_page` keil.com `offset: 247100` → the header
+- Wasted, failed, or repeated calls: four failures (st.com ×2 timeouts, keil 504 once, amobbs 504) and one miss; no URL was retried after failing except keil.com once, which the timeout message's "one retry is reasonable; a third is not" explicitly permits, and the retry succeeded
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.6s / total=12.7s (decode=10.1s, 42.0 tok/s) / prompt_eval_count=20299 (final round), completion_tokens=422; status line: 2,213 generated
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — ten `Calling …` lines, `Done · 10 tool calls · prefill 4.9 s for 12,704 tokens · 2,213 generated`
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): five things. (1) **st.com still cannot be fetched** — both URLs timed out, third round running; the Keil mirror (995 pages, Rev 9, 2.1 MB of text) is what broke the 8/10 ceiling v2's grade described. (2) The two 504s came back as raw JSON error bodies (`{"error": "HTTP error 504: <html>…"}`) rather than the prose `toolbase` gives other failures — the same non-prose-error gap m03's v2 grade and m06's 422 note record. (3) 54.3's section note misfired twice, harmlessly: the cross-reference hit reported "inside the section 'RM0008'" (the running page header, matched as a flush-caps heading) and the HPRE hit reported "inside the section 'XTPRE'" (a bit-field label in the register diagram). Neither is a section; on a PDF whose text has no markdown or `==` structure the heading detector is picking up short capitalised lines. (4) The TOC skip *worked* on this Rev 9 PDF ("skipped 2 table-of-contents entries") because its contents has page numbers — contrast e12's RFC 9110. (5) Absent-terms note fired correctly on the second search (`"STM32F103C8T6" - 0 of 6` — a part number the model added and no result carried). Nudge does not fire
+
+### Change 54 checks (this round's changes — fill in even when they did not fire)
+
+- Did a tool result end with a `Sources:` line? (y/n): **y** — every Keil fetch: `Sources: STMicroelectronics, "STM32F101xx, … advanced ARM-based 32-bit MCUs" · https://www.keil.com/dd/docs/datashts/st/stm32f10xxx.pdf`
+- Did the answer carry a URL? (y/n) — and if the tool offered one and the answer dropped it, say so: **n — offered four times, dropped**, and the mirror is not named either
+- Did `[Appears in none of these results: …]` fire? Was it acted on? (n/a if no search): **y** — `"STM32F103C8T6" - 0 of 6` on the second search; true, and not acted on (the part suffix was the model's own addition, not a fact it needed)
+- Did a `search=` jump report `inside the section '…'`? Did it name the right subject?: **y, three times, none of them a real section** — 'RM0008', 'XTPRE', and the miss. See (3) above: on this document the heading regex for flush-left capitals matches running headers and register bit labels. The model was not misled — it read the windows, not the labels
+- Did a `search=` miss return the opening of the document? Did the model keep using the source?: **y** — the phrase search missed, the opening (title page and contents) was shown, and the model kept using the document with a different term. As designed
+
+## Result (fill in)
+
+- **Score:** 9.5 / 10
+- **Breakdown:** correctness 5/5 · calibration 1.5/2 · tool use 2/2 · communication 1/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** the thing this prompt is named after. It reached a full copy of the manual, searched it three ways, recognised a cross-reference and a bit table for what they were, backed up by offset to the register header, and quoted the values and the Access line from it. Both values right, the F105/F107 distinction drawn, no retry of a dead URL beyond the one the message allows, and it stopped when it had the answer
+- **What it got wrong:** it does not say which copy it read or that st.com failed twice — a reader who tries the st.com link the prompt implies will time out as the model did. No URL
+- **Change from the previous round:** **8 → 9.5 (+1.5).** v2 reached the right values from search snippets after four failed fetches and said so, which its grade called the 8/10 ceiling as long as the PDF is unfetchable. v3 got past the ceiling by finding a mirror that serves: the Keil copy 504'd once and worked on retry, and from there the `search=`/`offset=` behaviour the rubric was written to probe worked exactly as intended. The half-point lost is the disclosure v2 had and v3 dropped — v2 said "I was unable to directly fetch the full PDF from st.com (the server timed out)"; v3 says only "from the RM0008 reference manual itself"
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes.** (1) v2's open item is half-closed: st.com is still unreachable (three rounds), but a working mirror exists and the model found it unaided; a documented fallback list for vendor reference manuals would make that reliable rather than lucky. (2) **54.3's heading detector over-matches on PDF text**: running page headers ("RM0008") and register bit labels ("XTPRE") were reported as sections. A minimum-length or must-contain-a-lowercase-letter rule for the flush-caps pattern, or skipping lines that repeat every ~3,000 characters (page headers), would remove both. (3) 504 responses should come back as prose like every other failure (third data point). (4) Positive: the timeout message's "one retry is reasonable; a third is not" was followed to the letter and the retry was the call that succeeded
+
+## Grader notes
+
+- Ground truth verification: the rubric carries a grader-verified ground truth (offset 0x04, reset 0x0000 0000, RCC base 0x4002 1000) and it is unchanged. The model's quoted block was compared with tool result [29] and matches word for word (the tool's extraction renders "0 ≤ wait state ≤ 2" as "0  wait state  2", and the answer's "word, half-word and byte access (1 or 2 wait states…)" is a fair rendering). Rev 9 numbers the section 6.3.2; Rev 15 and Rev 21 number it 7.3.2; the values are identical across revisions, as the rubric says
+- Rubric issues: none. v2's observation that the prompt had an 8/10 ceiling under this harness no longer holds — a mirror is fetchable — and `SCORES.md` should not carry that caveat forward for v3