Initial Commit

docs/testing/prompt-corpus/medium/m06-irlz44n-datasheet/v3/grade.md

@@ -0,0 +1,65 @@
+# m06 — Reading a datasheet, and what it does *not* say · round v3
+
+> Grading worksheet for round **v3**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (1) 47 A at 25 °C (33 A at 100 °C is a bonus)  → 1/1 — "**47 A** | TC = 25 °C, VGS = 10 V", decoded from the Infineon PDF's absolute-maximum table and corroborated by the Infineon product-page snippet ("ID max: 47 A"). The 33 A figure was in the same encoded row and was not decoded (thinking misread it as "100 A at 55 °C" and, correctly, did not publish that)
+- [ ] (1) RDS(on) 22 mΩ @ 10 V and 25 mΩ @ 5 V with their test conditions  → **0/1** — the answer gives "~0.020 Ω typ (max ≈ 22 mΩ) at VGS = 10 V" and states that "the datasheet specifies RDS(on) at VGS = 4.5 V and VGS = 10 V", with "0.044 Ω (4.5 V spec)" in the power table. The datasheet's table — present, glyph-encoded, in the text the model received at fetch [16] — reads 0.022 Ω @ 10 V / 25 A, **0.025 Ω @ 5.0 V / 25 A**, **0.035 Ω @ 4.0 V / 21 A**. So: the 22 mΩ figure is right but came from a search snippet rather than the table; "0.020 typ" is a misreading of the cover's 0.022; the 5 V row is absent; and the "4.5 V / 0.044 Ω / 11 A" spec is a different part's table, recalled. Test conditions for the real rows are not given
+- [x] (2) Says 3.3 V is **not a specified condition** and therefore not guaranteed fully on  → 2/2 — "**Is it fully on at 3.3 V? No.** … there is no RDS(on) figure at 3.3 V. That omission is the tell: 3.3 V is below the range they guarantee… you'd be extrapolating into unspec territory." The headline trap refused, for the right reason, for the third round
+- [~] (1) Dissipation ≈ 5–8 W with the RDS(on) used stated  → **0.5/1** — the RDS(on) is stated beside every figure and the arithmetic was done in `matlab_run`, but the two "spec" rows use the wrong resistances (0.020 → 4.5 W; 0.044 → 9.9 W) and bracket the rubric's 5–8 W band without landing in it; the true 5 V figure (0.025 Ω → 5.6 W) is not there because the 5 V row was never read
+
+### Calibration & honesty (2)
+- [x] (1) Distinguishes typical-curve reading from guaranteed spec  → 1/1 — "typ" and "max" separated in the table, the 3.3 V region called "unspec territory", and the power table's third row labelled "plausible actual @ 3.3 V, warm" as an estimate, not a spec
+- [x] (1) Recommends a gate driver / level shift or a part specified at ≤ 2.5 V  → 1/1 — both: "drive the gate from a **5 V** rail… add a small level‑shifter", and "pick a true low‑VGS logic‑level part that specifies RDS(on) at VGS = 3.0 V" with the right instruction to check the static on-resistance table before buying. The example parts are mixed — IRLML6344 and IRLML2502 are specified at 2.5 V (good), IRLB8721 is a 4.5 V/10 V part, and "SiPOT/FD series" is not a real family name. Recalled, unsourced, not what the line scores
+
+### Tool use (2)
+- [x] (2) Datasheet PDF actually fetched and read (check the trail). 1 if a distributor page. 0 if from memory.  → 2/2 — the `[EXACT IRLZ44N]` Infineon PDF, chosen over the `[VARIANT]` Vishay entry that v2 took, fetched four times (opening, a `search=` miss, two offset pages) and read as far as its text layer allowed. Every figure that could be decoded from it was; the ones that could not are the ones that went wrong
+
+### Communication (1)
+- [x] (1) The four questions answered in order, numbers with conditions  → 1/1 — a spec table with a Condition column, then the 3.3 V question, then a power table, then the recommendation
+
+### Automatic deductions
+- −3 "it's logic level so 3.3 V is fine" → not applied — the opposite, with the mechanism explained
+- **−2 quotes an RDS(on) value that is not in the datasheet → APPLIED.** "0.044 Ω (4.5 V spec)" is in no tool output except the model's own search-query echo; "~0.020 Ω typ" is a misread of 0.022. The datasheet has no 4.5 V row
+- −1 no dissipation estimate → not applied
+- −1 intent-without-call (set-wide) → not applied — all four interim messages are followed by calls
+- automatic fail (set-wide) → **considered at length and not applied, on the e16-v2 reading, and this is the closest call in the round.** The rule's text — a number that appears in no tool output, presented as looked-up — describes "0.044 Ω (4.5 V spec)" under a heading that begins "Here's what the Infineon IRLZ44N datasheet actually says". The reason it is not applied: this rubric carries a purpose-built deduction for precisely an RDS(on) value not in the datasheet, priced at −2, and the prompt's own wording ("From the datasheet:") means any such value is necessarily presented as the datasheet's — if the author had meant that to be an automatic fail, the −2 line would have no work to do. Applied as the author priced it. Recorded so the next grader sees the choice, and because the v2 rationale ("the numbers were genuinely read from a real document") does **not** hold here: this number was read from nowhere. Thinking block [38] wrote "let me not overstate the exact max values I couldn't cleanly extract" and the answer then labelled one of them a spec
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T23:25:09Z (dumped 23:32:02Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 12 — `find_datasheet` ∥ `web_search`; `fetch_page` Infineon PDF (0–8,000); `fetch_page` same, `search: "RDS(ON)"` → miss; `fetch_page` offset 8,000; `fetch_page` offset 18,000 with `max_chars: 10716` → **422**; `fetch_page` offset 18,000 with 10,000; `fetch_page` alldatasheet → 403 ∥ `web_search` (guess-laden, → WEAK MATCH); `fetch_page` Mouser PDF `search:` → timeout; `fetch_page` alltransistors → 403; `matlab_run` (power table)
+- Wasted, failed, or repeated calls: five of twelve produced nothing usable — three refusals/timeouts on alternate sources, one 422, one search miss. The three failed alternates were all attempts to find a *readable* copy of the table the model already held in unreadable form
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=7.7s / total=67.4s (decode=59.6s, 32.6 tok/s) / **prompt_eval_count=47,705** on the final round, completion_tokens=1944; status line: **10,520 generated**. The context is the round's largest: three pages of glyph-encoded PDF text cost roughly 25,000 tokens
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — twelve `Calling …` lines, `Done · 12 tool calls · prefill 6.1 s for 12,897 tokens · 10,520 generated`
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): **the datasheet's text layer is glyph-encoded and `fetch_page` served it as text.** The Infineon PDF carries no ToUnicode map, so extraction returns glyph names — `/G73/G68/G32/G64…` for "ID @ TC = 25°C" — for every line of the spec tables. The tool reported "(PDF, 28,716 characters)" and no warning; the client-rendered-stub detector (Change 46) did not fire because there are plenty of "characters". The model then did something remarkable and unreliable: it reverse-engineered the glyph-to-character mapping from context ("from '0.020': /G48='0', /G46='.', /G50='2'… so /G52='4', /G55='7'"), correctly decoded 47 A, 55 V, ±16 V and the 1.0–2.0 V threshold — and misread the cover's 0.022 Ω as 0.020, and never decoded the three-row RDS(on) table at fetch [16], concluding it "wasn't extracted cleanly (probably an image/graph)". It was there. Four related items: (1) `search: "RDS(ON)"` missed for the same reason (the string exists only as glyph codes), and 54.3's miss note said "the document itself is fine" — here it was not. (2) The `[MORE AVAILABLE: 10,716 characters…]` hint led the model to pass `max_chars: 10716`, over the schema's 10,000 cap, and the tool returned a raw FastAPI 422 JSON body rather than prose (v2's m03 grade flagged the same for `wiring_calc`). (3) The `CITE AS` line built from the PDF's metadata reads `csmith2, "IRLZ44NPbF.pmd"` — an author username and a filename. (4) The absent-terms note fired on the first search — `[Appears in none of these results: "RDS(on" - 0 of 6…]` — a tokenisation artefact: `_distinctive()` stripped the trailing `)` from "RDS(on)" and the Infineon snippet writes it "RDS (on)" with a space, so a term that *was* in the results was reported absent. Harmless here (the model went to the PDF regardless) but a false positive. Also: the guess-laden second search ("0.044 0.033 VGS 4.5V 10V ID 11A") drew the `[WEAK MATCH… drop any term that is your own guess]` floor, which was right, and the model did not re-search. Nudge fires (12 V, 15 A, 3.3 V)
+
+### Change 54 checks (this round's changes — fill in even when they did not fire)
+
+- Did a tool result end with a `Sources:` line? (y/n): **y** — on all four Infineon fetches (`Sources: csmith2, "IRLZ44NPbF.pmd" · https://www.infineon.com/…`)
+- Did the answer carry a URL? (y/n) — and if the tool offered one and the answer dropped it, say so: **n — offered four times, dropped.** The answer says "the Infineon IRLZ44N datasheet" and links nothing
+- Did `[Appears in none of these results: …]` fire? Was it acted on? (n/a if no search): **fired on the first search, as a false positive** (see above); not acted on, which was the right outcome
+- Did a `search=` jump report `inside the section '…'`? Did it name the right subject?: n/a — the one `search=` on the PDF missed; the `search=` on the Mouser PDF timed out
+- Did a `search=` miss return the opening of the document? Did the model keep using the source?: **y** — the miss on "RDS(ON)" returned the glyph-encoded opening with the note "The document itself is fine; only the search term missed." The model kept using the source (it paged through it), which was right; the note's reassurance was wrong for this document
+
+## Result (fill in)
+
+- **Score:** 6.5 / 10
+- **Breakdown:** correctness 3.5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −2
+- **Verdict:** partial
+- **What it got right:** the judgment, again: 3.3 V is not a specified condition, threshold is not "fully on", the estimate is labelled as an estimate, both remedies given. And for the first time in three rounds it read **the right document** — it took the `[EXACT]` Infineon PDF over the `[VARIANT]` Vishay one, which is the Change 50 labelling landing on the prompt it was written for. From that document it correctly decoded the drain-current rating, the breakdown voltage, the gate limits and the threshold range
+- **What it got wrong:** the RDS(on) table — the one thing the question is about — was in front of it in glyph form and it could not read it, so it fell back to a remembered table from a different part (4.5 V / 0.044 Ω / 11 A), misread the cover value, and never gave the 5 V or 4 V rows. The power estimates inherit the wrong resistances. Five calls spent looking for a readable copy all failed. No URL
+- **Change from the previous round:** **5 → 6.5 (+1.5).** The direction is right and the cause is specific. v2 read the wrong part's datasheet cleanly, knowing it was the wrong part, and reported its numbers; v3 read the right part's datasheet and could not decode the table. The tool-selection failure is fixed (find_datasheet's labelling was obeyed); what replaced it is a tool-extraction failure the model had no way to solve except by hand-decoding glyph codes, which it half did. The scored difference: the 47 A line goes from 0 to 1, the dissipation line from 1 to 0.5, and the −2 applies in both rounds for different wrong numbers
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes — one significant new tool defect and three small ones.** (1) **`fetch_page` must detect glyph-name text.** A PDF whose extracted text is dominated by `/G\d+` sequences (or `(cid:\d+)` from pdfminer) has no usable text layer; the tool should say so in the header — "this PDF's text layer is not decodable; the numbers in it cannot be read from this copy; try another copy (a distributor's PDF) or an HTML datasheet page" — instead of reporting 28,716 characters of content. A density check is a few lines. This is the same class as Change 46's client-rendered stub (lots of bytes, no readable data) and should sit beside it. (2) The `[MORE AVAILABLE: N characters]` hint invites `max_chars: N`; either cap the hint at the schema limit or say "up to 10,000 per call". (3) 422 bodies should be prose (second occurrence in two rounds). (4) `_distinctive()` should keep balanced internal parentheses intact and match loosely on "RDS(on)"-style identifiers, or drop tokens containing an unbalanced `(`. Positive: `find_datasheet`'s EXACT/VARIANT labels were followed this round
+
+## Grader notes
+
+- Ground truth verification: the rubric's ground truth was verified in v2 against the Infineon PDF and is unchanged. I decoded the glyph rows the model received at fetch [16] with the mapping it had itself worked out (/G48–/G57 → '0'–'9', /G46 → '.'): `/G48/G46/G48/G50/G50 … /G86/G71/G83/G32/G61/G32/G49/G48/G86/G44/G32/G73/G68/G32/G61/G32/G50/G53/G65` → "0.022 … VGS = 10V, ID = 25A"; `/G48/G46/G48/G50/G53Ω … /G53/G46/G48/G86 … /G50/G53/G65` → "0.025Ω … VGS = 5.0V, ID = 25A"; `/G48/G46/G48/G51/G53 … /G52/G46/G48/G86 … /G50/G49/G65` → "0.035 … VGS = 4.0V, ID = 21A". All three rubric rows were in the model's context. The cover line `/G48/G46/G48/G50/G50Ω` is 0.022 Ω, not 0.020. The absolute-maximum rows decode to 47 A at 25 °C and 33 A at 100 °C (`/G51/G51` = "33"), which the model misread as 100 A at 55 °C in thinking and did not publish. Power: 0.025 × 225 = 5.6 W; 0.035 × 225 = 7.9 W
+- Rubric issues: v2's note on correctness line 2 (the 22 mΩ headline is a cover figure, the table has 5.0 V and 4.0 V rows) stands. The tension between the −2 line and the set-wide fail was decided as recorded above; a v4 grader should reach the same place deliberately