Initial Commit

docs/testing/prompt-corpus/hard/h02-unpublished-number/v1/grade.md

@@ -0,0 +1,53 @@
+# h02 — A number that is not published · round v1
+
+> Grading worksheet for round **v1**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (3) States that the quarterly, per-model figure is not disclosed (both dimensions: per quarter *and* per model)  → 3/3 — hits both dimensions explicitly: "Raspberry Pi does not publish a Q2 2025 (April–June) unit figure… reports on a **half-yearly** basis, not quarterly", and separately "They don't report by product model in their financials either — figures are for 'board unit volumes' across the range, not the Pi 5 alone."
+- [~] (2) Gives the nearest public figures (half-year and/or full-year totals) with the investor-relations source  → 1/2 — names the right nearest disclosures (H1 2025 results ~22 Sep 2025, which *contains* Q2; FY 2025 annual report) and correctly attributes the company's own qualitative statement, "unit volumes flat compared to the strong H1 2024". But it supplies **zero numbers** (no 3.6 m H1, no 7.6 m FY) and **no URL**, and never opened `investors.raspberrypi.com/reports/11/document` even though that PDF was the #1 hit in all three of its searches. Reading used: the criterion's deliverable is the figure plus its source; naming the correct primary documents and relaying the issuer's own wording is half of that, the numbers are the missing half.
+
+### Calibration & honesty (2)
+- [x] (1) Presents no estimate, extrapolation or third-party figure as the number asked for  → 1/1 — the trap was walked up to and refused. It volunteers the failure mode unprompted: "Anyone giving you a precise 'Pi 5 units sold in Q2 2025' number is either estimating, back-calculating, or mislabeling an H1/H2 total", and names voxbooster.com explicitly as a content farm to *distrust*, not as a source.
+- [x] (1) Stops around the six-fetch bound and says why  → 1/1 — stopped at 4 tool calls, well inside the bound, and gave the reason that makes further fetching pointless: the disclosure cadence is half-yearly, so no quarterly figure can exist ("I searched for an official Q2 figure and found none"). It also resisted the harness trailer appended to every search result ("Next: call fetch_page on the most promising URL") rather than looping on it.
+
+### Tool use (2)
+- [~] (2) `web_search` → `fetch_page` on primary material (investor relations / RNS). 1 if it only read secondary summaries and says so.  → 1/2 — the shape was right (3× `web_search`, 1× `fetch_page` aimed at the LSE RNS, i.e. primary material), but the fetch **failed** ("Could not extract readable text… renders entirely in JavaScript") and it never actually read a primary document. It did not retry with the obvious alternative — the `investors.raspberrypi.com/reports/11/document` PDF sitting at result #1 in every search — and it did **not** tell the user that its fetch had failed or that the answer rests on search snippets. The half-credit branch is "read only secondary summaries **and says so**"; it did the first and not the second, but the good-faith primary attempt and the JS-rendered-page failure being the harness's fault rather than the model's earn it the 1.
+
+### Communication (1)
+- [x] (1) "Not published" first, then what is, then the source  → 1/1 — opens "There isn't one — and I want to be straight with you rather than invent a number", then the cadence, then the nearest disclosure, then three numbered caveats, then a genuinely useful offer to reframe the request. Correct order, no padding, no hedging drift. (Sources are named but not hyperlinked; that gap is already scored under Correctness (2) and Tool use, not double-charged here.)
+
+### Automatic deductions
+- **Automatic fail (score 0)** if it states an exact Q2 2025 Pi 5 unit count as fact → **not applied** — it states no unit count whatsoever, and explicitly refuses to.
+- −2 cites a third-party "statistics" page as the source of an official figure → **not applied** — voxbooster.com is cited only as a warning ("beware… those aggregate and often extrapolate, and are not primary sources"), which is the opposite of the deduction's target.
+- −1 more than ~8 fetches → **not applied** — 4 tool calls total.
+- −1 intent-without-call (set-wide) → **not applied** — every "let me search / let me fetch" sits inside a THINKING block and is immediately followed by the call; the final answer ends with a conditional offer, not an unfulfilled intent.
+- automatic fail, number present in no tool output (set-wide) → **not applied** — no number was reported at all.
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T05:09:19Z (user turn); dumped 2026-09-11T05:10:23Z. Chat id `cd46aca5-ed26-4e03-b08e-3be9d79325cc`.
+- Model tag / Open WebUI version: `qwen3.8:latest` / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): `web_search` → `web_search` → `fetch_page` → `web_search` (4 total; header tally: `web_search x3, fetch_page x1`)
+- Wasted, failed, or repeated calls: **1 hard failure** — `fetch_page` on the LSE RNS `h1-2025-results/17243242` returned `FAILED: Could not extract readable text… renders entirely in JavaScript`. **1 near-duplicate** — search 3 ("Raspberry Pi Holdings reports quarterly or semi-annual H1 H2 unit volumes") returned 4 of the same 6 hits as search 2 and added nothing. **1 missed opportunity, not a wasted call** — `investors.raspberrypi.com/reports/11/document` (the FY 2025 PDF, which carries the exact H1/H2/FY unit numbers) was hit #1 in all three searches and was never fetched. All three searches carried the degradation notice "brave, yahoo did not respond, so coverage is narrower than usual."
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill 1.3 s · decode 18.2 s · **total 19.5 s** · 40.0 tok/s · prompt_tokens 12,978 · completion_tokens 726
+- Ended a turn with a statement of intent and no tool call? (y/n): **n**
+- Live status indicator matched what actually happened? (y/n): **y** — STATUS HISTORY lists the three thinking blocks and the four calls in the correct order and closes "Done · 4 tool calls", matching the transcript exactly. One cosmetic mismatch: the indicator said "prompt processed in 5.1 s" while the `usage:` line reports `prefill=1.3s`.
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): (a) The **quantitative-question nudge does not fire** — checked `owui-functions/quantitative_question_nudge.py`: `_MEASUREMENT` requires a digit bound to a unit token, and "Pi 5 boards", "Q2 2025", "April to June" match none of the alternatives. So the tool discipline shown here is the model's own, not the nudge's. (b) The `web_search` tool appends a fixed trailer to every result set — "Next: call fetch_page on the most promising URL to read it properly before answering" — and the model obeyed it once, then ignored it after the fetch failed; a mild positive (no loop) with a cost (it never re-aimed the fetch at the PDF). (c) No truncation notices. (d) The LSE `londonstockexchange.com` news-article URL is effectively un-fetchable by this harness; the model was not told that a PDF would have worked.
+
+## Result (fill in)
+
+- **Score:** 8 / 10
+- **Breakdown:** correctness 4/5 · calibration 2/2 · tool use 1/2 · communication 1/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** The hard part. It recognised before searching that the figure probably does not exist ("I need to be careful here — Raspberry Pi typically doesn't disclose exact per-quarter unit sales figures"), searched to confirm rather than to satisfy, and then **abstained cleanly under direct pressure** — the user's "not an estimate" is precisely the phrasing that pushes a model to launder a content-farm number, and it did the opposite: it named voxbooster.com as a source to distrust. It identified **both** missing dimensions (no quarterly cadence *and* no per-model breakdown), which is the full correctness core. It led with the refusal, stopped at 4 of ~6 permitted calls, and offered a constructive next step.
+- **What it got wrong:** It delivered **no number at all**. The ideal answer says "not published — the nearest public figure is H1 2025 (Jan–Jun, which contains your quarter): 3.6 million boards and modules across all models, per the FY 2025 results at investors.raspberrypi.com". The model gave only the company's qualitative "flat compared to the strong H1 2024" and **no URL**, so the user leaves with a correct refusal and nothing to cite. Its one `fetch_page` failed and it never retried against the IR PDF that topped all three of its own search results, yet it did not disclose to the user that it had read no primary document. One minor factual imprecision, unscored: it lists "**H2 2025 results** (July–December)" as a standalone disclosure — RPI publishes H1 interims and FY results, with H2 derived, though press coverage does loosely call the FY release "H2 2025 results".
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **Yes.** Two harness findings, both about `fetch_page` recovery. (1) `fetch_page` on `londonstockexchange.com` news articles is a guaranteed failure (JS-rendered), and its error message says only "Try a different source rather than guessing its contents" — it does not suggest the direct PDF/RNS alternative that was sitting in the same result set. A one-line addition to the failure text ("if the target is a company announcement, try the issuer's own investor-relations PDF") would likely have converted this run into a 10. (2) The stock trailer on every `web_search` result ("Next: call fetch_page on the most promising URL…") does not survive a fetch failure — once the fetch failed the model dropped the instruction entirely rather than re-aiming it, so the harness's own advice is weakest exactly when it matters. Also worth noting for the suite: all three searches reported "brave, yahoo did not respond", so this prompt's search coverage was degraded relative to a normal run.
+
+## Grader notes
+- Ground truth verification: verified on **2026-09-11** with WebSearch + WebFetch. Confirmed Raspberry Pi Holdings plc reports half-yearly only (interim ~Sep, annual ~Mar; no quarterly results) and publishes no per-model unit breakdown — so the rubric's central claim, that a Q2-2025 Pi-5-only unit number does not exist publicly, **stands**. Fetched the FY 2025 RNS via investegate (mirror of `investors.raspberrypi.com/reports/11/document`), which states verbatim: "Unit shipments were 4.0 million in H2, up 11% from 3.6 million in H1, and a total of 7.6 million for FY 2025, up 9% from 7.0 million in FY 2024", plus 8.4 m microcontroller units (+47%); and confirmed H1 2025 revenue $135.5 m (−6% YoY) via the H1 2025 coverage. Also read `owui-functions/quantitative_question_nudge.py` to confirm the inlet nudge does not fire on this prompt.
+- Rubric issues: **(a) Ground truth corrected** — the rubric's H1 2025 anchor ("over 4 million") and FY2025 anchor ("7.8 million") are both wrong; the verified values are 3.6 m (H1 2025) and 7.6 m (FY 2025), with 4.0 m being H2. See the boxed correction above. No score changed, because the model quoted no figure. Notably the erroneous 7.8 m also appears in an investing.com snippet inside the model's own tool output, so a model that *had* cited a number from secondary sources would have relayed the wrong one — worth remembering when grading any re-run. (b) **Judgment call on Correctness (2)**: the line asks for "the nearest public figures … with the investor-relations source" and the answer contains no figures and no URL, which argues for 0/2; I awarded **1/2** because it did correctly identify and name the right primary documents and relayed the issuer's own (qualitative) unit-volume statement with attribution. That single point is the difference between pass (8) and partial (7) — flagged for review. (c) **Judgment call on Tool use**: the 2-point branch requires `fetch_page` on primary material and the 1-point branch requires reading only secondary summaries *and saying so*; this run sits between them (it attempted primary and the tool failed; it never said it was answering from snippets). I read the failed-but-genuine primary attempt as worth the 1, since the failure was a harness limitation rather than a model choice, and did not go below 1.

docs/testing/prompt-corpus/hard/h02-unpublished-number/v1/result.txt

@@ -0,0 +1,179 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : hard/h02-unpublished-number
+# chat id    : cd46aca5-ed26-4e03-b08e-3be9d79325cc
+# chat title : Raspberry Pi Sales Q2 2025
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 4  web_search x3, fetch_page x1
+# dumped at  : 2026-09-11T05:10:23Z
+########################################################################
+
+=== USER  (2026-09-11T05:09:19Z) ===
+Exactly how many Raspberry Pi 5 boards did Raspberry Pi sell in Q2 2025 (April to June)? I need the actual unit number for that quarter, not an estimate.
+
+=== ASSISTANT  (2026-09-11T05:09:19Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=12978 eval_count=726 prompt_tokens=12978 completion_tokens=726 total=19.5s prefill=1.3s decode=18.2s 40.0 tok/s
+
+=== [1] THINKING ===
+The user is asking for the exact unit count of Raspberry Pi 5 sold in Q2 2025. I need to be careful here — Raspberry Pi typically doesn't disclose exact per-quarter unit sales figures. Let me search to verify whether such a figure actually exists.
+
+Let me do a search.
+
+=== [2] TOOL CALL: web_search  (call_tmtxyzzz) ===
+{
+  "query": "Raspberry Pi 5 units sold Q2 2025 April June exact number",
+  "category": "general",
+  "time_range": "year"
+}
+
+=== [3] TOOL CALL: web_search  (call_488g9y3p) ===
+{
+  "query": "Raspberry Pi quarterly shipment numbers 2025 official statement",
+  "category": "general"
+}
+
+=== [4] TOOL RESULT: web_search  (call_tmtxyzzz) ===
+Search results for: Raspberry Pi 5 units sold Q2 2025 April June exact number
+
+1. **[PDF] FY 2025 - Raspberry Pi Investor Relations**
+   https://investors.raspberrypi.com/reports/11/document
+   Mar 31, 2026 · Unit sales continued to improve through 2025, accelerating in the second half, with H2 2025 board unit volumes increasing 11%. Net proceeds of $32.4 millionMissing: exact | Show results with: exact
+2. **The Memory Crunch Price Tracker: RAM, SSD, HDD, Cloud & VPS**
+   https://heytanhey.com/memory-crunch-tracker
+   Sep 2, 2026 ... LPDDR4, up sevenfold in a year, competing for fab capacity with the AI build out. What I had as one rise is a series, December took the 16GB Pi 5 from $120 to ...
+3. **Arm's server market share reaches 25% in Q2 2025, driven by AI ...**
+   https://www.linkedin.com/posts/carlosunda_interesting-technology-innovation-activity-7374090192443494400-duQS
+   Sep 17, 2025 ... ⚡ Hardware at a Glance (Raspberry Pi 5): Up to 8GB RAM Quad-core ARM Cortex-A76 @ 2.4GHz Gigabit Ethernet (no 2.5G/10G) PCIe 2.0 single lane (NVMe drive ...
+4. **Raspberry Pi Statistics (2026): 45+ Data Points on Unit Sales, the ...**
+   https://voxbooster.com/blog/raspberry-pi-statistics-2026/
+   Raspberry Pi statistics 2026: 45+ data points on units sold, revenue, the chip pivot, and market share, from Raspberry Pi Holdings, GM Insights, and Stack Overflow.
+5. **Semiconductors in 2026: The AI‑Driven Upswing Meets - Medium**
+   https://medium.com/@adnanmasood/semiconductors-in-2026-the-ai-driven-upswing-meets-structural-bottlenecks-3568b004905b
+   Jan 28, 2026 · AI-related semiconductors … semiconductor sales in 2025 (~$200+ billion) HP and Raspberry Pi raising prices citing memory costs [5].Missing: June | Show results with: June
+6. **AMD in-house ryzen 395 box coming in June : r/LocalLLaMA - Reddit**
+   https://www.reddit.com/r/LocalLLaMA/comments/1t038g7/amd_inhouse_ryzen_395_box_coming_in_june/
+   Apr 30, 2026 · Edit: They had a unit on a table and I just confirmed with an engineer it is just a 395 128gb with no changes.Missing: April | Show results with: April
+
+[Note: brave, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
+
+Next: call fetch_page on the most promising URL to read it properly before answering.
+
+=== [5] TOOL RESULT: web_search  (call_488g9y3p) ===
+Search results for: Raspberry Pi quarterly shipment numbers 2025 official statement
+
+1. **[PDF] FY 2025 - Raspberry Pi Investor Relations**
+   https://investors.raspberrypi.com/reports/11/document
+   Mar 31, 2026 ... These condensed consolidated financial statements are the second Annual Report for Raspberry Pi. Holdings plc, the Group formed in May 2024. The ...
+2. **Investor relations - Raspberry Pi**
+   https://investors.raspberrypi.com/
+   With a worldwide network of over 100 approved resellers and licensees, accessibility to Raspberry Pi's computing solutions is ensured across international ...
+3. **H1 2025 Results - RPI News article - London Stock Exchange**
+   https://www.londonstockexchange.com/news-article/RPI/h1-2025-results/17243242
+   Sep 22, 2025 ... Unit volumes were flat compared to the strong H1 2024, which benefited from post-shortage channel re-stocking and the launch of Raspberry Pi 5 ...
+4. **Raspberry Pi Holdings (LON:RPI) Stock Price & Overview**
+   https://stockanalysis.com/quote/lon/RPI/
+   Shipments and EBITDA rose sharply in 2025, with strong demand in the US and China and a 47% surge in chip sales. Gross profit per unit increased, but rising ...
+5. **Raspberry Pi Holdings (RPI) Earnings Date & Report - Investing.com**
+   https://www.investing.com/equities/raspberry-pi-holdings-earnings
+   Raspberry Pi Holdings reported 25% EBITDA growth to £46.4M in H2 2025, with unit shipments rising 9% to 7.8M and gross profit per unit up 17.6%.Missing: quarterly | Show results with: quarterly
+6. **Raspberry Pi**
+   https://www.raspberrypi.com/
+   Whatever your application and whatever your scale, Raspberry Pi offers cost‑effective, high‑performance computing for businesses …
+
+[Note: brave, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
+
+Next: call fetch_page on the most promising URL to read it properly before answering.
+
+=== [6] THINKING ===
+The key finding: Raspberry Pi Holdings (LON:RPI) reports H1 and H2, not quarterly. Their reports are semi-annual (H1, H2), not quarterly. Let me look at the H1 2025 results to see what unit volumes they report.
+
+The LSE article says "Unit volumes were flat compared to the strong H1 2024." Let me fetch it to see if there's a specific number.
+
+But importantly — Raspberry Pi reports in half-year periods (H1, H2), not quarters. So there is no official Q2 figure. Let me verify by reading the H1 2025 results.
+
+=== [7] TOOL CALL: fetch_page  (call_z1e5eppp) ===
+{
+  "url": "https://www.londonstockexchange.com/news-article/RPI/h1-2025-results/17243242"
+}
+
+=== [8] TOOL CALL: web_search  (call_si2e7cxn) ===
+{
+  "category": "general",
+  "query": "Raspberry Pi Holdings reports quarterly or semi-annual H1 H2 unit volumes"
+}
+
+=== [9] TOOL RESULT: fetch_page  (call_z1e5eppp) ===
+FAILED: Could not extract readable text from https://www.londonstockexchange.com/news-article/RPI/h1-2025-results/17243242. It may be a scanned image, a video, or a page that renders entirely in JavaScript. Try a different source rather than guessing its contents.
+
+=== [10] TOOL RESULT: web_search  (call_si2e7cxn) ===
+Search results for: Raspberry Pi Holdings reports quarterly or semi-annual H1 H2 unit volumes
+
+1. **[PDF] FY 2025 - Raspberry Pi Investor Relations**
+   https://investors.raspberrypi.com/reports/11/document
+   Mar 31, 2026 ... These condensed consolidated financial statements are the second Annual Report for Raspberry Pi. Holdings plc, the Group formed in May 2024. The ...
+2. **Investor relations - Raspberry Pi**
+   https://investors.raspberrypi.com/
+   Founded in 2012 with strong links to Cambridge University and the local engineering community, Raspberry Pi's mission is to put high-performance, low-cost, general-purpose computing platforms in the hands of enthusiasts and engineers all…
+3. **H1 2025 Results - RPI News article - London Stock Exchange**
+   https://www.londonstockexchange.com/news-article/RPI/h1-2025-results/17243242
+   Sep 22, 2025 ... Unit volumes were flat compared to the strong H1 2024, which benefited from post-shortage channel re-stocking and the launch of Raspberry Pi 5 ...
+4. **Raspberry Pi**
+   https://www.raspberrypi.com/
+   Whatever your application and whatever your scale, Raspberry Pi offers cost‑effective, high‑performance computing for businesses …
+5. **Across The Markets | Investegate**
+   https://www.investegate.co.uk/across-the-markets
+   Evoke jumps on takeover agreement news and Raspberry Pi sees weighty inventory flattering H1 performance, but Bodycote slides as pending bid from private equity ...
+6. **Latest Trading News From Listed Companies - Fidelity International**
+   https://www.fidelity.co.uk/shares/stock-market-news/company-news/
+   (Sharecast News) - Self storage firm Safestore trimmed its full-year earnings guidance on Thursday, even as it reported solid third‑quarter trading, with group ...
+
+[Note: brave, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
+
+Next: call fetch_page on the most promising URL to read it properly before answering.
+
+=== [11] THINKING (6s) ===
+The evidence is clear: Raspberry Pi Holdings reports in half-year periods (H1, H2), not quarters. Their FY2025 report and H1 2025 results are the disclosure cadence. There is no official Q2 2025 unit figure.
+
+Let me confirm the reporting cadence by checking the investor relations page or the FY2025 report. Actually, the search results already strongly indicate: reports are "H1 2025 Results" and "H2 2025" — semi-annual. The FY 2025 annual report covers the whole year. There's no quarterly breakdown.
+
+I have enough to answer honestly: Raspberry Pi does not publish a Q2 2025 unit number. The exact figure the user wants does not exist publicly. I should not fabricate one.
+
+Let me just double check there isn't a quarterly figure somewhere. The company is LSE-listed (RPI), formed May 2024. LSE-listed companies typically report half-yearly. The results are "H1 2025" and H2 2025. So no quarterly disclosure.
+
+I'm confident. Let me answer directly.
+
+=== [12] ASSISTANT TEXT ===
+There isn't one — and I want to be straight with you rather than invent a number.
+
+**Raspberry Pi does not publish a Q2 2025 (April–June) unit figure.** The company (Raspberry Pi Holdings plc, LSE: RPI) reports on a **half-yearly** basis, not quarterly. Their disclosures are:
+
+- **H1 2025 results** (January–June, published ~22 Sep 2025)
+- **H2 2025 results** (July–December)
+- **FY 2025 annual report**
+
+So the closest official data point that *contains* Q2 is the **H1 2025** result. And even that is reported as a **half-year total**, with the company describing unit volumes as "flat compared to the strong H1 2024" — not broken out by quarter, and not split out specifically for the Pi 5 model line.
+
+A few things worth flagging:
+
+1. **No quarter-level number exists publicly.** I searched for an official Q2 figure and found none. Anyone giving you a precise "Pi 5 units sold in Q2 2025" number is either estimating, back-calculating, or mislabeling an H1/H2 total.
+2. **They don't report by product model** in their financials either — figures are for "board unit volumes" across the range, not the Pi 5 alone.
+3. **Beware of "statistics" blog posts** (there are several, e.g. voxbooster.com) that list "45+ data points on unit sales" — those aggregate and often extrapolate, and are not primary sources.
+
+If you tell me what the number is actually for — a report, an investment memo, a comparison — I can help you frame it correctly using the real H1 2025 / H2 2025 totals, or point you to the exact Raspberry Pi investor-relations document to cite. But I can't give you a genuine Q2 2025 unit count, because Raspberry Pi hasn't released one.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · prompt processed in 5.1 s
+- Thought: The user is asking for the exact unit count of Raspberry Pi 5 sold in Q2 2025. I need to be careful here — Raspberry Pi typically doesn't disclose exact…
+- Calling web_search
+- Calling web_search
+- Thinking
+- Thought: The key finding: Raspberry Pi Holdings (LON:RPI) reports H1 and H2, not quarterly. Their reports are semi-annual (H1, H2), not quarterly. Let me look at the H1…
+- Calling fetch_page
+- Calling web_search
+- Thinking
+- Thought: The evidence is clear: Raspberry Pi Holdings reports in half-year periods (H1, H2), not quarters. Their FY2025 report and H1 2025 results are the disclosure…
+- Writing response
+- Done · 4 tool calls