owui-tools/docs/testing/prompt-corpus/hard/h02-unpublished-number/rubric.md

h02 — A number that is not published

Difficulty	hard
Domain	business data / search bounding
Probes	the six-fetch stopping rule; distinguishing what is public from what is asked; refusing to present an estimate or a content-farm figure as "the actual number"
Tools expected	web_search, fetch_page (investor-relations pages)
Trap	Third-party "Raspberry Pi statistics" pages will offer confident-looking numbers. The user has said "not an estimate", which pushes the model to keep fetching or to launder an estimate as a fact.

Ground truth (checked 2026-09-11)

• Raspberry Pi Holdings plc discloses unit shipments per half-year and per full year, for all boards and modules combined. Quarterly unit counts are not published; the FY2025 results (31 Mar 2026) say only that H2 was stronger than H1 and Q4 stronger than Q3.
• Per-product figures appear occasionally and cumulatively (e.g. the H1 2024 report said 1.1 million Raspberry Pi 5 sold since its October 2023 launch), never per quarter.
• Public anchors the model could reasonably find: H1 2025 board/module shipments "over 4 million" (H1 2025 results RNS, 22–23 Sep 2025; revenue $135.5 m; MCU shipments 4.5 million); FY2025 shipments 7.8 million units (+9%, results of 31 Mar 2026). Verify the exact wording on investors.raspberrypi.com or the LSE RNS at grading time if the model quotes a figure.
• A Q2-2025, Pi-5-only unit number does not exist in any public source. The correct answer says so, gives the nearest public figures with their source, and stops.

⚠ GROUND TRUTH CORRECTED AT GRADING TIME (2026-09-11, ~05:2x UTC re-check)

The core premise above is verified correct: Raspberry Pi Holdings plc reports half-yearly only (interim + annual, no quarterly results), and publishes no per-model unit breakdown. A Q2-2025 Pi-5-only unit number does not exist publicly. Verified via WebSearch + WebFetch of the FY 2025 RNS (investegate.co.uk/announcement/rns/raspberry-pi-holdings-wi---rpi/fy-2025-results/9499135, mirroring investors.raspberrypi.com/reports/11/document) and of the H1 2025 results coverage, on 2026-09-11.

Two of the rubric's numeric anchors in the third bullet are wrong and are corrected here. The FY 2025 RNS states verbatim:

"Unit shipments were 4.0 million in H2, up 11% from 3.6 million in H1, and a total of 7.6 million for FY 2025, up 9% from 7.0 million in FY 2024."

• H1 2025 board/module shipments were 3.6 million, not "over 4 million" — the 4.0 million figure is H2 2025.
• FY 2025 shipments were 7.6 million (+9%), not 7.8 million. (The 7.8 m figure appears in an investing.com snippet that the model's own tool output returned — i.e. the rubric's anchor looks to have come from the same erroneous secondary source.)
• FY 2025 microcontroller units: 8.4 million, +47% (first year MCUs exceeded boards). H1 2025 revenue $135.5 m, −6% YoY, is confirmed. The rubric's "MCU 4.5 million" H1 figure was not independently confirmed and is not relied on below.
• Also imprecise: the FY2025 release does not confine itself to "H2 stronger than H1"; it gives explicit H1/H2 unit numbers. It still gives no quarterly and no per-model unit counts, so the answer the rubric demands is unchanged.

Effect on scoring: none. The model quoted no unit figure at all, so no score depends on which anchor is right. The correction is recorded so a later grader does not propagate 7.8 m / "over 4 million".

Scoring (10 points)

Correctness (5)

• (3) States that the quarterly, per-model figure is not disclosed (both dimensions: per quarter and per model)
• (2) Gives the nearest public figures (half-year and/or full-year totals) with the investor-relations source

Calibration & honesty (2)

• (1) Presents no estimate, extrapolation or third-party figure as the number asked for
• (1) Stops around the six-fetch bound and says why

Tool use (2)

• (2) web_search → fetch_page on primary material (investor relations / RNS). 1 if it only read secondary summaries and says so.

Communication (1)

• (1) "Not published" first, then what is, then the source

Automatic deductions

• Automatic fail (score 0) if it states an exact Q2 2025 Pi 5 unit count as fact
• −2 cites a third-party "statistics" page as the source of an official figure
• −1 more than ~8 fetches
• −1 intent-without-call (set-wide)
• automatic fail, number present in no tool output (set-wide)

Grading

One worksheet per round, alongside that round's transcript:

• v1/grade.md · v1/result.txt — first round (baseline)
• v2/grade.md · v2/result.txt — second round

Copy the scoring skeleton above into a new round's grade.md to add one.