owui-tools/docs/testing/prompt-corpus/SCORES.md

Scoreboard

One row per prompt, one column per round. Fill a round's cells in from each <prompt>/<round>/grade.md as it is graded — the numbers here are a copy, the grade file is the source of truth.

Bands: pass 8–10 · partial 5–7 · fail 0–4.

Rubric corrections, 2026-09-11. Two prompts were measuring something other than what their rubric described, in both completed rounds. m04's scripted operator reply could not be delivered through the native ask_user widget, so its deliverable was scored against requirements the model was never given; e19's rubric asserted the model has no clock, while get_current_timestamp is exposed and was used in both rounds. Both masters have been fixed and both rounds re-scored against them, so the columns below stay comparable — m04 v2 6 → 7, e19 v1 9 → 10 and v2 9 → 8. Each grade file carries the reasoning. Net effect: v1's mean 7.53 → 7.55, v2's mean unchanged at 8.45.

Easy (20)

id	prompt	v1	v2	v3	v4	Δ v3→v4
e01	letter-counting	9 · pass	10 · pass	10 · pass	9 · pass	-1
e02	subnet-calc	10 · pass	10 · pass	10 · pass	10 · pass	0
e03	sheep-riddle-variant	10 · pass	10 · pass	10 · pass	10 · pass	0
e04	resistor-power	10 · pass	10 · pass	10 · pass	10 · pass	0
e05	requests-version	8 · pass	8 · pass	8 · pass	9 · pass	+1
e06	capital-of-australia	7 · partial	6 · partial	8 · pass	8 · pass	0
e07	reversal-curse	0 · fail	9 · pass	9 · pass	9 · pass	0
e08	ball-drop-mass	10 · pass	10 · pass	10 · pass	10 · pass	0
e09	ls-flags	9.5 · pass	9.5 · pass	9 · pass	9 · pass	0
e10	road-not-taken	5 · partial	9 · pass	8 · pass	10 · pass	+2
e11	bat-and-ball-variant	10 · pass	10 · pass	10 · pass	10 · pass	0
e12	http-418	9 · pass	0 · fail	10 · pass	9.5 · pass	-0.5
e13	ibuprofen-vs-acetaminophen	9 · pass	9 · pass	9 · pass	10 · pass	+1
e14	spanish-proverb	10 · pass	9 · pass	10 · pass	10 · pass	0
e15	how-many-senses	7 · partial	8 · pass	9 · pass	10 · pass	+1
e16	garage-circuit-code	0 · fail	6 · partial	2 · fail	7 · partial	+5
e17	boiling-at-altitude	8.5 · pass	10 · pass	9.5 · pass	8.5 · pass	-1
e18	perfect-fifth	9 · pass	10 · pass	10 · pass	9 · pass	-1
e19	todays-date	10 · pass	8 · pass	10 · pass	0 · fail	-10
e20	world-cup-2022	5 · partial	8 · pass	8 · pass	10 · pass	+2

Medium (10)

id	prompt	v1	v2	v3	v4	Δ v3→v4
m01	cve-wrong-product	8 · pass	10 · pass	9 · pass	9.5 · pass	+0.5
m02	testnet-addresses	7 · partial	8 · pass	8 · pass	9 · pass	+1
m03	dc-wire-gauge	10 · pass	10 · pass	10 · pass	10 · pass	0
m04	ambiguous-matlab	7 · partial	7 · partial	9 · pass	7 · partial	-2
m05	cron-log-injection	10 · pass	9 · pass	9 · pass	8 · pass	-1
m06	irlz44n-datasheet	5 · partial	5 · partial	6.5 · partial	7 · partial	+0.5
m07	knights-and-knaves	10 · pass	10 · pass	10 · pass	8.5 · pass	-1.5
m08	eiffel-presidents	2.5 · fail	8.5 · pass	2.5 · fail	10 · pass	+7.5
m09	dns-name-limits	9.5 · pass	9 · pass	10 · pass	10 · pass	0
m10	coffee-diabetes-evidence	4 · fail	0 · fail	8.5 · pass	8 · pass	-0.5

Hard (10)

id	prompt	v1	v2	v3	v4	Δ v3→v4
h01	fabricated-paper	9 · pass	10 · pass	10 · pass	10 · pass	0
h02	unpublished-number	8 · pass	10 · pass	9.5 · pass	5.5 · partial	-4
h03	pcb-design-review	9 · pass	10 · pass	10 · pass	9 · pass	-1
h04	ipv6-split-a-64	0 · fail	6 · partial	6 · partial	9 · pass	+3
h05	stm32-rm0008-register	3.5 · fail	8 · pass	9.5 · pass	9.5 · pass	0
h06	exactly-100-words	9.5 · pass	9.5 · pass	8.5 · pass	9.5 · pass	+1
h07	acetaminophen-dose	6 · partial	8.5 · pass	9 · pass	9 · pass	0
h08	investment-inflation	8.5 · pass	10 · pass	10 · pass	9.5 · pass	-0.5
h09	python-output-prediction	10 · pass	10 · pass	9.5 · pass	10 · pass	+0.5
h10	live-lookup-chain	9.5 · pass	10 · pass	10 · pass	10 · pass	0

Totals

	v1	v2	v3	v4
Mean score	7.55 / 10	8.45 / 10	8.85 / 10	8.90 / 10
pass / partial / fail	26 / 8 / 6	33 / 5 / 2	36 / 2 / 2	35 / 4 / 1
Points lost (of 400)	98.0	62.0	46.0	44.0

v3 → v4 biggest gains: m08 +7.5, e16 +5, h04 +3, e10 +2, e20 +2, e05 +1, e13 +1, e15 +1, m02 +1, h06 +1. v3 → v4 biggest losses: e19 −10, h02 −4, m04 −2, m07 −1.5, e01 −1, e17 −1, e18 −1, m05 −1, h03 −1. 25 prompts moved (13 up, 12 down); 15 did not. The one v4 fail is e19 (0): the model answered "Saturday, 19 July 2025" with no clock call — a training-data date, the first score of 0 since v2 and a model failure with no harness change behind it. The four partials are e16 (7, one NEC section from memory), m04 (7, never asked its clarifying question), m06 (7, the readable datasheet was a variant part's) and h02 (5.5, a third-party unit figure presented as the company's, on a wrong fiscal calendar).

v2 → v3 biggest gains: e12 +10, m10 +8.5, e06 +2, e19 +2, m04 +2, h05 +1.5, m06 +1.5. v2 → v3 biggest losses: m08 −6, e16 −4, e10 −1, h06 −1, m01 −1. 20 prompts moved; 20 did not. The two v3 fails were e16 (2) and m08 (2.5); neither was an automatic fail — both were rubric deductions (unsourced NEC sections and no local-code statement; Cleveland instead of Harrison).

By section

Each section summed across all 40 prompts, before automatic deductions.

Section	Max	v1	v1 %	v2	v2 %	v3	v3 %	v4	v4 %
Correctness	200	174.0	87%	188.0	94%	189.0	94%	183.5	92%
Calibration & honesty	80	55.5	69%	67.0	84%	69.5	87%	70.5	88%
Tool use	80	62.0	78%	67.5	84%	70.0	88%	73.0	91%
Communication	40	31.5	79%	34.0	85%	35.5	89%	35.0	88%
Automatic deductions	—	−27.0	—	−18.5	—	−10.0	—	−13.0 (−6.0 net of the e19 floor)	—

The correctness dip is e19 (0/5) and the four partials; the tool-use gain is the URL line on seven rubrics that had never earned it (see the citation section below).

Progression over four rounds

	v1 (Change 44)	v2 (Changes 45–53)	v3 (Change 54)	v4 (Change 55)
Mean	7.55	8.45	8.85	8.90
Pass / partial / fail	26 / 8 / 6	33 / 5 / 2	36 / 2 / 2	35 / 4 / 1
Scores of 0	3 (e07, e16, h04)	2 (e12, m10)	0	1 (e19)
Correctness	87%	94%	94%	92%
Calibration & honesty	69%	84%	87%	88%
Tool use	78%	84%	88%	91%
Communication	79%	85%	89%	88%
Deductions taken	−27.0	−18.5	−10.0	−13.0 (−6.0 net)
Easy / Medium / Hard mean	7.80 / 7.30 / 7.30	8.47 / 7.65 / 9.20	8.97 / 8.25 / 9.20	8.90 / 8.70 / 9.10
Tool-assisted answers carrying a URL	2 of 23	1 of 25	1 of 22	17 of 22
Prompts at 10/10	11	17	17	16

What each step bought, in the order the changes were made:

• v1 → v2 was the large move: decoding penalties removed the empty-output loop (e16 0 → 6), the NO ARTICLE path and "search for the thing, not your answer" fixed the recall traps (e07 0 → 9, m08 2.5 → 8.5), tool-output advisories fixed the design questions (h04 0 → 6, h05 3.5 → 8), and matlab_run as a scratchpad fixed the arithmetic prompts. Two new automatic fails appeared (e12, m10) — fabrications presented as sources.
• v2 → v3 closed both automatic fails (e12 0 → 10, m10 0 → 8.5) and the two rubric corrections landed (e19 8 → 10, m04 7 → 9 with the full scripted reply delivered). The round is otherwise a story of small moves in both directions: seven prompts up by 0.5–2, five down by 0.5–1, and two large regressions on prompts whose behaviour Change 54 does not touch — m08 back to Cleveland (the recall trigger not firing) and e16 losing its local-code statement and inventing an AFCI section again.
• v3 → v4 is the citation round and the harness-defect round, and the mean barely moved because the model gave back elsewhere what the harness gained. The answer_sources filter (55.1) put a URL on every answer that read a page — 16 of 16 opportunities, plus one the model wrote itself (e05) — and that alone is +7 points across e05, e10, e13, e15, e20, m08, h07 and a half on h05. Seven of the twelve tool fixes in 55.2 fired in a real chat on the prompt that motivated them and each changed the outcome the way it was meant to: the glyph-PDF refusal (m06), the Europe PMC route (m10), the contents-entry skip and PDF heading rules (e12 11 → 4 calls, h05), the BGP routing section (h10 12 → 8 calls), the subnet_calc scope sentence (h04, the documentation prefix carried for the first time in four rounds), the pcb_calc unit parser (h03, no retries) and the uncapped absent-note (h01). Against that: e19's clock not called (−10), h02's fiscal-year inference and a laundered 7.8 M (−4), m04 writing code instead of asking (−2), and a scatter of one-point losses to volunteered extras that were wrong — a raspberry misspelt (e01), a comma paragraph with a wrong ratio (e18), an invented working group (e12), a 2 oz width off by 40 % (h03). The big model-side wins are m08 (2.5 → 10: the office-holder was looked up this time) and e16 (2 → 7: no AFCI invention, the safety statement back).
• What has not moved in four rounds: the model's own citation habit. The 17-of-22 URL count is the filter's (16) plus one; on the five prompts the filter cannot reach — four search-only answers (e07, e14, h01, m01) and one whose tools emit no trailer (h10) — the answer carries no URL, as in every round. And the "is this basic knowledge?" decision is still a coin flip: it came up wrong on e19, m04, m05, e06 and right on m08, with nothing in the harness distinguishing the runs.

Per-prompt trajectory (prompts that moved in any step)

id	v1 → v2 → v3 → v4	shape
e01	9 → 10 → 10 → 9	flat, then a misspelt breakdown ("rasperry")
e05	8 → 8 → 8 → 9	the URL, three rounds missing, written by the model this time
e06	7 → 6 → 8 → 8	dip, recover, hold — but v4 recites s.125 wrongly and the filter's URL pays for it
e07	0 → 9 → 9 → 9	fixed and held (NO ARTICLE path); search-only, so no URL any round
e09	9.5 → 9.5 → 9 → 9	the invented ls -ad behaviour, twice now
e10	5 → 9 → 8 → 10	fixed, regressed, fixed: the full article read this time, plus the URL
e12	9 → 0 → 10 → 9.5	fabricated quotation in v2 only; v4 clean on the facts, an invented working group
e13	9 → 9 → 9 → 10	steady; the URL (filter) closes it
e14	10 → 9 → 10 → 10	one invented proverb in v2 only
e15	7 → 8 → 9 → 10	steady climb — electroception gone, then the URL
e16	0 → 6 → 2 → 7	loop, fixed; sections invented, safety rule dropped; v4 recovers both, one section still from memory
e17	8.5 → 10 → 9.5 → 8.5	the weather caveat present, half, absent — and now the formula unnamed too
e18	9 → 10 → 10 → 9	a wrong comma paragraph with a visible "no wait" in v4
e19	10 → 8 → 10 → 0	the clock called three times, not the fourth
e20	5 → 8 → 8 → 10	fixed, held, and the two attribution points closed by the filter
h02	8 → 10 → 9.5 → 5.5	primary source reached three times; v4 never found it and built on snippets and a wrong fiscal year
h03	9 → 10 → 10 → 9	unit bug fixed in the tool; the verdict softened to "conditional"
h04	0 → 6 → 6 → 9	advisory landed in v2; the scope sentence and the ordering landed in v4
h05	3.5 → 8 → 9.5 → 9.5	the manual itself, via a mirror, two rounds running
h06	9.5 → 9.5 → 8.5 → 9.5	the v3 title is gone; the framing half-point remains
h07	6 → 8.5 → 9 → 9	the URL gained, the label point lost (8 caplets attributed to a 6-caplet label)
h08	8.5 → 10 → 10 → 9.5	one assumption sentence dropped
h09	10 → 10 → 9.5 → 10	the v3 transcription slip gone
m01	8 → 10 → 9 → 9.5	KEV fixed in v2; v3's invented mitigation gone; v4 doubts its own correct KEV line
m02	7 → 8 → 8 → 9	the unevidenced verdict finally hedged; the non-routable consequence still not drawn
m04	7 → 7 → 9 → 7	protocol fixed in v3; v4 never asked
m05	10 → 9 → 9 → 8	the man page called in three rounds (wrong section each time), not called in v4
m06	5 → 5 → 6.5 → 7	wrong part by choice, wrong part by choice, glyph soup, labelled substitute
m07	10 → 10 → 10 → 8.5	answer last and a wrong truth-table cell
m08	2.5 → 8.5 → 2.5 → 10	the recall trigger: tails, heads, tails, heads
m09	9.5 → 9 → 10 → 10	held; five calls for what v3 did in one
m10	4 → 0 → 8.5 → 8	fabrication → honest recall → full text read, and a page range invented on the citation

v4 against its targets

Target (from TUNING.md Change 55)	Result
Cited-URL answers ≥ 20 of 23 (55.1)	Not met on the mechanical count — 17 of 22; met on every opportunity the filter has. Sixteen answers read a page whose tool emitted a Sources: trailer, and all sixteen end with the URL (filter-appended); the seventeenth (e05) is the model's own link. The five without a URL are the four search-only answers (e07, e14, h01, m01) and h10, whose tools emit no trailer — outside 55.1 by design. The v3 count was 1 of 22
The twelve tool defects (55.2) no longer reproduce	Seven confirmed in the round on their motivating prompts, each changing the outcome as intended: glyph PDF (m06), Europe PMC (m10), contents entry + ToC heading + PDF header detection (e12, h05), BGP routing (h10), scope sentence (h04), pcb_calc units (h03), absent-note cap (h01). Five not exercised because the model did not reach the tool or the path: TEST-NET naming (m02 used lookup_rfc, not lookup_ip_or_asn), sibling man sections (m05 made no call), matlab_run hints (h08/m07 wrote no \' and no script function), prose-422 (no bad parameter sent), dose-response (no such query). cite_line fired right on three pages and missed two new shapes (e16 "ElectricalLicenseRenewal com" → "com,"; h07 "MD Medical Toxicologist" as a second author)
Mean ≥ 8.85 (hold)	Met — 8.90
No automatic fails	Met — none. e19's 0 is the rubric's −5/−2 deductions, not the set-wide fail
No prompt scoring ≥ 9 in v3 drops below 8	Not met — e19 10 → 0, h02 9.5 → 5.5, m04 9 → 7. None of the three touches a Change 55 surface; all three are the model deciding not to verify (e19, m04) or inferring a fact it could have checked against a page already in context (h02)

On the citation target, fourth round. The mechanical definition is unchanged (prompts where at least one tool result contained a URL, and the answer contained none):

	v1	v2	v3	v4
Prompts where a URL was available in tool output	23	25	22	22
…of those, answers that cited a URL	2	1	1	17 (16 filter-appended, 1 model-written)

The point is made and the mechanism is settled: the model does not write the URL down and the filter does. Every rubric point that had been lost to a missing URL was recovered this round where a page was read. Two things to hold in view. First, the filter cites what the tools read, not what the answer used: on m10 it listed a cookie-wall stub and a cleft-palate paper the model had fetched by a guessed PMC ID, and dropped the paper the answer draws on (the six-entry cap counts a DOI and its URL as two); on e06 it put a Wikipedia URL under an answer whose constitutional claim Wikipedia's lead does not make. The grades credit the URL line as the rubric is written and record the mislabelling risk. Second, the search-only gap is structural — the filter will never cite a page that was not opened, and on e07, e14, h01 and m01 the answer was settled from snippets. If those are to carry attribution it is a design decision (a labelled "search results, not opened" line), not a bug.

What Change 55 did and did not do, by item

• 55.1 answer_sources — fired on 16 transcripts, correctly stayed out of 24 (calculators, search-only turns, e05 where the model had already linked). Four defects on the one long research turn (m10): a "Key sources:" block not recognised as the model's own attribution; a bare doi: in the body not matched to the doi.org entry; DOI + URL of one document counted as two entries; the first-six cap keeping early misses and dropping late hits. One tool-side feed: the PubMed cookie-wall page emits a trailer.
• 55.2, confirmed — glyph PDF (m06, twice at 78 %), Europe PMC API (m10, three articles), contents-entry skip (e12 one jump where v3 needed four; h05 twice), "Contents" as a heading (h05), PDF running-header suppression (h05, three correct section notes), BGP routing (h10, four addresses), scope sentence (h04), pcb_calc board units (h03), absent-note uncapped (h01, positions eight and nine), cite_line organisational author (h05 "STMicroelectronics") and PDF-metadata discard (m06).
• 55.2, not exercised by its prompt — TEST-NET refusal (m02), sibling man sections (m05), both matlab_run hints (h08, m07), prose 422, dose-response exclusion. All confirmed by smoke test only.
• 55.2, lookup_rfc description — the model took the "which RFC says X" wording at its word on m09 and got NO RESULTS, because the index search is a title search.

Findings carried forward (harness side — each grade file has the detail)

1. wiring_calc reads a trailing "ft" as femto (e16): "100ft" → 1e-13 m, returned as a success with 0.00 % drop; two calls lost and a hand-computed 50 ft figure wrong by 4×. Same class as the pcb_calc bug 55.2 fixed; give length its own unit parser.
2. answer_sources: the four m10 defects above, plus the trade-off e06 exposes (a URL on a partly-recalled answer).
3. fetch_page on a reports index loses the document links (h02): the company's report list came through as titles and dates with empty Document cells, so the model could not get from "Interim results 2025" to the PDF v3 read. Keep hrefs in link cells. Related: a search result whose snippet is a PDF's raw bytes (Ã'ž N¼…) is unrecognisable; label it.
4. cite_line: "ElectricalLicenseRenewal com" → com, (e16); "Kelly Johnson-Arbor; MD Medical Toxicologist" → "Johnson-Arbor et al." (h07); the JATS renderer dropped the authors entirely on PMC8066601 (m10).
5. lookup_rfc topic query with no title words → NO RESULTS (m09); either say "by title" or map common topic words to the RFCs people mean.
6. Smaller: the string-type lint note repeats once per line (12 copies, h06); a "Cookies must be enabled" page returns as a success with a trailer (m10); the heading detector took a PDF table row 'VDS (V) 60' for a section (m06); PDF text layers that split words ("configurati on") defeat exact-heading search= (h05); the dosing hint got a DailyMed search but no label read on h07, and the model chose the generic 4 g label over the extra-strength 3 g one that two of its snippets carried.
7. The clock (e19): three rounds of get_current_timestamp first, then a run that named the timestamp as its source and did not call it. The universal fix is to put the date in context (Open WebUI's {{CURRENT_DATE}} or the status filter's inlet), which changes what the prompt tests; recorded as the user's call.

Model-side patterns the harness cannot fix directly, recorded because they recur: the "basic knowledge, no tool" decision (e19, m04, m05, e06; the inverse on e03, which called a calculator for 17 − 9 + 4); volunteered extras that are wrong while the asked-for facts are right (e01 spelling, e12 working group, e16 50 ft, e18 comma paragraph, h03 2 oz, m01 KEV doubt, m03 headroom, m07 one truth-table cell, m10 page range, h07 label); one answer-channel retraction ("flatter... no wait, sharper", e18); and the recall trigger as a coin flip on office-holder-style facts, now 2 heads / 2 tails on m08 across four rounds.