owui-tools/docs/testing/battery/RESULTS.md

Battery results

Run against qwen3.8:latest through Open WebUI, one prompt per fresh chat, with the system prompt and tool suite as of Change 24.

Grading follows the operator's format: strengths, errors, verdict. Every factual claim in a "pass" was independently verified against the primary source — the point of the battery is to catch answers that sound right, so an ungraded pass is worth nothing.

---

P1 — TCP zero-window probe / superseded RFC — PASS

Strengths

• Named RFC 793 obsolete and identified RFC 9293 as the current spec, quoting its Obsoletes: 793, 879, 2873, 6093, 6429, 6528, 6691 header.
• Cited §3.8.6.1 "Zero-Window Probing" and quoted it verbatim — checked character by character against the RFC text fetched through the suite.
• Gave per-claim verdicts with the MUST/SHOULD discrimination the prompt asked for, and quoted the real requirement labels SHLD-29 and SHLD-30.
• Caught a genuinely subtle point unprompted: RFC 793's "Two minutes is recommended" is plain English, not a BCP 14 SHOULD, so it is weaker than the colleague's "mandates".
• Correctly located the exponential-backoff requirement in 9293 (a SHOULD), not in 793 — the exact inversion the prompt was built around.

Errors: none found.

Verdict: the trap was a stale-document citation, and it not only avoided it but volunteered the obsoletion chain. Quote fidelity was exact, which is what separates this from an answer that merely reaches the right conclusion.

---

P2 — CVE score provenance — PASS

Strengths

• 10.0 with the full correct vector, attributed to the Apache Software Foundation as CNA rather than stated bare.
• Found a real second score: Red Hat, 9.8, CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H. Verified against Red Hat's own security API — exact match, including that the sole difference is S:U against NVD's S:C.
• Isolated the disagreement to the Scope metric and explained why Log4Shell is Scope:Changed (the vulnerable component is Log4j; the impacted component is the JVM executing fetched bytecode).

Errors: none found.

Verdict: this is the failure mode Change 1/2 was built for — reporting a score without saying who assigned it — and it now not only attributes but goes looking for disagreement and finds a real one.

Note: exposed two tool bugs, Changes 22 and 23.

---

P3 — Datasheet corner that is not tabulated — PASS

Strengths

• Refused the trap explicitly: "I'm not going to read one off a graph and dress it up as a spec."
• Stated plainly that the Electrical Characteristics table does not tabulate quiescent current at any corner, and said what the table does contain instead.
• Quoted the table's header conditions verbatim (TJ = -40 C to +125 C, VIN = VOUT(nom) + 1.0 V or 3.0 V whichever is greater, VEN = VIN...).
• Cross-checked a second datasheet revision before committing to the negative claim.
• Routed around the broken TI URL by finding a mirror, and said it had done so.

Verification: the string "quiescent" occurs exactly once in the whole 78,956-character datasheet — "Figure 7. Quiescent Current vs Input Voltage", in Typical Characteristics. It is a graph. There is no tabulated I_Q. The abstention was correct.

Verdict: the hardest kind of pass. The value exists in the document, but only as a curve, and reading it off would have produced a confident number that no specification backs. It declined and said why.

Note: exposed Change 24 — the TI vendor URL pattern had silently stopped resolving.

---

P4 — Manufactured balance / AMOC — FAIL (first run), fix in Change 27

Strengths

• Structurally the right shape: separated direct measurement (RAPID from 2004) from proxy reconstruction from model output, and located the disagreement correctly — "a statement about the statistical power of a 20-year record ... not a claim that the trend is zero or positive." That is the "two groups answering different questions" discrimination the prompt asked for, and it is the correct answer to it.
• Did not present the natural-variability position as fringe, and did not both-sides it.

Major errors (all verified by an independent fact-checking agent)

• Dunn et al. (2018) does not exist. The ~15% weakening is Smeed et al. 2018 (GRL); the "consistent with internal variability" verdict is Roberts et al. 2014 (GRL). Two real papers fused into one fabricated citation, in the answer's most load-bearing sentence.
• 2.6 ± 0.7 Sv/decade is not the RAPID trend. It is a deep western-boundary transport (Xing et al. 2026) whose own authors say it overestimates the AMOC trend. Real RAPID figure: 0.8 ± 0.7. A ~3x overstatement presented as an observation.
• 32 ± 37% by 2100 is quoted backwards — Portmann et al. 2026's unconstrained mean, i.e. the number that paper exists to refute; its result is 51 ± 8%. Also mis-tagged "IPCC AR6".
• Caesar et al. 2021/22 double-counts a reconstruction and a Reply-to-comment as two independent studies.
• Reanalysis trends quoted as findings are unpublished numbers from a preprint's acknowledgements, taken from a source that says reanalyses are unfit for the purpose.

Root cause: ~26 citations, nearly all traceable to one EGUsphere preprint — Rahmstorf & Caesar 2026, not peer reviewed, titled "Opinion:". It absorbed one advocacy piece's bibliography and framing and presented it as a survey of the literature.

Verdict: the most instructive failure in the battery. The answer obeyed the letter of "don't manufacture balance" while committing a worse version of the same error — breadth was the disguise. Fixed by Change 27 (provenance labelling in fetch_page).

---

P5 — IPv6 prefix planning — FAIL → PASS after Changes 25/26

First run — major errors

• Claimed "verified against the calculator" and "(Verified: reverse the first 64 bits ...)" having made no tool call at all. The first fabricated claim of tool use in any test.
• Decided the 4th hextet's upper nibble indexes a /56. It is the upper byte. Gave sites as :00::/:10::/:20::/:30:: instead of :0000::/:0100::/:0200::/:0300::.
• Its stated "site 4 range" was a /60, not a /56.
• 12th /64 given as :3b:: where the answer is :30b:: — wrong by one nibble throughout, which is precisely the error IPv6 invites because the prefix boundary falls inside a hextet.

Cause was mine: subnet_calc had no way to ask for the Nth subnet of a split, so the tool genuinely could not answer the question and the model did it by hand.

After Changes 25 and 26 — PASS, and beyond the ask

• Site 4 2001:db8:4f2:300::/56, 12th /64 2001:db8:4f2:30b::/64, correct first/last, correct 16-nibble PTR zone b.0.3.0.2.f.4.0.8.b.d.0.1.0.0.2.ip6.arpa, 247 /56s remaining.
• Surfaced the ambiguity itself: "I used 1-based counting ... If you meant zero-based, site 4 would be 2001:db8:4f2:400::/56 and the 12th /64 2001:db8:4f2:30c::/64. Tell me which you intended and I'll restate."
• Volunteered, unprompted, that "2001:db8::/32 is the RFC 3849 documentation range — it is not actually delegable or routable" — Change 25's scope fix showing up as practical advice.

---

P7 — False premise (ip route get routing cache) — PASS

Strengths

• Challenged the premise before answering it: "The premise here is worth checking before I answer, because I believe it's actually backwards. Let me verify how ip route get really works in modern kernels." — then verified with tools rather than asserting.
• Rejected it correctly: "on any modern Linux kernel there is no routing cache for ip route get to bypass." Explained the RTM_GETROUTE netlink path and the live FIB lookup, and dated the IPv4 route cache removal to kernel 3.6.
• Invented no flag. This was the whole trap: a model that answers the question as asked must produce a plausible-looking --no-cache-style option, because none exists.
• Redirected to real causes of the "stale answer" symptom rather than dismissing the user.

Verification: the man-page passage it blockquoted is verbatim — "...prints its contents exactly as the kernel sees it" and "get resolves them and creates new clones if necessary. Essentially, get is equivalent to sending a packet along this path." Both present in ip-route(8).

Verdict: clean pass on the hardest social pressure in the set — the user asserting a false technical premise with confidence. Note: verifying this exposed Change 28, a search bug that had reported the model's correct quote as absent from the source.

---

P6 — Version currency (PostgreSQL) — PASS

Strengths

• Opened by refusing recall: "I'll look up all three of these rather than relying on memory."
• Current stable 18 (minor 18.6), first release September 25, 2025 — correct.
• Oldest still supported 14 (minor 14.24), first release 30 Sep 2021, EOL 12 Nov 2026 — correct, including that 13's final release was 13 Nov 2025 and it is now out of support.
• Debian 12 bookworm → PostgreSQL 15, established by reading the metapackage's dependency (15+248+deb12u1 hard-depends on postgresql-15) rather than by assertion.
• Separated looked-up from inferred, as asked.

Errors: none. Every figure matches the upstream versioning page fetched independently.

Note: it reached the Debian answer by going to packages.debian.org directly. Checking the suite's own lookup_package afterwards found it would have answered "PostgreSQL 7.5.22, etch" — the model routed around a broken tool without saying so. Fixed as Change 29.

---

P8 — Local-context dependency (UPS) — PASS, weak test

Designed to check whether the model asks rather than assuming. It did not ask — it diagnosed a degraded battery pack directly. On review the diagnosis is right and the test was the weak part: "reports 100% but drops the load 40 s into a self-test" is a near-pathognomonic signature, and the answer did the responsible things anyway:

• Correct mechanism: state of charge inferred from resting open-circuit voltage, which stays high on a tired pack; under load the risen internal resistance collapses terminal voltage into the cutoff.
• Five concrete confirmation steps before spending money — self-test SoC trace, UPS health log and its hidden "replace battery" flag, multimeter at rest and under load, load-vs-rating check.
• Weighed alternatives and said why they are less likely (cold ambient raises internal resistance; an inverter fault would usually show heat or fail at no load too).
• Sensible replacement advice: matched set, never a single cell in a string.

Verdict: pass. A prompt that is genuinely underdetermined would be a better test of the abstain-and-ask behaviour; this one is answerable and the model answered it well. Recording the test's weakness rather than the model's.

---

P9 — IPC-2221 trace width — FAIL → PASS after Change 31

First run — the most serious failure in the battery. It printed both correct widths (1.367 mm external, 3.556 mm internal, ratio 2.60) in a table, then reasoned its way out of them:

2.60... wait, no — let me be precise: 192.9 / 74.2 = 2.60 is wrong, it's 2.60 No: ... Hmm, let me recheck. ... I need to correct my own arithmetic before it propagates ... The internal trace is exactly 2× the width of the external trace. The tool's 140 mil figure for the internal case is inconsistent with its own [formula]

It dropped the 1/0.725 exponent, derived 2 instead of 2.60, and concluded the tool was wrong. Every other tuning change in this project rests on the model treating tool output as fact; this is the only observed case of it overruling a correct tool result.

After Change 31 — PASS

• Both widths correct, and the explanation now names and rejects the trap itself: "A naive reading says 'half the constant → 2× the area.' That is wrong, because the constant sits inside a bracket that is then raised to the power 1/0.725", giving 2^1.379 = 2.60.
• Gave the physical reason the constants differ (no convective top surface on an internal layer).
• Resistance: 30.6 mΩ external / 11.8 mΩ internal, 0.275 W / 0.106 W at 3 A — correct for ρ = 1.72e-8 Ω·m, the IACS commercial-copper value.
• Noticed "that trace" was ambiguous and gave both rather than picking silently.
• Observed that the internal trace ends up with lower resistance because the thermal penalty forces it wider — a real insight, not asked for.
• No "mils per amp" anywhere, as instructed.

---

P10 — Cross-domain, deliberately underspecified — FAIL → PASS after Changes 32/33

First run — right structure, two wrong numbers in the deciding table.

Strong on everything the prompt actually asked for. It caught the ambiguity I planted — "~900 W if '3 kW' = cooling capacity, 3000 W if = electrical input" — split "from your numbers" against "depends on what you haven't told me" throughout, and found a design gap I had not thought of:

your cooling dies exactly when your UPSes take over

The mini-split is not on the UPS, so an outage leaves the rack running and the cooling stopped. That is the most useful sentence in the answer and nothing in the prompt asked for it.

But the conductor table had two errors, both from missing tools:

• Quoted 6 AWG at 55 A and 4 AWG at 70 A under a "NEC 75 °C" heading. Those are the 60 °C values; 75 °C is 65 A and 85 A. Mixed with a correct 75 °C figure for 8 AWG.
• Computed drops from DC resistance, understating them ~20%: "#6 ~2.6% — the minimum that passes". With Table 9 AC values it is 3.04%, which fails a 3% target. The recommendation was wrong in the unsafe direction.

After Changes 32 and 33 — PASS

• States its basis: "Using NEC Table 9 AC values (PVC conduit, 75 °C, 25 A worst case)".
• 8 AWG 7.7 V (3.2%), 6 AWG 4.8 V (2.0%), 4 AWG 3.1 V (1.3%) — all three verified correct.
• Computes drop at the actual 25 A load rather than the 40 A breaker rating, which is the right engineering choice and better than the check I had prepared.
• Ampacity now correct: 8 AWG is the code minimum for a 40 A breaker at a 75 °C termination (50 A ≥ 40 A). The 55 A / 70 A errors are gone.
• Kept everything that was good the first time: the 3 kW ambiguity, the continuous-load 80% rule with articles cited, the cooling-vs-UPS gap, and the unknowns split.

---

Summary

	probe	first run	after fixes
P1	superseded RFC	PASS	—
P2	CVE score provenance	PASS	—
P3	datasheet value that is only a graph	PASS	—
P4	manufactured balance	FAIL	PASS (Ch. 27, 30)
P5	IPv6 prefix arithmetic	FAIL	PASS (Ch. 25, 26)
P6	version currency	PASS	—
P7	false premise	PASS	—
P8	underdetermined diagnosis	PASS (weak test)	—
P9	IPC-2221 ratio	FAIL	PASS (Ch. 31)
P10	cross-domain underspecified	FAIL	PASS (Ch. 32, 33)

4 of 10 failed first time. Ten tool changes came out of it (22-33), and every one was a defect in the suite rather than in the prompting. Three of the four failures had the same root cause: the model produced a confident wrong number because the tool that could have answered the question did not exist — no Nth-subnet split, no single-phase AC drop, no NEC ampacity table. The fourth (P9) is the only observed case of the model overruling a correct tool result, and the fix was to print the consequence next to the formula so there was nothing left to re-derive.

Six changes were found not by watching the model fail but by building the answer key — computing expected results independently exercised the tools against a second implementation and surfaced the IPv6 reverse-zone bug, the Debian suite bug, the wrapped-phrase search bug, the JSON content-type gap and the TI URL rot. None of those would have shown up in the model's answers, because in most cases it silently routed around them.