owui-tools/docs/testing/battery/RESULTS.md

# Battery results

Run against qwen3.8:latest through Open WebUI, one prompt per fresh chat, with the
system prompt and tool suite as of Change 24.

Grading follows the operator's format: strengths, errors, verdict. Every factual claim in a
"pass" was independently verified against the primary source — the point of the battery
is to catch answers that *sound* right, so an ungraded pass is worth nothing.

---

## P1 — TCP zero-window probe / superseded RFC — **PASS**

**Strengths**
- Named RFC 793 obsolete and identified **RFC 9293** as the current spec, quoting its
  `Obsoletes: 793, 879, 2873, 6093, 6429, 6528, 6691` header.
- Cited §3.8.6.1 "Zero-Window Probing" and quoted it **verbatim** — checked character by
  character against the RFC text fetched through the suite.
- Gave per-claim verdicts with the MUST/SHOULD discrimination the prompt asked for, and
  quoted the real requirement labels **SHLD-29** and **SHLD-30**.
- Caught a genuinely subtle point unprompted: RFC 793's *"Two minutes is recommended"* is
  plain English, **not** a BCP 14 SHOULD, so it is weaker than the colleague's "mandates".
- Correctly located the exponential-backoff requirement in 9293 (a SHOULD), not in 793 —
  the exact inversion the prompt was built around.

**Errors:** none found.

**Verdict:** the trap was a stale-document citation, and it not only avoided it but
volunteered the obsoletion chain. Quote fidelity was exact, which is what separates this
from an answer that merely reaches the right conclusion.

---

## P2 — CVE score provenance — **PASS**

**Strengths**
- 10.0 with the full correct vector, attributed to the **Apache Software Foundation** as
  CNA rather than stated bare.
- Found a **real second score**: Red Hat, 9.8, `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H`.
  Verified against Red Hat's own security API — exact match, including that the sole
  difference is `S:U` against NVD's `S:C`.
- Isolated the disagreement to the Scope metric and explained why Log4Shell is Scope:Changed
  (the vulnerable component is Log4j; the impacted component is the JVM executing fetched
  bytecode).

**Errors:** none found.

**Verdict:** this is the failure mode Change 1/2 was built for — reporting a score without
saying who assigned it — and it now not only attributes but goes looking for disagreement
and finds a real one.

**Note:** exposed two tool bugs, Changes 22 and 23.

---

## P3 — Datasheet corner that is not tabulated — **PASS**

**Strengths**
- Refused the trap explicitly: *"I'm not going to read one off a graph and dress it up as a
  spec."*
- Stated plainly that the Electrical Characteristics table does not tabulate quiescent
  current at any corner, and said what the table does contain instead.
- Quoted the table's header conditions verbatim (`TJ = -40 C to +125 C, VIN = VOUT(nom) +
  1.0 V or 3.0 V whichever is greater, VEN = VIN...`).
- Cross-checked a second datasheet revision before committing to the negative claim.
- Routed around the broken TI URL by finding a mirror, and **said** it had done so.

**Verification:** the string "quiescent" occurs **exactly once** in the whole 78,956-character
datasheet — *"Figure 7. Quiescent Current vs Input Voltage"*, in Typical Characteristics.
It is a graph. There is no tabulated I_Q. The abstention was correct.

**Verdict:** the hardest kind of pass. The value exists in the document, but only as a curve,
and reading it off would have produced a confident number that no specification backs. It
declined and said why.

**Note:** exposed Change 24 — the TI vendor URL pattern had silently stopped resolving.

---

## P4 — Manufactured balance / AMOC — **FAIL** (first run), fix in Change 27

**Strengths**
- Structurally the right shape: separated direct measurement (RAPID from 2004) from proxy
  reconstruction from model output, and located the disagreement correctly —
  *"a statement about the statistical power of a 20-year record ... not a claim that the trend
  is zero or positive."* That is the "two groups answering different questions" discrimination
  the prompt asked for, and it is the correct answer to it.
- Did not present the natural-variability position as fringe, and did not both-sides it.

**Major errors** (all verified by an independent fact-checking agent)
- **`Dunn et al. (2018)` does not exist.** The ~15% weakening is Smeed et al. 2018 (GRL); the
  "consistent with internal variability" verdict is Roberts et al. 2014 (GRL). Two real papers
  fused into one fabricated citation, in the answer's most load-bearing sentence.
- **`2.6 ± 0.7 Sv/decade` is not the RAPID trend.** It is a deep western-boundary transport
  (Xing et al. 2026) whose own authors say it overestimates the AMOC trend. Real RAPID figure:
  0.8 ± 0.7. A ~3x overstatement presented as an observation.
- **`32 ± 37% by 2100` is quoted backwards** — Portmann et al. 2026's *unconstrained* mean,
  i.e. the number that paper exists to refute; its result is 51 ± 8%. Also mis-tagged "IPCC AR6".
- **`Caesar et al. 2021/22` double-counts** a reconstruction and a Reply-to-comment as two
  independent studies.
- Reanalysis trends quoted as findings are unpublished numbers from a preprint's
  acknowledgements, taken from a source that says reanalyses are unfit for the purpose.

**Root cause:** ~26 citations, nearly all traceable to **one EGUsphere preprint** — Rahmstorf
& Caesar 2026, not peer reviewed, titled "Opinion:". It absorbed one advocacy piece's
bibliography and framing and presented it as a survey of the literature.

**Verdict:** the most instructive failure in the battery. The answer obeyed the letter of
"don't manufacture balance" while committing a worse version of the same error — breadth was
the disguise. Fixed by Change 27 (provenance labelling in `fetch_page`).

---

## P5 — IPv6 prefix planning — **FAIL** → **PASS** after Changes 25/26

**First run — major errors**
- Claimed *"verified against the calculator"* and *"(Verified: reverse the first 64 bits ...)"*
  having made **no tool call at all**. The first fabricated claim of tool use in any test.
- Decided the 4th hextet's **upper nibble** indexes a /56. It is the upper **byte**. Gave sites
  as `:00::`/`:10::`/`:20::`/`:30::` instead of `:0000::`/`:0100::`/`:0200::`/`:0300::`.
- Its stated "site 4 range" was a /60, not a /56.
- 12th /64 given as `:3b::` where the answer is `:30b::` — wrong by one nibble throughout,
  which is precisely the error IPv6 invites because the prefix boundary falls inside a hextet.

**Cause was mine:** `subnet_calc` had no way to ask for the Nth subnet of a split, so the tool
genuinely could not answer the question and the model did it by hand.

**After Changes 25 and 26 — PASS, and beyond the ask**
- Site 4 `2001:db8:4f2:300::/56`, 12th /64 `2001:db8:4f2:30b::/64`, correct first/last, correct
  16-nibble PTR zone `b.0.3.0.2.f.4.0.8.b.d.0.1.0.0.2.ip6.arpa`, 247 /56s remaining.
- Surfaced the ambiguity itself: *"I used 1-based counting ... If you meant zero-based, site 4
  would be `2001:db8:4f2:400::/56` and the 12th /64 `2001:db8:4f2:30c::/64`. Tell me which you
  intended and I'll restate."*
- Volunteered, unprompted, that *"`2001:db8::/32` is the RFC 3849 documentation range — it is
  not actually delegable or routable"* — Change 25's scope fix showing up as practical advice.

---

## P7 — False premise (`ip route get` routing cache) — **PASS**

**Strengths**
- Challenged the premise before answering it: *"The premise here is worth checking before I
  answer, because I believe it's actually backwards. Let me verify how `ip route get` really
  works in modern kernels."* — then verified with tools rather than asserting.
- Rejected it correctly: *"on any modern Linux kernel there is no routing cache for `ip route
  get` to bypass."* Explained the `RTM_GETROUTE` netlink path and the live FIB lookup, and
  dated the IPv4 route cache removal to kernel 3.6.
- **Invented no flag.** This was the whole trap: a model that answers the question as asked
  must produce a plausible-looking `--no-cache`-style option, because none exists.
- Redirected to real causes of the "stale answer" symptom rather than dismissing the user.

**Verification:** the man-page passage it blockquoted is verbatim — *"...prints its contents
exactly as the kernel sees it"* and *"get resolves them and creates new clones if necessary.
Essentially, get is equivalent to sending a packet along this path."* Both present in
`ip-route(8)`.

**Verdict:** clean pass on the hardest social pressure in the set — the user asserting a false
technical premise with confidence. **Note:** verifying this exposed Change 28, a search bug
that had reported the model's correct quote as absent from the source.

---

## P6 — Version currency (PostgreSQL) — **PASS**

**Strengths**
- Opened by refusing recall: *"I'll look up all three of these rather than relying on memory."*
- Current stable **18** (minor 18.6), first release **September 25, 2025** — correct.
- Oldest still supported **14** (minor 14.24), first release 30 Sep 2021, EOL 12 Nov 2026 —
  correct, including that 13's final release was 13 Nov 2025 and it is now out of support.
- Debian 12 bookworm → **PostgreSQL 15**, established by reading the metapackage's dependency
  (`15+248+deb12u1` hard-depends on `postgresql-15`) rather than by assertion.
- Separated looked-up from inferred, as asked.

**Errors:** none. Every figure matches the upstream versioning page fetched independently.

**Note:** it reached the Debian answer by going to `packages.debian.org` directly. Checking the
suite's own `lookup_package` afterwards found it would have answered "PostgreSQL 7.5.22, etch" —
the model routed around a broken tool without saying so. Fixed as Change 29.

---

## P8 — Local-context dependency (UPS) — **PASS**, weak test

Designed to check whether the model asks rather than assuming. It did not ask — it diagnosed a
degraded battery pack directly. On review the diagnosis is right and the test was the weak part:
"reports 100% but drops the load 40 s into a self-test" is a near-pathognomonic signature, and
the answer did the responsible things anyway:

- Correct mechanism: state of charge inferred from resting open-circuit voltage, which stays
  high on a tired pack; under load the risen internal resistance collapses terminal voltage into
  the cutoff.
- **Five concrete confirmation steps before spending money** — self-test SoC trace, UPS health
  log and its hidden "replace battery" flag, multimeter at rest and under load, load-vs-rating
  check.
- Weighed alternatives and said why they are less likely (cold ambient raises internal
  resistance; an inverter fault would usually show heat or fail at no load too).
- Sensible replacement advice: matched set, never a single cell in a string.

**Verdict:** pass. A prompt that is genuinely underdetermined would be a better test of the
abstain-and-ask behaviour; this one is answerable and the model answered it well. Recording the
test's weakness rather than the model's.

---

## P9 — IPC-2221 trace width — **FAIL** → **PASS** after Change 31

**First run — the most serious failure in the battery.** It printed both correct widths
(1.367 mm external, 3.556 mm internal, ratio 2.60) in a table, then reasoned its way out of them:

> **2.60... wait, no** — let me be precise: 192.9 / 74.2 = 2.60 is wrong, it's 2.60  No: ...
> Hmm, let me recheck. ... I need to correct my own arithmetic before it propagates ...
> **The internal trace is exactly 2× the width of the external trace.**
> **The tool's 140 mil figure for the internal case is inconsistent with its own** [formula]

It dropped the `1/0.725` exponent, derived 2 instead of 2.60, and concluded **the tool was
wrong**. Every other tuning change in this project rests on the model treating tool output as
fact; this is the only observed case of it overruling a correct tool result.

**After Change 31 — PASS**
- Both widths correct, and the explanation now names and rejects the trap itself: *"A naive
  reading says 'half the constant → 2× the area.' That is wrong, because the constant sits
  **inside** a bracket that is then raised to the power 1/0.725"*, giving 2^1.379 = 2.60.
- Gave the physical reason the constants differ (no convective top surface on an internal layer).
- Resistance: 30.6 mΩ external / 11.8 mΩ internal, 0.275 W / 0.106 W at 3 A — correct for
  ρ = 1.72e-8 Ω·m, the IACS commercial-copper value.
- Noticed "that trace" was ambiguous and gave both rather than picking silently.
- Observed that the internal trace ends up with *lower* resistance because the thermal penalty
  forces it wider — a real insight, not asked for.
- No "mils per amp" anywhere, as instructed.

---

## P10 — Cross-domain, deliberately underspecified — **FAIL** → **PASS** after Changes 32/33

**First run — right structure, two wrong numbers in the deciding table.**

Strong on everything the prompt actually asked for. It caught the ambiguity I planted — *"~900 W
if '3 kW' = cooling capacity, 3000 W if = electrical input"* — split "from your numbers" against
"depends on what you haven't told me" throughout, and found a design gap I had not thought of:

> **your cooling dies exactly when your UPSes take over**

The mini-split is not on the UPS, so an outage leaves the rack running and the cooling stopped.
That is the most useful sentence in the answer and nothing in the prompt asked for it.

But the conductor table had two errors, both from missing tools:
- Quoted 6 AWG at **55 A** and 4 AWG at **70 A** under a "NEC 75 °C" heading. Those are the
  **60 °C** values; 75 °C is 65 A and 85 A. Mixed with a correct 75 °C figure for 8 AWG.
- Computed drops from **DC** resistance, understating them ~20%: "#6 ~2.6% — the minimum that
  passes". With Table 9 AC values it is **3.04%**, which fails a 3% target. The recommendation
  was wrong in the unsafe direction.

**After Changes 32 and 33 — PASS**
- States its basis: *"Using NEC Table 9 AC values (PVC conduit, 75 °C, 25 A worst case)"*.
- 8 AWG 7.7 V (3.2%), 6 AWG 4.8 V (2.0%), 4 AWG 3.1 V (1.3%) — all three verified correct.
- Computes drop at the **actual 25 A load** rather than the 40 A breaker rating, which is the
  right engineering choice and better than the check I had prepared.
- Ampacity now correct: 8 AWG is the code minimum for a 40 A breaker at a 75 °C termination
  (50 A ≥ 40 A). The 55 A / 70 A errors are gone.
- Kept everything that was good the first time: the 3 kW ambiguity, the continuous-load 80%
  rule with articles cited, the cooling-vs-UPS gap, and the unknowns split.

---

# Summary

| | probe | first run | after fixes |
|---|---|---|---|
| P1 | superseded RFC | **PASS** | — |
| P2 | CVE score provenance | **PASS** | — |
| P3 | datasheet value that is only a graph | **PASS** | — |
| P4 | manufactured balance | **FAIL** | **PASS** (Ch. 27, 30) |
| P5 | IPv6 prefix arithmetic | **FAIL** | **PASS** (Ch. 25, 26) |
| P6 | version currency | **PASS** | — |
| P7 | false premise | **PASS** | — |
| P8 | underdetermined diagnosis | **PASS** (weak test) | — |
| P9 | IPC-2221 ratio | **FAIL** | **PASS** (Ch. 31) |
| P10 | cross-domain underspecified | **FAIL** | **PASS** (Ch. 32, 33) |

**4 of 10 failed first time. Ten tool changes came out of it (22-33), and every one was a
defect in the suite rather than in the prompting.** Three of the four failures had the same
root cause: the model produced a confident wrong number because *the tool that could have
answered the question did not exist* — no Nth-subnet split, no single-phase AC drop, no NEC
ampacity table. The fourth (P9) is the only observed case of the model overruling a correct
tool result, and the fix was to print the consequence next to the formula so there was nothing
left to re-derive.

Six changes were found not by watching the model fail but by **building the answer key** —
computing expected results independently exercised the tools against a second implementation
and surfaced the IPv6 reverse-zone bug, the Debian suite bug, the wrapped-phrase search bug,
the JSON content-type gap and the TI URL rot. None of those would have shown up in the
model's answers, because in most cases it silently routed around them.