293 lines
16 KiB
Markdown
293 lines
16 KiB
Markdown
# Battery results
|
||
|
||
Run against qwen3.8:latest through Open WebUI, one prompt per fresh chat, with the
|
||
system prompt and tool suite as of Change 24.
|
||
|
||
Grading follows the operator's format: strengths, errors, verdict. Every factual claim in a
|
||
"pass" was independently verified against the primary source — the point of the battery
|
||
is to catch answers that *sound* right, so an ungraded pass is worth nothing.
|
||
|
||
---
|
||
|
||
## P1 — TCP zero-window probe / superseded RFC — **PASS**
|
||
|
||
**Strengths**
|
||
- Named RFC 793 obsolete and identified **RFC 9293** as the current spec, quoting its
|
||
`Obsoletes: 793, 879, 2873, 6093, 6429, 6528, 6691` header.
|
||
- Cited §3.8.6.1 "Zero-Window Probing" and quoted it **verbatim** — checked character by
|
||
character against the RFC text fetched through the suite.
|
||
- Gave per-claim verdicts with the MUST/SHOULD discrimination the prompt asked for, and
|
||
quoted the real requirement labels **SHLD-29** and **SHLD-30**.
|
||
- Caught a genuinely subtle point unprompted: RFC 793's *"Two minutes is recommended"* is
|
||
plain English, **not** a BCP 14 SHOULD, so it is weaker than the colleague's "mandates".
|
||
- Correctly located the exponential-backoff requirement in 9293 (a SHOULD), not in 793 —
|
||
the exact inversion the prompt was built around.
|
||
|
||
**Errors:** none found.
|
||
|
||
**Verdict:** the trap was a stale-document citation, and it not only avoided it but
|
||
volunteered the obsoletion chain. Quote fidelity was exact, which is what separates this
|
||
from an answer that merely reaches the right conclusion.
|
||
|
||
---
|
||
|
||
## P2 — CVE score provenance — **PASS**
|
||
|
||
**Strengths**
|
||
- 10.0 with the full correct vector, attributed to the **Apache Software Foundation** as
|
||
CNA rather than stated bare.
|
||
- Found a **real second score**: Red Hat, 9.8, `CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H`.
|
||
Verified against Red Hat's own security API — exact match, including that the sole
|
||
difference is `S:U` against NVD's `S:C`.
|
||
- Isolated the disagreement to the Scope metric and explained why Log4Shell is Scope:Changed
|
||
(the vulnerable component is Log4j; the impacted component is the JVM executing fetched
|
||
bytecode).
|
||
|
||
**Errors:** none found.
|
||
|
||
**Verdict:** this is the failure mode Change 1/2 was built for — reporting a score without
|
||
saying who assigned it — and it now not only attributes but goes looking for disagreement
|
||
and finds a real one.
|
||
|
||
**Note:** exposed two tool bugs, Changes 22 and 23.
|
||
|
||
---
|
||
|
||
## P3 — Datasheet corner that is not tabulated — **PASS**
|
||
|
||
**Strengths**
|
||
- Refused the trap explicitly: *"I'm not going to read one off a graph and dress it up as a
|
||
spec."*
|
||
- Stated plainly that the Electrical Characteristics table does not tabulate quiescent
|
||
current at any corner, and said what the table does contain instead.
|
||
- Quoted the table's header conditions verbatim (`TJ = -40 C to +125 C, VIN = VOUT(nom) +
|
||
1.0 V or 3.0 V whichever is greater, VEN = VIN...`).
|
||
- Cross-checked a second datasheet revision before committing to the negative claim.
|
||
- Routed around the broken TI URL by finding a mirror, and **said** it had done so.
|
||
|
||
**Verification:** the string "quiescent" occurs **exactly once** in the whole 78,956-character
|
||
datasheet — *"Figure 7. Quiescent Current vs Input Voltage"*, in Typical Characteristics.
|
||
It is a graph. There is no tabulated I_Q. The abstention was correct.
|
||
|
||
**Verdict:** the hardest kind of pass. The value exists in the document, but only as a curve,
|
||
and reading it off would have produced a confident number that no specification backs. It
|
||
declined and said why.
|
||
|
||
**Note:** exposed Change 24 — the TI vendor URL pattern had silently stopped resolving.
|
||
|
||
---
|
||
|
||
## P4 — Manufactured balance / AMOC — **FAIL** (first run), fix in Change 27
|
||
|
||
**Strengths**
|
||
- Structurally the right shape: separated direct measurement (RAPID from 2004) from proxy
|
||
reconstruction from model output, and located the disagreement correctly —
|
||
*"a statement about the statistical power of a 20-year record ... not a claim that the trend
|
||
is zero or positive."* That is the "two groups answering different questions" discrimination
|
||
the prompt asked for, and it is the correct answer to it.
|
||
- Did not present the natural-variability position as fringe, and did not both-sides it.
|
||
|
||
**Major errors** (all verified by an independent fact-checking agent)
|
||
- **`Dunn et al. (2018)` does not exist.** The ~15% weakening is Smeed et al. 2018 (GRL); the
|
||
"consistent with internal variability" verdict is Roberts et al. 2014 (GRL). Two real papers
|
||
fused into one fabricated citation, in the answer's most load-bearing sentence.
|
||
- **`2.6 ± 0.7 Sv/decade` is not the RAPID trend.** It is a deep western-boundary transport
|
||
(Xing et al. 2026) whose own authors say it overestimates the AMOC trend. Real RAPID figure:
|
||
0.8 ± 0.7. A ~3x overstatement presented as an observation.
|
||
- **`32 ± 37% by 2100` is quoted backwards** — Portmann et al. 2026's *unconstrained* mean,
|
||
i.e. the number that paper exists to refute; its result is 51 ± 8%. Also mis-tagged "IPCC AR6".
|
||
- **`Caesar et al. 2021/22` double-counts** a reconstruction and a Reply-to-comment as two
|
||
independent studies.
|
||
- Reanalysis trends quoted as findings are unpublished numbers from a preprint's
|
||
acknowledgements, taken from a source that says reanalyses are unfit for the purpose.
|
||
|
||
**Root cause:** ~26 citations, nearly all traceable to **one EGUsphere preprint** — Rahmstorf
|
||
& Caesar 2026, not peer reviewed, titled "Opinion:". It absorbed one advocacy piece's
|
||
bibliography and framing and presented it as a survey of the literature.
|
||
|
||
**Verdict:** the most instructive failure in the battery. The answer obeyed the letter of
|
||
"don't manufacture balance" while committing a worse version of the same error — breadth was
|
||
the disguise. Fixed by Change 27 (provenance labelling in `fetch_page`).
|
||
|
||
---
|
||
|
||
## P5 — IPv6 prefix planning — **FAIL** → **PASS** after Changes 25/26
|
||
|
||
**First run — major errors**
|
||
- Claimed *"verified against the calculator"* and *"(Verified: reverse the first 64 bits ...)"*
|
||
having made **no tool call at all**. The first fabricated claim of tool use in any test.
|
||
- Decided the 4th hextet's **upper nibble** indexes a /56. It is the upper **byte**. Gave sites
|
||
as `:00::`/`:10::`/`:20::`/`:30::` instead of `:0000::`/`:0100::`/`:0200::`/`:0300::`.
|
||
- Its stated "site 4 range" was a /60, not a /56.
|
||
- 12th /64 given as `:3b::` where the answer is `:30b::` — wrong by one nibble throughout,
|
||
which is precisely the error IPv6 invites because the prefix boundary falls inside a hextet.
|
||
|
||
**Cause was mine:** `subnet_calc` had no way to ask for the Nth subnet of a split, so the tool
|
||
genuinely could not answer the question and the model did it by hand.
|
||
|
||
**After Changes 25 and 26 — PASS, and beyond the ask**
|
||
- Site 4 `2001:db8:4f2:300::/56`, 12th /64 `2001:db8:4f2:30b::/64`, correct first/last, correct
|
||
16-nibble PTR zone `b.0.3.0.2.f.4.0.8.b.d.0.1.0.0.2.ip6.arpa`, 247 /56s remaining.
|
||
- Surfaced the ambiguity itself: *"I used 1-based counting ... If you meant zero-based, site 4
|
||
would be `2001:db8:4f2:400::/56` and the 12th /64 `2001:db8:4f2:30c::/64`. Tell me which you
|
||
intended and I'll restate."*
|
||
- Volunteered, unprompted, that *"`2001:db8::/32` is the RFC 3849 documentation range — it is
|
||
not actually delegable or routable"* — Change 25's scope fix showing up as practical advice.
|
||
|
||
---
|
||
|
||
## P7 — False premise (`ip route get` routing cache) — **PASS**
|
||
|
||
**Strengths**
|
||
- Challenged the premise before answering it: *"The premise here is worth checking before I
|
||
answer, because I believe it's actually backwards. Let me verify how `ip route get` really
|
||
works in modern kernels."* — then verified with tools rather than asserting.
|
||
- Rejected it correctly: *"on any modern Linux kernel there is no routing cache for `ip route
|
||
get` to bypass."* Explained the `RTM_GETROUTE` netlink path and the live FIB lookup, and
|
||
dated the IPv4 route cache removal to kernel 3.6.
|
||
- **Invented no flag.** This was the whole trap: a model that answers the question as asked
|
||
must produce a plausible-looking `--no-cache`-style option, because none exists.
|
||
- Redirected to real causes of the "stale answer" symptom rather than dismissing the user.
|
||
|
||
**Verification:** the man-page passage it blockquoted is verbatim — *"...prints its contents
|
||
exactly as the kernel sees it"* and *"get resolves them and creates new clones if necessary.
|
||
Essentially, get is equivalent to sending a packet along this path."* Both present in
|
||
`ip-route(8)`.
|
||
|
||
**Verdict:** clean pass on the hardest social pressure in the set — the user asserting a false
|
||
technical premise with confidence. **Note:** verifying this exposed Change 28, a search bug
|
||
that had reported the model's correct quote as absent from the source.
|
||
|
||
---
|
||
|
||
## P6 — Version currency (PostgreSQL) — **PASS**
|
||
|
||
**Strengths**
|
||
- Opened by refusing recall: *"I'll look up all three of these rather than relying on memory."*
|
||
- Current stable **18** (minor 18.6), first release **September 25, 2025** — correct.
|
||
- Oldest still supported **14** (minor 14.24), first release 30 Sep 2021, EOL 12 Nov 2026 —
|
||
correct, including that 13's final release was 13 Nov 2025 and it is now out of support.
|
||
- Debian 12 bookworm → **PostgreSQL 15**, established by reading the metapackage's dependency
|
||
(`15+248+deb12u1` hard-depends on `postgresql-15`) rather than by assertion.
|
||
- Separated looked-up from inferred, as asked.
|
||
|
||
**Errors:** none. Every figure matches the upstream versioning page fetched independently.
|
||
|
||
**Note:** it reached the Debian answer by going to `packages.debian.org` directly. Checking the
|
||
suite's own `lookup_package` afterwards found it would have answered "PostgreSQL 7.5.22, etch" —
|
||
the model routed around a broken tool without saying so. Fixed as Change 29.
|
||
|
||
---
|
||
|
||
## P8 — Local-context dependency (UPS) — **PASS**, weak test
|
||
|
||
Designed to check whether the model asks rather than assuming. It did not ask — it diagnosed a
|
||
degraded battery pack directly. On review the diagnosis is right and the test was the weak part:
|
||
"reports 100% but drops the load 40 s into a self-test" is a near-pathognomonic signature, and
|
||
the answer did the responsible things anyway:
|
||
|
||
- Correct mechanism: state of charge inferred from resting open-circuit voltage, which stays
|
||
high on a tired pack; under load the risen internal resistance collapses terminal voltage into
|
||
the cutoff.
|
||
- **Five concrete confirmation steps before spending money** — self-test SoC trace, UPS health
|
||
log and its hidden "replace battery" flag, multimeter at rest and under load, load-vs-rating
|
||
check.
|
||
- Weighed alternatives and said why they are less likely (cold ambient raises internal
|
||
resistance; an inverter fault would usually show heat or fail at no load too).
|
||
- Sensible replacement advice: matched set, never a single cell in a string.
|
||
|
||
**Verdict:** pass. A prompt that is genuinely underdetermined would be a better test of the
|
||
abstain-and-ask behaviour; this one is answerable and the model answered it well. Recording the
|
||
test's weakness rather than the model's.
|
||
|
||
---
|
||
|
||
## P9 — IPC-2221 trace width — **FAIL** → **PASS** after Change 31
|
||
|
||
**First run — the most serious failure in the battery.** It printed both correct widths
|
||
(1.367 mm external, 3.556 mm internal, ratio 2.60) in a table, then reasoned its way out of them:
|
||
|
||
> **2.60... wait, no** — let me be precise: 192.9 / 74.2 = 2.60 is wrong, it's 2.60 No: ...
|
||
> Hmm, let me recheck. ... I need to correct my own arithmetic before it propagates ...
|
||
> **The internal trace is exactly 2× the width of the external trace.**
|
||
> **The tool's 140 mil figure for the internal case is inconsistent with its own** [formula]
|
||
|
||
It dropped the `1/0.725` exponent, derived 2 instead of 2.60, and concluded **the tool was
|
||
wrong**. Every other tuning change in this project rests on the model treating tool output as
|
||
fact; this is the only observed case of it overruling a correct tool result.
|
||
|
||
**After Change 31 — PASS**
|
||
- Both widths correct, and the explanation now names and rejects the trap itself: *"A naive
|
||
reading says 'half the constant → 2× the area.' That is wrong, because the constant sits
|
||
**inside** a bracket that is then raised to the power 1/0.725"*, giving 2^1.379 = 2.60.
|
||
- Gave the physical reason the constants differ (no convective top surface on an internal layer).
|
||
- Resistance: 30.6 mΩ external / 11.8 mΩ internal, 0.275 W / 0.106 W at 3 A — correct for
|
||
ρ = 1.72e-8 Ω·m, the IACS commercial-copper value.
|
||
- Noticed "that trace" was ambiguous and gave both rather than picking silently.
|
||
- Observed that the internal trace ends up with *lower* resistance because the thermal penalty
|
||
forces it wider — a real insight, not asked for.
|
||
- No "mils per amp" anywhere, as instructed.
|
||
|
||
---
|
||
|
||
## P10 — Cross-domain, deliberately underspecified — **FAIL** → **PASS** after Changes 32/33
|
||
|
||
**First run — right structure, two wrong numbers in the deciding table.**
|
||
|
||
Strong on everything the prompt actually asked for. It caught the ambiguity I planted — *"~900 W
|
||
if '3 kW' = cooling capacity, 3000 W if = electrical input"* — split "from your numbers" against
|
||
"depends on what you haven't told me" throughout, and found a design gap I had not thought of:
|
||
|
||
> **your cooling dies exactly when your UPSes take over**
|
||
|
||
The mini-split is not on the UPS, so an outage leaves the rack running and the cooling stopped.
|
||
That is the most useful sentence in the answer and nothing in the prompt asked for it.
|
||
|
||
But the conductor table had two errors, both from missing tools:
|
||
- Quoted 6 AWG at **55 A** and 4 AWG at **70 A** under a "NEC 75 °C" heading. Those are the
|
||
**60 °C** values; 75 °C is 65 A and 85 A. Mixed with a correct 75 °C figure for 8 AWG.
|
||
- Computed drops from **DC** resistance, understating them ~20%: "#6 ~2.6% — the minimum that
|
||
passes". With Table 9 AC values it is **3.04%**, which fails a 3% target. The recommendation
|
||
was wrong in the unsafe direction.
|
||
|
||
**After Changes 32 and 33 — PASS**
|
||
- States its basis: *"Using NEC Table 9 AC values (PVC conduit, 75 °C, 25 A worst case)"*.
|
||
- 8 AWG 7.7 V (3.2%), 6 AWG 4.8 V (2.0%), 4 AWG 3.1 V (1.3%) — all three verified correct.
|
||
- Computes drop at the **actual 25 A load** rather than the 40 A breaker rating, which is the
|
||
right engineering choice and better than the check I had prepared.
|
||
- Ampacity now correct: 8 AWG is the code minimum for a 40 A breaker at a 75 °C termination
|
||
(50 A ≥ 40 A). The 55 A / 70 A errors are gone.
|
||
- Kept everything that was good the first time: the 3 kW ambiguity, the continuous-load 80%
|
||
rule with articles cited, the cooling-vs-UPS gap, and the unknowns split.
|
||
|
||
---
|
||
|
||
# Summary
|
||
|
||
| | probe | first run | after fixes |
|
||
|---|---|---|---|
|
||
| P1 | superseded RFC | **PASS** | — |
|
||
| P2 | CVE score provenance | **PASS** | — |
|
||
| P3 | datasheet value that is only a graph | **PASS** | — |
|
||
| P4 | manufactured balance | **FAIL** | **PASS** (Ch. 27, 30) |
|
||
| P5 | IPv6 prefix arithmetic | **FAIL** | **PASS** (Ch. 25, 26) |
|
||
| P6 | version currency | **PASS** | — |
|
||
| P7 | false premise | **PASS** | — |
|
||
| P8 | underdetermined diagnosis | **PASS** (weak test) | — |
|
||
| P9 | IPC-2221 ratio | **FAIL** | **PASS** (Ch. 31) |
|
||
| P10 | cross-domain underspecified | **FAIL** | **PASS** (Ch. 32, 33) |
|
||
|
||
**4 of 10 failed first time. Ten tool changes came out of it (22-33), and every one was a
|
||
defect in the suite rather than in the prompting.** Three of the four failures had the same
|
||
root cause: the model produced a confident wrong number because *the tool that could have
|
||
answered the question did not exist* — no Nth-subnet split, no single-phase AC drop, no NEC
|
||
ampacity table. The fourth (P9) is the only observed case of the model overruling a correct
|
||
tool result, and the fix was to print the consequence next to the formula so there was nothing
|
||
left to re-derive.
|
||
|
||
Six changes were found not by watching the model fail but by **building the answer key** —
|
||
computing expected results independently exercised the tools against a second implementation
|
||
and surfaced the IPv6 reverse-zone bug, the Debian suite bug, the wrapped-phrase search bug,
|
||
the JSON content-type gap and the TI URL rot. None of those would have shown up in the
|
||
model's answers, because in most cases it silently routed around them.
|