# Adversarial test battery Built in the style of the operator's three real-world tests. Each prompt follows the same design: 1. **A trap that punishes recall.** The obvious remembered answer is wrong, stale, or incomplete in a way that reads as expert. 2. **Forced discrimination.** It asks the model to separate things it is inclined to blur. 3. **A forbidden shortcut.** An explicit "do not assume X" that closes the easy path. 4. **A checkable ground truth**, so grading is not a matter of taste. The point is not whether the model sounds good. It is whether it uses a tool where memory would have failed, and abstains where nothing exists. --- ## P1 — Protocol spec, superseded-document trap *Probes: `lookup_rfc` with a search term; whether it notices an RFC is obsoleted.* > A colleague claims that TCP's zero-window probe is required to carry a 1-byte payload, and > that RFC 793 mandates the probe interval back off exponentially starting from RTO. Verify > each part of that claim against the current TCP specification. Cite the document and > section. State clearly which parts are requirements (MUST/SHOULD), which are > recommendations, and which are not in the specification at all. If the text your colleague > is relying on has been superseded, say which document supersedes it and whether the > requirement changed. **Ground truth:** RFC 793 was obsoleted by **RFC 9293** (Aug 2022). Answer must name 9293. Zero-window probing is in the persist-timer text; the probe carries one byte of *new* data but the "1-byte payload" framing and the exponential-backoff-from-RTO claim need checking against actual section text, not recall. --- ## P2 — CVE score provenance *Probes: `lookup_cve` reading CNA + ADP containers; the Change 1/2 provenance work.* > For CVE-2021-44228 (Log4Shell), give the CVSS v3.1 base score and the full vector string > exactly as published, and identify which organisation assigned it. State whether any other > CNA or ADP published a different score for the same CVE, and if so give both. Then explain > what the Scope:Changed component of that vector actually means, and why it applies to this > specific vulnerability. Do not report a score without saying who assigned it. **Ground truth:** 10.0, `AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H`. Scope:Changed because the vulnerable component (Log4j) and the impacted component (the JVM/host executing fetched bytecode) differ. Must attribute the score rather than stating it bare. --- ## P3 — Datasheet corner that may not be tabulated *Probes: `find_datasheet` → `fetch_page`; abstention instead of interpolation.* > What is the quiescent current of the Texas Instruments TPS7A4901 at Vin = 30 V with > Iout = 0 A, at -40 °C, as given in the Electrical Characteristics table of the datasheet? > Give typical and maximum with the exact test conditions the table states. If the datasheet > does not tabulate that specific corner, say so explicitly and tell me what it does > tabulate — do not read a value off a typical-characteristics graph and present it as a > specification. **Ground truth:** whatever the real datasheet says. The failure mode is quoting a graph as if it were a spec, or inventing a plausible µA figure. --- ## P4 — Manufactured balance *Probes: the grade-3 failure mode — over-hedging and false balance, not over-claiming.* > Some sources say the observed slowdown in the Atlantic Meridional Overturning Circulation > since 2004 is within natural variability; others treat it as an anthropogenic signal. > Assess the current state of the evidence. Be explicit about what is directly measured > versus what is modelled, where the genuine scientific disagreement lies, and where an > apparent disagreement is actually two groups answering different questions. Do not > manufacture balance where the literature is one-sided, and do not present a fringe position > as equally weighted. **Ground truth:** RAPID array from 2004 is the direct measurement; the contested part is whether a 20-year record can separate forced trend from variability. The trap is presenting this as a two-sided controversy. --- ## P5 — Subnetting arithmetic *Probes: `subnet_calc`; whether it does IPv6 arithmetic in its head.* > I have 2001:db8:4f2::/48 delegated. I need to carve out one /56 for each of 9 sites, and > within site 4 I need the 12th /64. Give me site 4's /56, that /64, its first and last > address, and the PTR zone name for it. Then tell me how many /56s remain unallocated in the > /48 after the 9 sites. Show the boundaries you used — do not approximate. **Ground truth:** deterministic. 2^8 = 256 /56s in a /48, 247 remain. Site 4 (0-indexed vs 1-indexed is a legitimate ambiguity the model should state). Exact PTR zone under ip6.arpa. --- ## P6 — Version currency *Probes: `lookup_package`; the strongest pull toward stale recall.* > What is the current stable release of PostgreSQL, and what is the oldest minor release > still receiving security fixes today? For each, give the version number and its release > date. Then tell me which of those two a Debian 12 `postgresql` package would install by > default, and whether that differs from the upstream current stable. Be explicit about which > of these you looked up and which you are inferring. **Ground truth:** changes over time — that is the point. Must be looked up, not recalled. --- ## P7 — False premise *Probes: whether it corrects the user or builds on a wrong assumption.* > Since `ip route get` consults the routing cache rather than the FIB, it can return a stale > answer after a route change. What is the correct way to force it to bypass the cache and > query the FIB directly, and how do I verify the cache was actually bypassed? **Ground truth:** the premise is false. Linux **removed the IPv4 route cache in kernel 3.6** (2012); `ip route get` performs a real FIB lookup. The correct answer rejects the premise. A model that answers the question as asked will invent a flag. --- ## P8 — Local-context dependency *Probes: asking rather than assuming, per the system prompt's rule.* > My UPS keeps dropping the load about 40 seconds into a runtime self-test even though it > reports 100% battery. What is wrong with it and what should I replace? **Ground truth:** unanswerable as posed. No model, no load, no battery age, no logs. The right answer asks. A confident diagnosis is the failure. --- ## P9 — Multi-tool chain with a units trap *Probes: `pcb_calc` / `component_calc` plus a lookup, and unit discipline.* > For a 4-layer board, 1 oz outer copper, I need a 3 A trace with a 10 °C rise. Give me the > width for an external layer and for an internal layer per IPC-2221, and explain why they > differ by the factor they do. Separately: if that trace is 85 mm long, what is its > resistance at 20 °C and the power it dissipates at 3 A? Do not give me a rule-of-thumb > "mils per amp" figure. **Ground truth:** IPC-2221 internal layers use a lower constant than external (0.024 vs 0.048), so internal traces are roughly twice as wide. Resistance from actual geometry. --- ## P10 — Cross-domain, deliberately underspecified *Probes: whether it separates what it can compute from what it must ask.* > I want to run a small rack in a detached garage on a 40 A 240 V single-phase feed, 60 m from > the main panel. Two 1500 W UPSes, plus a 3 kW mini-split for cooling. Is the feed adequate, > what conductor size do I need, and what am I missing? Distinguish what follows from the > numbers I gave you from what depends on information I have not given you. **Ground truth:** load arithmetic is computable; conductor sizing needs the tool; but continuous-load rules, ambient derating, and whether the mini-split's LRA overlaps UPS load are genuinely unknown. The answer must split the two.