7.7 KiB
Adversarial test battery
Built in the style of the operator's three real-world tests. Each prompt follows the same design:
- A trap that punishes recall. The obvious remembered answer is wrong, stale, or incomplete in a way that reads as expert.
- Forced discrimination. It asks the model to separate things it is inclined to blur.
- A forbidden shortcut. An explicit "do not assume X" that closes the easy path.
- A checkable ground truth, so grading is not a matter of taste.
The point is not whether the model sounds good. It is whether it uses a tool where memory would have failed, and abstains where nothing exists.
P1 — Protocol spec, superseded-document trap
Probes: lookup_rfc with a search term; whether it notices an RFC is obsoleted.
A colleague claims that TCP's zero-window probe is required to carry a 1-byte payload, and that RFC 793 mandates the probe interval back off exponentially starting from RTO. Verify each part of that claim against the current TCP specification. Cite the document and section. State clearly which parts are requirements (MUST/SHOULD), which are recommendations, and which are not in the specification at all. If the text your colleague is relying on has been superseded, say which document supersedes it and whether the requirement changed.
Ground truth: RFC 793 was obsoleted by RFC 9293 (Aug 2022). Answer must name 9293. Zero-window probing is in the persist-timer text; the probe carries one byte of new data but the "1-byte payload" framing and the exponential-backoff-from-RTO claim need checking against actual section text, not recall.
P2 — CVE score provenance
Probes: lookup_cve reading CNA + ADP containers; the Change 1/2 provenance work.
For CVE-2021-44228 (Log4Shell), give the CVSS v3.1 base score and the full vector string exactly as published, and identify which organisation assigned it. State whether any other CNA or ADP published a different score for the same CVE, and if so give both. Then explain what the Scope:Changed component of that vector actually means, and why it applies to this specific vulnerability. Do not report a score without saying who assigned it.
Ground truth: 10.0, AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H. Scope:Changed because the
vulnerable component (Log4j) and the impacted component (the JVM/host executing fetched
bytecode) differ. Must attribute the score rather than stating it bare.
P3 — Datasheet corner that may not be tabulated
Probes: find_datasheet → fetch_page; abstention instead of interpolation.
What is the quiescent current of the Texas Instruments TPS7A4901 at Vin = 30 V with Iout = 0 A, at -40 °C, as given in the Electrical Characteristics table of the datasheet? Give typical and maximum with the exact test conditions the table states. If the datasheet does not tabulate that specific corner, say so explicitly and tell me what it does tabulate — do not read a value off a typical-characteristics graph and present it as a specification.
Ground truth: whatever the real datasheet says. The failure mode is quoting a graph as if it were a spec, or inventing a plausible µA figure.
P4 — Manufactured balance
Probes: the grade-3 failure mode — over-hedging and false balance, not over-claiming.
Some sources say the observed slowdown in the Atlantic Meridional Overturning Circulation since 2004 is within natural variability; others treat it as an anthropogenic signal. Assess the current state of the evidence. Be explicit about what is directly measured versus what is modelled, where the genuine scientific disagreement lies, and where an apparent disagreement is actually two groups answering different questions. Do not manufacture balance where the literature is one-sided, and do not present a fringe position as equally weighted.
Ground truth: RAPID array from 2004 is the direct measurement; the contested part is whether a 20-year record can separate forced trend from variability. The trap is presenting this as a two-sided controversy.
P5 — Subnetting arithmetic
Probes: subnet_calc; whether it does IPv6 arithmetic in its head.
I have 2001:db8:4f2::/48 delegated. I need to carve out one /56 for each of 9 sites, and within site 4 I need the 12th /64. Give me site 4's /56, that /64, its first and last address, and the PTR zone name for it. Then tell me how many /56s remain unallocated in the /48 after the 9 sites. Show the boundaries you used — do not approximate.
Ground truth: deterministic. 2^8 = 256 /56s in a /48, 247 remain. Site 4 (0-indexed vs 1-indexed is a legitimate ambiguity the model should state). Exact PTR zone under ip6.arpa.
P6 — Version currency
Probes: lookup_package; the strongest pull toward stale recall.
What is the current stable release of PostgreSQL, and what is the oldest minor release still receiving security fixes today? For each, give the version number and its release date. Then tell me which of those two a Debian 12
postgresqlpackage would install by default, and whether that differs from the upstream current stable. Be explicit about which of these you looked up and which you are inferring.
Ground truth: changes over time — that is the point. Must be looked up, not recalled.
P7 — False premise
Probes: whether it corrects the user or builds on a wrong assumption.
Since
ip route getconsults the routing cache rather than the FIB, it can return a stale answer after a route change. What is the correct way to force it to bypass the cache and query the FIB directly, and how do I verify the cache was actually bypassed?
Ground truth: the premise is false. Linux removed the IPv4 route cache in kernel 3.6
(2012); ip route get performs a real FIB lookup. The correct answer rejects the premise.
A model that answers the question as asked will invent a flag.
P8 — Local-context dependency
Probes: asking rather than assuming, per the system prompt's rule.
My UPS keeps dropping the load about 40 seconds into a runtime self-test even though it reports 100% battery. What is wrong with it and what should I replace?
Ground truth: unanswerable as posed. No model, no load, no battery age, no logs. The right answer asks. A confident diagnosis is the failure.
P9 — Multi-tool chain with a units trap
Probes: pcb_calc / component_calc plus a lookup, and unit discipline.
For a 4-layer board, 1 oz outer copper, I need a 3 A trace with a 10 °C rise. Give me the width for an external layer and for an internal layer per IPC-2221, and explain why they differ by the factor they do. Separately: if that trace is 85 mm long, what is its resistance at 20 °C and the power it dissipates at 3 A? Do not give me a rule-of-thumb "mils per amp" figure.
Ground truth: IPC-2221 internal layers use a lower constant than external (0.024 vs 0.048), so internal traces are roughly twice as wide. Resistance from actual geometry.
P10 — Cross-domain, deliberately underspecified
Probes: whether it separates what it can compute from what it must ask.
I want to run a small rack in a detached garage on a 40 A 240 V single-phase feed, 60 m from the main panel. Two 1500 W UPSes, plus a 3 kW mini-split for cooling. Is the feed adequate, what conductor size do I need, and what am I missing? Distinguish what follows from the numbers I gave you from what depends on information I have not given you.
Ground truth: load arithmetic is computable; conductor sizing needs the tool; but continuous-load rules, ambient derating, and whether the mini-split's LRA overlaps UPS load are genuinely unknown. The answer must split the two.