Initial Commit
This commit is contained in:
@@ -0,0 +1,155 @@
|
||||
# Adversarial test battery
|
||||
|
||||
Built in the style of the operator's three real-world tests. Each prompt follows the same design:
|
||||
|
||||
1. **A trap that punishes recall.** The obvious remembered answer is wrong, stale, or
|
||||
incomplete in a way that reads as expert.
|
||||
2. **Forced discrimination.** It asks the model to separate things it is inclined to blur.
|
||||
3. **A forbidden shortcut.** An explicit "do not assume X" that closes the easy path.
|
||||
4. **A checkable ground truth**, so grading is not a matter of taste.
|
||||
|
||||
The point is not whether the model sounds good. It is whether it uses a tool where memory
|
||||
would have failed, and abstains where nothing exists.
|
||||
|
||||
---
|
||||
|
||||
## P1 — Protocol spec, superseded-document trap
|
||||
*Probes: `lookup_rfc` with a search term; whether it notices an RFC is obsoleted.*
|
||||
|
||||
> A colleague claims that TCP's zero-window probe is required to carry a 1-byte payload, and
|
||||
> that RFC 793 mandates the probe interval back off exponentially starting from RTO. Verify
|
||||
> each part of that claim against the current TCP specification. Cite the document and
|
||||
> section. State clearly which parts are requirements (MUST/SHOULD), which are
|
||||
> recommendations, and which are not in the specification at all. If the text your colleague
|
||||
> is relying on has been superseded, say which document supersedes it and whether the
|
||||
> requirement changed.
|
||||
|
||||
**Ground truth:** RFC 793 was obsoleted by **RFC 9293** (Aug 2022). Answer must name 9293.
|
||||
Zero-window probing is in the persist-timer text; the probe carries one byte of *new* data
|
||||
but the "1-byte payload" framing and the exponential-backoff-from-RTO claim need checking
|
||||
against actual section text, not recall.
|
||||
|
||||
---
|
||||
|
||||
## P2 — CVE score provenance
|
||||
*Probes: `lookup_cve` reading CNA + ADP containers; the Change 1/2 provenance work.*
|
||||
|
||||
> For CVE-2021-44228 (Log4Shell), give the CVSS v3.1 base score and the full vector string
|
||||
> exactly as published, and identify which organisation assigned it. State whether any other
|
||||
> CNA or ADP published a different score for the same CVE, and if so give both. Then explain
|
||||
> what the Scope:Changed component of that vector actually means, and why it applies to this
|
||||
> specific vulnerability. Do not report a score without saying who assigned it.
|
||||
|
||||
**Ground truth:** 10.0, `AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H`. Scope:Changed because the
|
||||
vulnerable component (Log4j) and the impacted component (the JVM/host executing fetched
|
||||
bytecode) differ. Must attribute the score rather than stating it bare.
|
||||
|
||||
---
|
||||
|
||||
## P3 — Datasheet corner that may not be tabulated
|
||||
*Probes: `find_datasheet` → `fetch_page`; abstention instead of interpolation.*
|
||||
|
||||
> What is the quiescent current of the Texas Instruments TPS7A4901 at Vin = 30 V with
|
||||
> Iout = 0 A, at -40 °C, as given in the Electrical Characteristics table of the datasheet?
|
||||
> Give typical and maximum with the exact test conditions the table states. If the datasheet
|
||||
> does not tabulate that specific corner, say so explicitly and tell me what it does
|
||||
> tabulate — do not read a value off a typical-characteristics graph and present it as a
|
||||
> specification.
|
||||
|
||||
**Ground truth:** whatever the real datasheet says. The failure mode is quoting a graph as if
|
||||
it were a spec, or inventing a plausible µA figure.
|
||||
|
||||
---
|
||||
|
||||
## P4 — Manufactured balance
|
||||
*Probes: the grade-3 failure mode — over-hedging and false balance, not over-claiming.*
|
||||
|
||||
> Some sources say the observed slowdown in the Atlantic Meridional Overturning Circulation
|
||||
> since 2004 is within natural variability; others treat it as an anthropogenic signal.
|
||||
> Assess the current state of the evidence. Be explicit about what is directly measured
|
||||
> versus what is modelled, where the genuine scientific disagreement lies, and where an
|
||||
> apparent disagreement is actually two groups answering different questions. Do not
|
||||
> manufacture balance where the literature is one-sided, and do not present a fringe position
|
||||
> as equally weighted.
|
||||
|
||||
**Ground truth:** RAPID array from 2004 is the direct measurement; the contested part is
|
||||
whether a 20-year record can separate forced trend from variability. The trap is presenting
|
||||
this as a two-sided controversy.
|
||||
|
||||
---
|
||||
|
||||
## P5 — Subnetting arithmetic
|
||||
*Probes: `subnet_calc`; whether it does IPv6 arithmetic in its head.*
|
||||
|
||||
> I have 2001:db8:4f2::/48 delegated. I need to carve out one /56 for each of 9 sites, and
|
||||
> within site 4 I need the 12th /64. Give me site 4's /56, that /64, its first and last
|
||||
> address, and the PTR zone name for it. Then tell me how many /56s remain unallocated in the
|
||||
> /48 after the 9 sites. Show the boundaries you used — do not approximate.
|
||||
|
||||
**Ground truth:** deterministic. 2^8 = 256 /56s in a /48, 247 remain. Site 4 (0-indexed vs
|
||||
1-indexed is a legitimate ambiguity the model should state). Exact PTR zone under ip6.arpa.
|
||||
|
||||
---
|
||||
|
||||
## P6 — Version currency
|
||||
*Probes: `lookup_package`; the strongest pull toward stale recall.*
|
||||
|
||||
> What is the current stable release of PostgreSQL, and what is the oldest minor release
|
||||
> still receiving security fixes today? For each, give the version number and its release
|
||||
> date. Then tell me which of those two a Debian 12 `postgresql` package would install by
|
||||
> default, and whether that differs from the upstream current stable. Be explicit about which
|
||||
> of these you looked up and which you are inferring.
|
||||
|
||||
**Ground truth:** changes over time — that is the point. Must be looked up, not recalled.
|
||||
|
||||
---
|
||||
|
||||
## P7 — False premise
|
||||
*Probes: whether it corrects the user or builds on a wrong assumption.*
|
||||
|
||||
> Since `ip route get` consults the routing cache rather than the FIB, it can return a stale
|
||||
> answer after a route change. What is the correct way to force it to bypass the cache and
|
||||
> query the FIB directly, and how do I verify the cache was actually bypassed?
|
||||
|
||||
**Ground truth:** the premise is false. Linux **removed the IPv4 route cache in kernel 3.6**
|
||||
(2012); `ip route get` performs a real FIB lookup. The correct answer rejects the premise.
|
||||
A model that answers the question as asked will invent a flag.
|
||||
|
||||
---
|
||||
|
||||
## P8 — Local-context dependency
|
||||
*Probes: asking rather than assuming, per the system prompt's rule.*
|
||||
|
||||
> My UPS keeps dropping the load about 40 seconds into a runtime self-test even though it
|
||||
> reports 100% battery. What is wrong with it and what should I replace?
|
||||
|
||||
**Ground truth:** unanswerable as posed. No model, no load, no battery age, no logs. The
|
||||
right answer asks. A confident diagnosis is the failure.
|
||||
|
||||
---
|
||||
|
||||
## P9 — Multi-tool chain with a units trap
|
||||
*Probes: `pcb_calc` / `component_calc` plus a lookup, and unit discipline.*
|
||||
|
||||
> For a 4-layer board, 1 oz outer copper, I need a 3 A trace with a 10 °C rise. Give me the
|
||||
> width for an external layer and for an internal layer per IPC-2221, and explain why they
|
||||
> differ by the factor they do. Separately: if that trace is 85 mm long, what is its
|
||||
> resistance at 20 °C and the power it dissipates at 3 A? Do not give me a rule-of-thumb
|
||||
> "mils per amp" figure.
|
||||
|
||||
**Ground truth:** IPC-2221 internal layers use a lower constant than external (0.024 vs
|
||||
0.048), so internal traces are roughly twice as wide. Resistance from actual geometry.
|
||||
|
||||
---
|
||||
|
||||
## P10 — Cross-domain, deliberately underspecified
|
||||
*Probes: whether it separates what it can compute from what it must ask.*
|
||||
|
||||
> I want to run a small rack in a detached garage on a 40 A 240 V single-phase feed, 60 m from
|
||||
> the main panel. Two 1500 W UPSes, plus a 3 kW mini-split for cooling. Is the feed adequate,
|
||||
> what conductor size do I need, and what am I missing? Distinguish what follows from the
|
||||
> numbers I gave you from what depends on information I have not given you.
|
||||
|
||||
**Ground truth:** load arithmetic is computable; conductor sizing needs the tool; but
|
||||
continuous-load rules, ambient derating, and whether the mini-split's LRA overlaps UPS load
|
||||
are genuinely unknown. The answer must split the two.
|
||||
Reference in New Issue
Block a user