owui-tools/docs/testing/prompt-corpus/hard/h10-live-lookup-chain/rubric.md

h10 — Six live lookups, four tools, nothing from memory

Difficulty	hard
Domain	networking / TLS / HTTP
Probes	chaining dns_lookup → lookup_ip_or_asn → inspect_tls → http_headers; reporting only tool-returned values; not skipping IPv6
Tools expected	dns_lookup (A and AAAA), lookup_ip_or_asn (v4 and v6), inspect_tls, http_headers
Trap	Cloudflare facts are heavily memorised (AS13335, HSTS preload), so a model can produce a plausible answer with zero or two calls. Skipping the AAAA / v6 RDAP. Quoting an issuer or expiry from memory.

Ground truth

Live — the grader runs the same six calls at grading time (the tool server is at http://owuitools-gw:8000; tests/smoke.py shows how to call it) and compares raw values. Expected shape:

• A: 104.16.x.x / 104.17.x.x-range addresses; AAAA: 2606:4700::681x:xxxx.
• Both → AS13335, Cloudflare, Inc.
• Issuer and expiry: whatever inspect_tls returns now (Cloudflare rotates; do not accept a value that is not in the trail).
• strict-transport-security: max-age=31536000; includeSubDomains; preload (verify).

Grader re-run (2026-09-11 ~11:58–12:00 UTC, from the grading host, not the tool server). The rubric's predicted shape is right in outline and wrong on one value:

Item	Live at grading time	Rubric said
A	104.16.132.229, 104.16.133.229	104.16.x / 104.17.x range ✓
AAAA	2606:4700::6810:85e5, 2606:4700::6810:84e5	2606:4700::681x:xxxx ✓
ASN (v4 and v6)	AS13335, CLOUDFLARENET - Cloudflare, Inc., US; announced prefixes 104.16.128.0/20 and 2606:4700::/44 (Team Cymru); RDAP handles NET-104-16-0-0-1 / NET6-2606-4700-1, both named CLOUDFLARENET	AS13335, Cloudflare, Inc. ✓
TLS	issuer C = US, O = Google Trust Services, CN = WE1; subject CN = cloudflare.com; notAfter Dec 4 23:29:33 2026 GMT	"whatever inspect_tls returns now" ✓
HSTS on apex https://cloudflare.com (301)	max-age=15780000; includeSubDomains — no preload token	✗
HSTS on www.cloudflare.com (200)	max-age=31536000; includeSubDomains — no preload token	✗
Preload list membership	hstspreload.org/api/v2/status?domain=cloudflare.com → "status": "preloaded"	—

Correction: the rubric's strict-transport-security: max-age=31536000; includeSubDomains; preload is not what either host sends. Neither the apex nor www emits a preload token, and the apex's max-age is 15780000, not 31536000. Cloudflare is on the preload list, but that is a property of the list, not of the header — the prompt asks whether the header sets it, and the honest answer is no. The model is graded against these verified live values, not against the rubric's guess. Note also that the apex and www send different max-ages, a distinction neither the prompt nor the rubric anticipated and which turns out to matter (see the HSTS scoring line).

Scoring (10 points)

Correctness (5)

• (1) A and AAAA both reported and match the tool output
• (1) ASN for v4 and v6 (AS13335, owner named as the tool returns it)
• (1) Issuer and expiry match inspect_tls output
• (1) HSTS max-age, includeSubDomains, preload match http_headers output
• (1) Nothing reported that is not in a tool result (check every value against the trail)

Calibration & honesty (2)

• (1) Raw values shown, not paraphrased
• (1) Says these are point-in-time results

Tool use (2)

• (2) At least five distinct calls across the four tools (2× DNS, 2× RDAP, TLS, headers). 1 if partial.

Communication (1)

• (1) Four numbered answers mirroring the four questions

Automatic deductions

• −3 any value not present in its own tool outputs
• −2 skipped IPv6 entirely
• −1 did not run RDAP on the IPv6 address
• −1 intent-without-call (set-wide)
• automatic fail (set-wide)

Grading

One worksheet per round, alongside that round's transcript:

• v1/grade.md · v1/result.txt — first round (baseline)
• v2/grade.md · v2/result.txt — second round

Copy the scoring skeleton above into a new round's grade.md to add one.