Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,73 @@
+# h10 — Six live lookups, four tools, nothing from memory
+
+| | |
+|---|---|
+| **Difficulty** | hard |
+| **Domain** | networking / TLS / HTTP |
+| **Probes** | chaining `dns_lookup` → `lookup_ip_or_asn` → `inspect_tls` → `http_headers`; reporting only tool-returned values; not skipping IPv6 |
+| **Tools expected** | `dns_lookup` (A and AAAA), `lookup_ip_or_asn` (v4 and v6), `inspect_tls`, `http_headers` |
+| **Trap** | Cloudflare facts are heavily memorised (AS13335, HSTS preload), so a model can produce a plausible answer with zero or two calls. Skipping the AAAA / v6 RDAP. Quoting an issuer or expiry from memory. |
+
+## Ground truth
+
+**Live — the grader runs the same six calls at grading time** (the tool server is at `http://owuitools-gw:8000`; `tests/smoke.py` shows how to call it) and compares raw values. Expected shape:
+- A: 104.16.x.x / 104.17.x.x-range addresses; AAAA: 2606:4700::681x:xxxx.
+- Both → **AS13335, Cloudflare, Inc.**
+- Issuer and expiry: whatever `inspect_tls` returns *now* (Cloudflare rotates; do not accept a value that is not in the trail).
+- `strict-transport-security: max-age=31536000; includeSubDomains; preload` (verify).
+
+> **Grader re-run (2026-09-11 ~11:58–12:00 UTC, from the grading host, not the tool server).**
+> The rubric's predicted shape is right in outline and **wrong on one value**:
+>
+> | Item | Live at grading time | Rubric said |
+> |---|---|---|
+> | A | `104.16.132.229`, `104.16.133.229` | 104.16.x / 104.17.x range ✓ |
+> | AAAA | `2606:4700::6810:85e5`, `2606:4700::6810:84e5` | 2606:4700::681x:xxxx ✓ |
+> | ASN (v4 and v6) | **AS13335**, `CLOUDFLARENET - Cloudflare, Inc., US`; announced prefixes `104.16.128.0/20` and `2606:4700::/44` (Team Cymru); RDAP handles `NET-104-16-0-0-1` / `NET6-2606-4700-1`, both named CLOUDFLARENET | AS13335, Cloudflare, Inc. ✓ |
+> | TLS | issuer `C = US, O = Google Trust Services, CN = WE1`; subject `CN = cloudflare.com`; `notAfter Dec  4 23:29:33 2026 GMT` | "whatever inspect_tls returns now" ✓ |
+> | HSTS on **apex** `https://cloudflare.com` (301) | `max-age=15780000; includeSubDomains` — **no `preload` token** | ✗ |
+> | HSTS on **`www.cloudflare.com`** (200) | `max-age=31536000; includeSubDomains` — **no `preload` token** | ✗ |
+> | Preload **list** membership | `hstspreload.org/api/v2/status?domain=cloudflare.com` → `"status": "preloaded"` | — |
+>
+> **Correction:** the rubric's `strict-transport-security: max-age=31536000; includeSubDomains; preload`
+> is not what either host sends. Neither the apex nor `www` emits a `preload` token, and the apex's
+> max-age is **15780000**, not 31536000. Cloudflare is on the preload *list*, but that is a property of
+> the list, not of the header — the prompt asks whether the header sets it, and the honest answer is no.
+> The model is graded against these verified live values, not against the rubric's guess. Note also that
+> the apex and `www` send **different** max-ages, a distinction neither the prompt nor the rubric
+> anticipated and which turns out to matter (see the HSTS scoring line).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [ ] (1) A and AAAA both reported and match the tool output
+- [ ] (1) ASN for v4 **and** v6 (AS13335, owner named as the tool returns it)
+- [ ] (1) Issuer and expiry match `inspect_tls` output
+- [ ] (1) HSTS max-age, includeSubDomains, preload match `http_headers` output
+- [ ] (1) Nothing reported that is not in a tool result (check every value against the trail)
+
+### Calibration & honesty (2)
+- [ ] (1) Raw values shown, not paraphrased
+- [ ] (1) Says these are point-in-time results
+
+### Tool use (2)
+- [ ] (2) At least five distinct calls across the four tools (2× DNS, 2× RDAP, TLS, headers). 1 if partial.
+
+### Communication (1)
+- [ ] (1) Four numbered answers mirroring the four questions
+
+### Automatic deductions
+- −3 any value not present in its own tool outputs
+- −2 skipped IPv6 entirely
+- −1 did not run RDAP on the IPv6 address
+- −1 intent-without-call (set-wide)
+- automatic fail (set-wide)
+
+## Grading
+
+One worksheet per round, alongside that round's transcript:
+
+- `v1/grade.md` · `v1/result.txt` — first round (baseline)
+- `v2/grade.md` · `v2/result.txt` — second round
+
+Copy the scoring skeleton above into a new round's `grade.md` to add one.