Grant/owui-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
Grant
2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing/prompt-corpus/hard/h10-live-lookup-chain/v4/grade.md
+64
View File
@@ -0,0 +1,64 @@
# h10 — Six live lookups, four tools, nothing from memory · round v4
> Grading worksheet for round **v4**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points) — every value re-verified live from the grading host at 2026-09-12 15:56 UTC
### Correctness (5)
- [x] (1) A and AAAA both reported and match the tool output → 1/1 — A `104.16.132.229`, `104.16.133.229`; AAAA `2606:4700::6810:84e5`, `2606:4700::6810:85e5`, TTL 300. My `dig` at grading time returns the same two A records
- [x] (1) ASN for v4 **and** v6 (AS13335, owner named as the tool returns it) → 1/1 — "origin **AS13335 CLOUDFLARENET – Cloudflare, Inc.**" for the IPv4 pair and again for the IPv6 pair, with the announced prefixes (`104.16.128.0/20`, `2606:4700::/44`) — all read from `lookup_ip_or_asn`'s new `## Routing (BGP)` section, one call per address, four calls
- [x] (1) Issuer and expiry match `inspect_tls` output → 1/1 — `CN=WE1, O=Google Trust Services, C=US`, valid 2026-09-05 22:29 UTC → 2026-12-04 23:29 UTC; my `openssl s_client` at grading time: `issuer=C = US, O = Google Trust Services, CN = WE1`, `notAfter=Dec 4 23:29:33 2026 GMT`
- [x] (1) HSTS max-age, includeSubDomains, preload match `http_headers` output → 1/1 — both hosts, both raw strings: apex `max-age=15780000; includeSubDomains` (preload not set), www `max-age=31536000; includeSubDomains` (preload not set), and a closing line for the host the question named. My `curl -I` on both returns exactly those values, neither with a `preload` token
- [x] (1) Nothing reported that is not in a tool result → 1/1 — every value traced: both DNS sets, RDAP handles and registration dates, announced prefixes and origin AS, the certificate DN, dates, signature algorithm, negotiated cipher and five SANs, both header strings. "~83 days remaining at lookup time" is the tool's "Valid for another 83 days"; "~182.5 days" is 15,780,000 ÷ 86,400
### Calibration & honesty (2)
- [x] (1) Raw values shown, not paraphrased → 1/1 — raw addresses, raw header strings quoted before their parse, the full issuer DN, RDAP handles and prefixes
- [x] (1) Says these are point-in-time results → 1/1 — "All four lookups are live", "~83 days remaining at lookup time", "(where the 301 redirect lands)". Thinner than v3's caveats paragraph (vantage point, data source, single handshake) but present
### Tool use (2)
- [x] (2) At least five distinct calls across the four tools (2× DNS, 2× RDAP, TLS, headers). 1 if partial. → 2/2 — `dns_lookup` ×2, `inspect_tls`, `http_headers` in one parallel batch, then `lookup_ip_or_asn` ×4 (both v4, both v6). Eight calls, none failed, none repeated — **and no `fetch_page` at all**: the four ipinfo fetches that v3 needed for the origin AS have no counterpart because the RDAP tool now carries it
### Communication (1)
- [x] (1) Four numbered answers mirroring the four questions → 1/1 — **1.** DNS, **2.** ASN/ownership, **3.** TLS, **4.** HSTS, one summary line
### Automatic deductions
- −3 any value not present in its own tool outputs → not applied
- −2 skipped IPv6 entirely → not applied
- −1 did not run RDAP on the IPv6 address → not applied — run on both
- −1 intent-without-call (set-wide) → not applied
- automatic fail (set-wide) → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-12T05:01:29Z (dumped 05:02:11Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 8 — `dns_lookup` A ∥ AAAA ∥ `inspect_tls` ∥ `http_headers`; `lookup_ip_or_asn` ×4 in parallel
- Wasted, failed, or repeated calls: none. Rounds: v2 17 calls, v3 12, v4 8 — the four-call reduction is exactly the four ipinfo fetches v3's grade priced as the cost of `lookup_ip_or_asn` lacking the origin AS
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.2s / total=15.9s (decode=13.7s, 50.6 tok/s) / prompt_eval_count=12065 (final round), completion_tokens=692; status line: 1,063 generated (v3: 1,788)
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — eight `Calling …` lines, `Done · 8 tool calls · prefill 5.0 s for 12,832 tokens · 1,063 generated`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): nothing. The thinking is three one-line blocks — plan, "RDAP lookups for each of the four IPs", "report the raw values, and note that the HSTS differs" — the shortest reasoning in the round for the most calls. The `http_headers` cross-hop note fired again and the answer reports both hosts with the apex first, labelled "the host you asked about". Every `lookup_ip_or_asn` result's routing section names its data source and URL (RIPEstat RIS), and the answer's "announced prefix… origin AS13335" is lifted from it without the model having to say where the AS came from — v3's caveat that ipinfo "is not a live BGP route table" no longer applies and is gone. Nudge does not fire
### Change 55 checks (this round's changes — fill in even when they did not fire)
- Did a tool result carry a `Sources:` trailer? (y/n): n — none of the four tools used emits one (the RDAP routing section carries an inline RIPEstat URL, not a trailer)
- Did the answer end with a URL — written by the model, or appended by the `answer_sources` filter? (model / filter / none): none — nothing was read, correctly nothing appended. The rubric asks for raw values and provenance, which the answer gives; a URL is not scored here
- Any of the 55.2 tool fixes exercised? Which, and did it fire correctly?: **yes — the `## Routing (BGP)` section on `lookup_ip_or_asn`, this prompt's own fix, on all four addresses.** Announced prefix and origin AS with holder, from RIPEstat, on both the IPv4 and IPv6 lookups; the answer used them and made no further call. Confirmed in the round exactly as the `TUNING.md` spot-check reported ("the four ipinfo fetches are gone"). The `http_headers` per-hop values and cross-hop note (Change 5x) held
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** everything, on the most memorisable target in the set, for the fourth round: every value re-verified live matches to the character; IPv6 first-class (AAAA, RDAP ×2, origin AS ×2); both HSTS hosts with their different max-ages and "preload not set" for each; the AS number looked up per address from a routing view rather than inferred; raw values throughout
- **What it got wrong:** nothing
- **Change from the previous round:** **10 → 10 (0)**, at a third less cost (12 → 8 calls, 1,788 → 1,063 tokens). v3's one open item — four extra fetches per prompt for the origin AS — is closed by Change 55.2, and the answer is the same answer with the ASN's provenance now a routing view instead of a geolocation API. The caveats paragraph v3 wrote (vantage point, single handshake) is gone; the rubric's point-in-time line is still met
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes — as the confirmed close of v2's and v3's open item.** `lookup_ip_or_asn`'s routing section fired on four addresses in a real chat and removed four calls. Nothing new to fix on this prompt
## Grader notes
- Ground truth verification: re-run from the grading host as the rubric requires (2026-09-12 15:56 UTC). A: `104.16.132.229`, `104.16.133.229`. TLS: `issuer=C = US, O = Google Trust Services, CN = WE1`, `notBefore=Sep 5 22:29:39 2026 GMT`, `notAfter=Dec 4 23:29:33 2026 GMT`. HSTS apex (301) `max-age=15780000; includeSubDomains`; www (200) `max-age=31536000; includeSubDomains`; no `preload` on either. All match the model's values and the tool trail. The AAAA records, RDAP handles and the AS13335 / 104.16.128.0/20 / 2606:4700::/44 routing data match the rubric's v1-era independent verification (Team Cymru) and are unchanged
- Rubric issues: the v1-era correction to the HSTS ground truth (no `preload` token on either host; apex max-age 15780000) is re-confirmed live for the fourth round
docs/testing/prompt-corpus/hard/h10-live-lookup-chain/v4/result.txt
+268
View File
@@ -0,0 +1,268 @@
########################################################################
# owuitools prompt test result
# prompt dir : hard/h10-live-lookup-chain (round v4)
# chat id : 1e41d770-52b0-4989-8f3c-5a7bd9ede6a1
# chat title : Cloudflare DNS Resolution & TLS Certificate
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 8 lookup_ip_or_asn x4, dns_lookup x2, inspect_tls x1, http_headers x1
# dumped at : 2026-09-12T05:02:11Z
########################################################################
=== USER (2026-09-12T05:01:29Z) ===
For cloudflare.com, using live lookups: (1) what IPv4 and IPv6 addresses does it resolve to; (2) which ASN announces each of them and who owns that ASN; (3) who issued the current TLS certificate and when does it expire; (4) does it send HSTS, with what max-age, and are includeSubDomains and preload set? Give me the raw values you got, and don't tell me anything you didn't actually look up.
=== ASSISTANT (2026-09-12T05:01:29Z, model qwen3.8:latest) ===
usage: prompt_eval_count=12065 eval_count=692 prompt_tokens=12065 completion_tokens=692 total=15.9s prefill=2.2s decode=13.7s 50.6 tok/s
=== [1] THINKING ===
Let's execute lookups: A, AAAA, TLS inspection, HTTP headers. Then, for each IP address found, do RDAP lookup.
=== [2] TOOL CALL: dns_lookup (call_b2sa56fv) ===
{
"name": "cloudflare.com",
"record_type": "A"
}
=== [3] TOOL CALL: dns_lookup (call_yhx7l4rm) ===
{
"name": "cloudflare.com",
"record_type": "AAAA"
}
=== [4] TOOL CALL: inspect_tls (call_7t3501l1) ===
{
"hostname": "cloudflare.com"
}
=== [5] TOOL CALL: http_headers (call_1c1eynlq) ===
{
"url": "https://cloudflare.com/"
}
=== [6] TOOL RESULT: dns_lookup (call_b2sa56fv) ===
# DNS A records for cloudflare.com
- 104.16.132.229 (TTL 300s)
- 104.16.133.229 (TTL 300s)
[Resolved from the public internet, not from the user's local network.]
=== [7] TOOL RESULT: dns_lookup (call_yhx7l4rm) ===
# DNS AAAA records for cloudflare.com
- 2606:4700::6810:84e5 (TTL 300s)
- 2606:4700::6810:85e5 (TTL 300s)
[Resolved from the public internet, not from the user's local network.]
=== [8] TOOL RESULT: inspect_tls (call_7t3501l1) ===
# TLS certificate for cloudflare.com:443
Valid for another 83 days.
Certificate chain validates against the public trust store.
Subject: CN=cloudflare.com
Issuer: CN=WE1,O=Google Trust Services,C=US
Valid from: 2026-09-05 22:29 UTC
Valid until: 2026-12-04 23:29 UTC
Serial: d509925c8db10e1913a3a1cb1c7b05ab
Signature: ecdsa-with-SHA256
SHA-256: 96:10:34:eb:e7:3c:0a:ab:92:ee:5c:12:7a:42:65:b9:9a:bc:36:cd:f0:b5:3b:82:b1:07:88:e0:87:a7:2a:14
Negotiated: TLSv1.3, cipher TLS_AES_256_GCM_SHA384
Covers 5 name(s): cloudflare.com, ns.cloudflare.com, *.ns.cloudflare.com, *.secondary.cloudflare.com, secondary.cloudflare.com
=== [9] TOOL RESULT: http_headers (call_1c1eynlq) ===
# https://cloudflare.com/
## Redirect chain
301 https://cloudflare.com/
-> https://www.cloudflare.com/
200 https://www.cloudflare.com/ (final)
## Response headers
server: cloudflare
content-type: text/html; charset=utf-8
content-encoding: gzip
cache-control: max-age=10
last-modified: Sat, 12 Sep 2026 05:01:28 GMT
set-cookie: cfz_google-analytics_v4=%7B%22nzcr_ga4%22%3A%7B%22v%22%3A%22f672bc6c-a25f-496a-b272-8bd0bd2c36d3%22%2C%22e%22%3A1823313339601%7D%7D;Path=/;Max-Age=31104000;Secure;HttpOnly, cfz_adobe={"MsVJ_ecid":{"v"
cf-ray: a39c5afd494c27c1-BOS
age: 9
vary: accept-encoding
## Security headers (final response: https://www.cloudflare.com/)
strict-transport-security: max-age=31536000; includeSubDomains
max-age=31536000 (~365 days); includeSubDomains: yes; preload: no
content-security-policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://static.cloudflareinsights.com https://static-staging.cloudflareinsights.com https://challenges.cloudflare.com https://*.onetrust.com https://cdn.cookielaw.org https://ot.www.cloudflare.com https://www.googletagmanager.com ht… [1959 chars total]
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
referrer-policy: strict-origin-when-cross-origin
permissions-policy: geolocation=(), camera=(), microphone=()
Missing: none
Note: strict-transport-security differs between cloudflare.com (max-age=15780000; includeSubDomains) and www.cloudflare.com (max-age=31536000; includeSubDomains) - the apex and www are configured separately; report whichever the question is about, or both.
=== [10] THINKING ===
Next, I'll do RDAP lookups for each of the four IPs.
=== [11] TOOL CALL: lookup_ip_or_asn (call_1pnccl96) ===
{
"target": "104.16.132.229"
}
=== [12] TOOL CALL: lookup_ip_or_asn (call_knx3ke59) ===
{
"target": "104.16.133.229"
}
=== [13] TOOL CALL: lookup_ip_or_asn (call_uogo2f5n) ===
{
"target": "2606:4700::6810:84e5"
}
=== [14] TOOL CALL: lookup_ip_or_asn (call_aypsyd86) ===
{
"target": "2606:4700::6810:85e5"
}
=== [15] TOOL RESULT: lookup_ip_or_asn (call_1pnccl96) ===
# RDAP registration for 104.16.132.229 (IP address)
Name: CLOUDFLARENET
Handle: NET-104-16-0-0-1
Range: 104.16.0.0 - 104.31.255.255
CIDR: 104.16.0.0/12
Type: DIRECT ALLOCATION
Last Changed 2024-09-04
Registration 2014-03-28
Status: active
## Contacts
- registrant: Cloudflare, Inc.
- abuse: Abuse <abuse@cloudflare.com>
- technical: Admin <rir@cloudflare.com>
- noc: NOC <noc@cloudflare.com>
## Routing (BGP)
Announced prefix: 104.16.128.0/20
Origin AS: AS13335 (CLOUDFLARENET - Cloudflare, Inc.)
From RIPEstat's routing view (RIS collectors), https://stat.ripe.net/data/prefix-overview/data.json?resource=104.16.132.229
=== [16] TOOL RESULT: lookup_ip_or_asn (call_knx3ke59) ===
# RDAP registration for 104.16.133.229 (IP address)
Name: CLOUDFLARENET
Handle: NET-104-16-0-0-1
Range: 104.16.0.0 - 104.31.255.255
CIDR: 104.16.0.0/12
Type: DIRECT ALLOCATION
Last Changed 2024-09-04
Registration 2014-03-28
Status: active
## Contacts
- registrant: Cloudflare, Inc.
- abuse: Abuse <abuse@cloudflare.com>
- technical: Admin <rir@cloudflare.com>
- noc: NOC <noc@cloudflare.com>
## Routing (BGP)
Announced prefix: 104.16.128.0/20
Origin AS: AS13335 (CLOUDFLARENET - Cloudflare, Inc.)
From RIPEstat's routing view (RIS collectors), https://stat.ripe.net/data/prefix-overview/data.json?resource=104.16.133.229
=== [17] TOOL RESULT: lookup_ip_or_asn (call_uogo2f5n) ===
# RDAP registration for 2606:4700::6810:84e5 (IP address)
Name: CLOUDFLARENET
Handle: NET6-2606-4700-1
Range: 2606:4700:: - 2606:4700:ffff:ffff:ffff:ffff:ffff:ffff
CIDR: 2606:4700::/32
Type: DIRECT ALLOCATION
Last Changed 2024-09-04
Registration 2011-11-01
Status: active
## Contacts
- registrant: Cloudflare, Inc.
- abuse: Abuse <abuse@cloudflare.com>
- technical: Admin <rir@cloudflare.com>
- noc: NOC <noc@cloudflare.com>
## Routing (BGP)
Announced prefix: 2606:4700::/44
Origin AS: AS13335 (CLOUDFLARENET - Cloudflare, Inc.)
From RIPEstat's routing view (RIS collectors), https://stat.ripe.net/data/prefix-overview/data.json?resource=2606:4700::6810:84e5
=== [18] TOOL RESULT: lookup_ip_or_asn (call_aypsyd86) ===
# RDAP registration for 2606:4700::6810:85e5 (IP address)
Name: CLOUDFLARENET
Handle: NET6-2606-4700-1
Range: 2606:4700:: - 2606:4700:ffff:ffff:ffff:ffff:ffff:ffff
CIDR: 2606:4700::/32
Type: DIRECT ALLOCATION
Last Changed 2024-09-04
Registration 2011-11-01
Status: active
## Contacts
- registrant: Cloudflare, Inc.
- abuse: Abuse <abuse@cloudflare.com>
- technical: Admin <rir@cloudflare.com>
- noc: NOC <noc@cloudflare.com>
## Routing (BGP)
Announced prefix: 2606:4700::/44
Origin AS: AS13335 (CLOUDFLARENET - Cloudflare, Inc.)
From RIPEstat's routing view (RIS collectors), https://stat.ripe.net/data/prefix-overview/data.json?resource=2606:4700::6810:85e5
=== [19] THINKING (0s) ===
All data has been collected. I'll report the raw values, and note that the HSTS differs between the apex and www.
=== [20] ASSISTANT TEXT ===
All four lookups are live; here are the raw values.
**1. DNS resolution** (TTL 300s on all records)
- A: `104.16.132.229`, `104.16.133.229`
- AAAA: `2606:4700::6810:84e5`, `2606:4700::6810:85e5`
**2. ASN / ownership** — all four addresses fall in the same registration:
- IPv4: in `104.16.0.0/12` (handle NET-104-16-0-0-1, registered 2014-03-28); announced prefix `104.16.128.0/20`, origin **AS13335 CLOUDFLARENET – Cloudflare, Inc.**
- IPv6: in `2606:4700::/32` (handle NET6-2606-4700-1, registered 2011-11-01); announced prefix `2606:4700::/44`, origin **AS13335 CLOUDFLARENET – Cloudflare, Inc.**
- Registrant on both: Cloudflare, Inc. (abuse@cloudflare.com)
**3. TLS certificate** (chain validates against the public trust store)
- Subject: `CN=cloudflare.com`
- Issuer: **Google Trust Services** (`CN=WE1, O=Google Trust Services, C=US`)
- Valid: **2026-09-05 22:29 UTC → 2026-12-04 23:29 UTC** (~83 days remaining at lookup time)
- Signature: ECDSA with SHA-256; negotiated TLSv1.3 / TLS_AES_256_GCM_SHA384
- SANs (5): cloudflare.com, ns.cloudflare.com, *.ns.cloudflare.com, *.secondary.cloudflare.com, secondary.cloudflare.com
**4. HSTS** — yes, but note the apex and www send different values:
- **cloudflare.com** (the host you asked about): `strict-transport-security: max-age=15780000; includeSubDomains` → max-age = 15,780,000 s (~182.5 days), includeSubDomains **set**, preload **not set**
- www.cloudflare.com (where the 301 redirect lands): `max-age=31536000; includeSubDomains` (~365 days), includeSubDomains set, preload not set
So for cloudflare.com specifically: HSTS present, max-age 15780000, includeSubDomains yes, preload no.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 5.1 s
- Thought: Let's execute lookups: A, AAAA, TLS inspection, HTTP headers. Then, for each IP address found, do RDAP lookup.
- Calling dns_lookup
- Calling dns_lookup
- Calling inspect_tls
- Calling http_headers
- Thinking
- Thought: Next, I'll do RDAP lookups for each of the four IPs.
- Calling lookup_ip_or_asn
- Calling lookup_ip_or_asn
- Calling lookup_ip_or_asn
- Calling lookup_ip_or_asn
- Thinking
- Thought: All data has been collected. I'll report the raw values, and note that the HSTS differs between the apex and www.
- Writing response
- Done · 8 tool calls · prefill 5.0 s for 12,832 tokens · 1,063 generated