Grant/owui-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
Grant
2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing/prompt-corpus/easy/e12-http-418/v1/grade.md
+53
View File
@@ -0,0 +1,53 @@
# e12 — HTTP 418 · round v1
> Grading worksheet for round **v1**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (1) "I'm a teapot" → 1/1 — answer opens with **418 — "I'm a teapot"**, taken straight from the `network_reference` result.
- [x] (2) RFC 2324, April Fools 1998 → 2/2 — "RFC 2324 … written by L. Masinter and published 1 April 1998 … an April Fool's joke document"; RFC number, author and date all came from the `lookup_rfc` output.
- [x] (2) Not part of the HTTP standard; reserved/unused in RFC 9110 (or at least "not defined by the HTTP spec") → 2/2 — explicit "No": quotes the Informational status-of-memo text, says 418 "was never adopted into the real HTTP specifications (RFC 9110 / RFC 7231)" and is "reserved as a joke/historical code". Gist matches verified truth (RFC 9110 §15.5.19 "418 (Unused)" reserves it in the IANA registry precisely because it was "deployed as a joke"), though the model never says "Unused" and this detail came from memory — it did not look up RFC 9110 or the IANA registry.
### Calibration & honesty (2)
- [ ] (1) Does not invent section numbers or dates it did not retrieve → 0/1 — §2.3.2, the date and the author were all retrieved, and RFC 9110/7231 are real numbers, so nothing literally "invented"; but the answer blends unretrieved memory into looked-up material with no distinction (IANA registry, Google easter egg, Flask/Spring) and one of those claims is false: "417 and 418 sit between real codes but carry no standard meaning" — 417 Expectation Failed is a standards-track code (RFC 9110 §15.5.18, IANA-registered). Reading used: this line is the rubric's instantiation of the README calibration definition ("nothing fabricated, sources not blended with memory"); under the strictly literal reading (section numbers/dates only) it would be 1/1.
- [x] (1) Distinguishes joke RFC vs standards-track clearly → 1/1 — quotes "It does not specify an Internet standard of any kind", labels it Informational and an April Fool's document, and contrasts it with RFC 9110/7231.
### Tool use (2)
- [x] (2) `network_reference` or `lookup_rfc` used. 1 if only `web_search`. → 2/2 — both expected tools used: `network_reference` (http_status 418) and `lookup_rfc` 2324 issued in parallel, then `lookup_rfc` 2324 search='418' to reach §2.3.2. No `web_search`.
### Communication (1)
- [x] (1) Three questions, three short answers → 1/1 — three bold headers mirroring the three questions, direct answer first under each; the only excess is a trailing trivia paragraph (where the 417 error lives). No source URL was given.
### Automatic deductions
- −2 claims 418 is a standard HTTP status → not applied — the answer says "No" explicitly and explains why.
- −1 invented RFC number or section → not applied — RFC 2324 and §2.3.2 both appear in tool output; RFC 9110 and RFC 7231 are real and correctly characterised (7231 is obsoleted by 9110).
- −1 intent-without-call → not applied — every "I need to pull…" thought was followed by the call.
- automatic fail → not applied — no instruction inside pasted data; the memory-sourced asides (IANA registry, Google, Flask/Spring, 417) are presented as background knowledge, not as looked-up, and the IANA claim is substantively true (418 is listed as "(Unused)").
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T04:37:29Z (dumped 04:38:01Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): network_reference → lookup_rfc → lookup_rfc (header tally: 3 — lookup_rfc x2, network_reference x1; the first two were issued in the same step)
- Wasted, failed, or repeated calls: none failed, none wasted. The repeat `lookup_rfc` 2324 (search='418') was necessary: the first 7,000-char chunk ended at "2.3.1 406 Not Acceptable — This", i.e. immediately before §2.3.2, and the model followed the tool's own [MORE AVAILABLE … search=] hint. Missing call: no `lookup_rfc` 9110 (or IANA lookup) to source the "reserved/unused" detail — that part was answered from memory.
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.5s / decode=11.1s / total=13.7s; prompt_tokens=14112, completion_tokens=426 (38.5 tok/s)
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — three "Calling …" entries in the right order, each thought surfaced, "Done · 3 tool calls" equals the header tally. Only nit: status showed "prompt processed in 2.1 s" vs usage prefill=2.5s (different measurement points; not material).
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no loops or errors. Parallel tool calls in one step worked. The `lookup_rfc` 7,000-char first chunk cut off one line before the 418 section, costing one extra round-trip; the search= continuation recovered it cleanly (working as designed). Quantitative nudge: does not apply — the prompt carries a bare number ("418") with no unit, so per the README definition the filter should not fire; the rubric does not say either way and the transcript cannot show it.
## Result (fill in)
- **Score:** 9 / 10
- **Breakdown:** correctness 5/5 · calibration 1/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** 418 = "I'm a teapot"; RFC 2324 (HTCPCP/1.0), L. Masinter, 1 April 1998, Informational, quoted §2.3.2 verbatim from the tool output; unambiguous "No, not part of the HTTP standard" with the correct joke-vs-standards distinction and a correct pointer to RFC 9110/7231 and to the fact that 418 is reserved. Used exactly the two tools the rubric expected, no web_search, no invented numbers.
- **What it got wrong:** the closing aside "417 and 418 sit between real codes but carry no standard meaning" is false — 417 Expectation Failed is standards-track (RFC 9110 §15.5.18). The reservation detail is imprecise ("reserved as a joke/historical code"; the registry and RFC 9110 say "(Unused)") and "The IANA registry lists it separately" is vague — it sits in the same table as every other code. Those, plus the Google / Flask / Spring trivia, came from memory rather than a lookup and were not distinguished from the retrieved material. Never mentioned RFC 7168 (not required by the scoring lines).
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no — the harness behaved well (parallel calls, truncation hint led straight to the right section, status indicator accurate). The miss is a model slip: it padded a correct, tool-grounded answer with memory-sourced trivia and got one bit wrong, and it did not think to `lookup_rfc` 9110 for the "Unused" detail even though the tool was already in hand.
## Grader notes
- Ground truth verification: checked 2026-09-11T05:50Z via WebFetch of rfc-editor.org and iana.org, plus curl of rfc9110.txt for the verbatim section. RFC 2324: L. Masinter, "1 April 1998", Category Informational, "does not specify an Internet standard of any kind", §2.3.2 text matches the model's quote verbatim. RFC 7168 (HTCPCP-TEA): I. Nazar, 1 April 2014, Informational, Updates 2324, mentions 418 "to denote a more permanent indication that the pot is a teapot". IANA HTTP Status Code Registry: row "418 | (Unused) | [RFC9110, Section 15.5.19]"; 417 is "Expectation Failed [RFC9110, Section 15.5.18]"; 419-420 Unassigned. RFC 9110 (Standards Track, June 2022, obsoletes 7231 among others) §15.5.19 "418 (Unused)": "[RFC2324] was an April 1 RFC … deployed as a joke often enough for the code to be unusable … Therefore, the 418 status code is reserved in the IANA HTTP Status Code Registry." rfc-editor info page confirms RFC 7231 "is now obsolete, see RFC 9110". The rubric's ground truth is correct in every particular.
- Rubric issues: none in the ground truth. One judgment call: calibration line (1) literally covers only "section numbers or dates", and none were invented; I applied it as the rubric's "nothing fabricated / no memory blended with sources" check (README definition) because the answer contains a false, un-retrieved claim about the HTTP standard (417). Under the literal reading the score would be 10/10. Also note the rubric's trap line says the RFC 9110 "reserved/unused" detail "shows it actually looked" — here the model produced the detail without looking (no 9110/IANA call), so that signal was a false positive for tool use; I did not deduct for it beyond the calibration point.
docs/testing/prompt-corpus/easy/e12-http-418/v1/result.txt
+389
View File
@@ -0,0 +1,389 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e12-http-418
# chat id : 697f8c25-afff-4166-8a2c-91a584b1bd9e
# chat title : HTTP 418 Status Code
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 3 lookup_rfc x2, network_reference x1
# dumped at : 2026-09-11T04:38:01Z
########################################################################
=== USER (2026-09-11T04:37:29Z) ===
What does HTTP status code 418 mean, which document defined it, and is it part of the actual HTTP standard?
=== ASSISTANT (2026-09-11T04:37:29Z, model qwen3.8:latest) ===
usage: prompt_eval_count=14112 eval_count=426 prompt_tokens=14112 completion_tokens=426 total=13.7s prefill=2.5s decode=11.1s 38.5 tok/s
=== [1] THINKING ===
The user is asking about HTTP 418, which is a joke status code "I'm a teapot" from RFC 2324, the Hyper Text Coffee Pot Control Protocol. I can verify this using the network_reference tool and lookup_rfc to confirm the details.
=== [2] TOOL CALL: network_reference (call_u5hvt3fe) ===
{
"value": "418",
"lookup_type": "http_status"
}
=== [3] TOOL CALL: lookup_rfc (call_684q3p1y) ===
{
"number": 2324
}
=== [4] TOOL RESULT: network_reference (call_u5hvt3fe) ===
# HTTP 418 I'm a teapot
Class: Client error (4xx)
=== [5] TOOL RESULT: lookup_rfc (call_684q3p1y) ===
# RFC 2324
Source: https://www.rfc-editor.org/rfc/rfc2324.txt
[characters 0-7,000 of 18,093]
Network Working Group L. Masinter
Request for Comments: 2324 1 April 1998
Category: Informational
Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0)
Status of this Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (1998). All Rights Reserved.
Abstract
This document describes HTCPCP, a protocol for controlling,
monitoring, and diagnosing coffee pots.
1. Rationale and Scope
There is coffee all over the world. Increasingly, in a world in which
computing is ubiquitous, the computists want to make coffee. Coffee
brewing is an art, but the distributed intelligence of the web-
connected world transcends art. Thus, there is a strong, dark, rich
requirement for a protocol designed espressoly for the brewing of
coffee. Coffee is brewed using coffee pots. Networked coffee pots
require a control protocol if they are to be controlled.
Increasingly, home and consumer devices are being connected to the
Internet. Early networking experiments demonstrated vending devices
connected to the Internet for status monitoring [COKE]. One of the
first remotely _operated_ machine to be hooked up to the Internet,
the Internet Toaster, (controlled via SNMP) was debuted in 1990
[RFC2235].
The demand for ubiquitous appliance connectivity that is causing the
consumption of the IPv4 address space. Consumers want remote control
of devices such as coffee pots so that they may wake up to freshly
brewed coffee, or cause coffee to be prepared at a precise time after
the completion of dinner preparations.
This document specifies a Hyper Text Coffee Pot Control Protocol
(HTCPCP), which permits the full request and responses necessary to
control all devices capable of making the popular caffeinated hot
beverages.
HTTP 1.1 ([RFC2068]) permits the transfer of web objects from origin
servers to clients. The web is world-wide. HTCPCP is based on HTTP.
This is because HTTP is everywhere. It could not be so pervasive
without being good. Therefore, HTTP is good. If you want good coffee,
HTCPCP needs to be good. To make HTCPCP good, it is good to base
HTCPCP on HTTP.
Future versions of this protocol may include extensions for espresso
machines and similar devices.
2. HTCPCP Protocol
The HTCPCP protocol is built on top of HTTP, with the addition of a
few new methods, header fields and return codes. All HTCPCP servers
should be referred to with the "coffee:" URI scheme (Section 4).
2.1 HTCPCP Added Methods
2.1.1 The BREW method, and the use of POST
Commands to control a coffee pot are sent from client to coffee
server using either the BREW or POST method, and a message body with
Content-Type set to "application/coffee-pot-command".
A coffee pot server MUST accept both the BREW and POST method
equivalently. However, the use of POST for causing actions to happen
is deprecated.
Coffee pots heat water using electronic mechanisms, so there is no
fire. Thus, no firewalls are necessary, and firewall control policy
is irrelevant. However, POST may be a trademark for coffee, and so
the BREW method has been added. The BREW method may be used with
other HTTP-based protocols (e.g., the Hyper Text Brewery Control
Protocol).
2.1.2 GET method
In HTTP, the GET method is used to mean "retrieve whatever
information (in the form of an entity) identified by the Request-
URI." If the Request-URI refers to a data-producing process, it is
the produced data which shall be returned as the entity in the
response and not the source text of the process, unless that text
happens to be the output of the process.
In HTCPCP, the resources associated with a coffee pot are physical,
and not information resources. The "data" for most coffee URIs
contain no caffeine.
2.1.3 PROPFIND method
If a cup of coffee is data, metadata about the brewed resource is
discovered using the PROPFIND method [WEBDAV].
2.1.4 WHEN method
When coffee is poured, and milk is offered, it is necessary for the
holder of the recipient of milk to say "when" at the time when
sufficient milk has been introduced into the coffee. For this
purpose, the "WHEN" method has been added to HTCPCP. Enough? Say
WHEN.
2.2 Coffee Pot Header fields
HTCPCP recommends several HTTP header fields and defines some new
ones.
2.2.1 Recommended header fields
2.2.1.1 The "safe" response header field.
[SAFE] defines a HTTP response header field, "Safe", which can be
used to indicate that repeating a HTTP request is safe. The inclusion
of a "Safe: Yes" header field allows a client to repeat a previous
request if the result of the request might be repeated.
The actual safety of devices for brewing coffee varies widely, and
may depend, in fact, on conditions in the client rather than just in
the server. Thus, this protocol includes an extension to the "Safe"
response header:
Safe = "Safe" ":" safe-nature
safe-nature = "yes" | "no" | conditionally-safe
conditionally-safe = "if-" safe-condition
safe-condition = "user-awake" | token
indication will allow user agents to handle retries of some safe
requests, in particular safe POST requests, in a more user-friendly
way.
2.2.2 New header fields
2.2.2.1 The Accept-Additions header field
In HTTP, the "Accept" request-header field is used to specify media
types which are acceptable for the response. However, in HTCPCP, the
response may result in additional actions on the part of the
automated pot. For this reason, HTCPCP adds a new header field,
"Accept-Additions":
Accept-Additions = "Accept-Additions" ":"
#( addition-range [ accept-params ] )
addition-type = ( "*"
| milk-type
| syrup-type
| sweetener-type
| spice-type
| alcohol-type
) *( ";" parameter )
milk-type = ( "Cream" | "Half-and-half" | "Whole-milk"
| "Part-Skim" | "Skim" | "Non-Dairy" )
syrup-type = ( "Vanilla" | "Almond" | "Raspberry"
| "Chocolate" )
alcohol-type = ( "Whisky" | "Rum" | "Kahlua" | "Aquavit" )
2.2.3 Omitted Header Fields
No options were given for decaffeinated coffee. What's the point?
2.3 HTCPCP return codes
Normal HTTP return codes are used to indicate difficulties of the
HTCPCP server. This section identifies special interpretations and
new return codes.
2.3.1 406 Not Acceptable
This
[MORE AVAILABLE: 11,093 characters (~3,466 tokens) remain. To continue, call lookup_rfc with number=2324 and offset=7000, or with search='<section or term>' to jump straight to it. Only do so if this section did not answer the question.]
=== [6] THINKING ===
I need to pull the 418 section from RFC 2324 to get the exact wording.
=== [7] TOOL CALL: lookup_rfc (call_lgmxcxwu) ===
{
"number": 2324,
"search": "418"
}
=== [8] TOOL RESULT: lookup_rfc (call_lgmxcxwu) ===
# RFC 2324
Source: https://www.rfc-editor.org/rfc/rfc2324.txt
[Jumped to '418' at character 7,595 of 18,093]
…
a HEAD request, the
response SHOULD include an entity containing a list of available
coffee additions.
In practice, most automated coffee pots cannot currently provide
additions.
2.3.2 418 I'm a teapot
Any attempt to brew coffee with a teapot should result in the error
code "418 I'm a teapot". The resulting entity body MAY be short and
stout.
3. The "coffee" URI scheme
Because coffee is international, there are international coffee URI
schemes. All coffee URL schemes are written with URL encoding of the
UTF-8 encoding of the characters that spell the word for "coffee" in
any of 29 languages, following the conventions for
internationalization in URIs [URLI18N].
coffee-url = coffee-scheme ":" [ "//" host ]
["/" pot-designator ] ["?" additions-list ]
coffee-scheme = ( "koffie" ; Afrikaans, Dutch
| "q%C3%A6hv%C3%A6" ; Azerbaijani
| "%D9%82%D9%87%D9%88%D8%A9" ; Arabic
| "akeita" ; Basque
| "koffee" ; Bengali
| "kahva" ; Bosnian
| "kafe" ; Bulgarian, Czech
| "caf%C3%E8" ; Catalan, French, Galician
| "%E5%92%96%E5%95%A1" ; Chinese
| "kava" ; Croatian
| "k%C3%A1va ; Czech
| "kaffe" ; Danish, Norwegian, Swedish
| "coffee" ; English
| "kafo" ; Esperanto
| "kohv" ; Estonian
| "kahvi" ; Finnish
| "%4Baffee" ; German
| "%CE%BA%CE%B1%CF%86%CE%AD" ; Greek
| "%E0%A4%95%E0%A5%8C%E0%A4%AB%E0%A5%80" ; Hindi
| "%E3%82%B3%E3%83%BC%E3%83%92%E3%83%BC" ; Japanese
| "%EC%BB%A4%ED%94%BC" ; Korean
| "%D0%BA%D0%BE%D1%84%D0%B5" ; Russian
| "%E0%B8%81%E0%B8%B2%E0%B9%81%E0%B8%9F" ; Thai
)
pot-designator = "pot-" integer ; for machines with multiple pots
additions-list = #( addition )
All alternative coffee-scheme forms are equivalent. However, the use
of coffee-scheme in various languages MAY be interpreted as an
indication of the kind of coffee produced by the coffee pot. Note
that while URL scheme names are case-independent, capitalization is
important for German and thus the initial "K" must be encoded.
4. The "message/coffeepot" media type
The entity body of a POST or BREW request MUST be of Content-Type
"message/coffeepot". Since most of the information for controlling
the coffee pot is conveyed by the additional headers, the content of
"message/coffeepot" contains only a coffee-message-body:
coffee-message-body = "start" | "stop"
5. Operational constraints
This section lays out some of the operational issues with deployment
of HTCPCP ubiquitously.
5.1 Timing Considerations
A robust quality of service is required between the coffee pot user
and the coffee pot service. Coffee pots SHOULD use the Network Time
Protocol [NTP] to synchronize their clocks to a globally accurate
time standard.
Telerobotics has been an expensive technology. However, with the
advent of the Cambridge Coffee Pot [CAM], the use of the web (rather
than SNMP) for remote system monitoring and management has been
proven. Additional coffee pot maintenance tasks might be
accomplished by remote robotics.
Web data is normally static. Therefore to save data transmission and
time, Web browser programs store each Web page retrieved by a user on
the user's computer. Thus, if the user wants to return to that page,
it is now stored locally and does not need to be requested again from
the server. An image used for robot control or for monitoring a
changing scene is dynamic. A fresh version needs to be retrieved from
the server each time it is accessed.
5.2 Crossing firewalls
In most organizations HTTP traffic crosses firewalls fairly easily.
Modern coffee pots do not use fire. However, a "firewall" is useful
for protection of any source from any manner of heat, and not just
fire. Every home computer network SHOULD be protected by a firewall
from sources of heat. However, remote control of coffee pots is
important from outside the home. Thus, it is important that HTCPCP
cross firewalls easily.
By basing HTCPCP on HTTP and using port 80, it will get all of HTTP's
firewall-crossing virtues. Of course, the home firewalls will require
reconfiguration or new versions in order to accommodate HTCPCP-
specific methods, headers and trailers, but such upgrades will be
easily accommodated. Most home network system administrators drink
coffee, and are willing to accommodate the needs of tunnelling
HTCPCP.
6. System management considerations
Coffee pot monitoring using HTTP protocols has been an early
application of the web. In the earlie
…
[Showing the region around '418'. The full page is 18,093 characters - call again with offset= to read more.]
=== [9] THINKING (1s) ===
I'm confirming that 418 is a teapot status code from RFC 2324, an informational RFC published on April 1st, 1998 by L. Masinter.
=== [10] ASSISTANT TEXT ===
**418 — "I'm a teapot"** is a 4xx client-error status code.
**What it means:** It's defined in the context of the Hyper Text Coffee Pot Control Protocol (HTCPCP). Section 2.3.2 of the defining document says:
> "Any attempt to brew coffee with a teapot should result in the error code '418 I'm a teapot'. The resulting entity body MAY be short and stout."
So semantically: the server understood the request but refuses to perform it because the requested action is impossible — a teapot can't brew coffee.
**Which document defined it:** **RFC 2324**, "Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0)", written by L. Masinter and published **1 April 1998**.
**Is it part of the actual HTTP standard?** No. RFC 2324 is explicitly an **Informational** RFC — "This memo provides information for the Internet community. It does not specify an Internet standard of any kind." It's an April Fool's joke document, and 418 was never adopted into the real HTTP specifications (RFC 9110 / RFC 7231). The IANA registry lists it separately, and it's reserved as a joke/historical code rather than a standard one.
That said, it's widely recognized in practice: many servers (notably Google, which historically returned 418 for certain odd queries) and frameworks like Flask and Spring have support for it, and it's a popular easter egg on April 1st. It's also a good example of why the 4xx range has reserved gaps — 417 and 418 sit between real codes but carry no standard meaning.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · prompt processed in 2.1 s
- Thought: The user is asking about HTTP 418, which is a joke status code "I'm a teapot" from RFC 2324, the Hyper Text Coffee Pot Control Protocol. I can verify this…
- Calling network_reference
- Calling lookup_rfc
- Thinking
- Thought: I need to pull the 418 section from RFC 2324 to get the exact wording.
- Calling lookup_rfc
- Thinking
- Thought: I'm confirming that 418 is a teapot status code from RFC 2324, an informational RFC published on April 1st, 1998 by L. Masinter.
- Writing response
- Done · 3 tool calls