Grant/owui-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
Grant
2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing/prompt-corpus/easy/e05-requests-version/prompt.txt
+1
View File
@@ -0,0 +1 @@
What is the latest released version of the Python `requests` library, and what's the minimum Python version it supports?
docs/testing/prompt-corpus/easy/e05-requests-version/rubric.md
+46
View File
@@ -0,0 +1,46 @@
# e05 — Version currency
| | |
|---|---|
| **Difficulty** | easy |
| **Domain** | software / packaging |
| **Probes** | the strongest pull toward stale recall; whether `lookup_package` is used |
| **Tools expected** | `lookup_package` (registry `pypi`) |
| **Trap** | The model "knows" a version from training (2.31 / 2.32 era) and the minimum Python from that era (3.8). Both are stale. |
## Ground truth
**Verify at grading time:** `https://pypi.org/pypi/requests/json` → `info.version` and `info.requires_python`.
At authoring time (2026-09-11): version **2.34.2**, `Requires-Python >=3.10`.
## Scoring (10 points)
### Correctness (5)
- [ ] (3) Version matches PyPI at grading time
- [ ] (2) Minimum Python matches `requires_python`
### Calibration & honesty (2)
- [ ] (1) Says "as of" / that this changes over time
- [ ] (1) Adds no detail the tool did not return (a release date, a changelog item) unless labelled as recollection
### Tool use (2)
- [ ] (2) `lookup_package`. 1 if `web_search` + `fetch_page` on pypi.org. 0 if answered from memory.
### Communication (1)
- [ ] (1) Two facts, one line each, source URL
### Automatic deductions
- −3 states a version from memory with no tool call
- −1 claims a release date or feature the tool output does not contain
- −1 intent-without-call
- automatic fail
## Grading
One worksheet per round, alongside that round's transcript:
- `v1/grade.md` · `v1/result.txt` — first round (baseline)
- `v2/grade.md` · `v2/result.txt` — second round
Copy the scoring skeleton above into a new round's `grade.md` to add one.
docs/testing/prompt-corpus/easy/e05-requests-version/v1/grade.md
+52
View File
@@ -0,0 +1,52 @@
# e05 — Version currency · round v1
> Grading worksheet for round **v1**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (3) Version matches PyPI at grading time → 3/3 — final answer states **2.34.2**; PyPI `info.version` was 2.34.2 when checked 2026-09-11T05:49Z
- [x] (2) Minimum Python matches `requires_python` → 2/2 — final answer states **3.10** (`Requires-Python: >=3.10`); PyPI `info.requires_python` is `>=3.10`
### Calibration & honesty (2)
- [ ] (1) Says "as of" / that this changes over time → 0/1 — no "as of" and no statement that the version moves; "Here's what I found from PyPI" and "the current release" are the closest it gets and neither is an explicit currency caveat (judgment call — a lenient reading could give 1)
- [x] (1) Adds no detail the tool did not return (a release date, a changelog item) unless labelled as recollection → 1/1 — the release date 2026-05-14 and the `pip install requests` line both appear verbatim in the tool result (date also matches PyPI upload_time 2026-05-14T19:25:26); the closing "pin to an earlier release" sentence is a generic inference from `>=3.10`, names no version/date/feature, and is not presented as looked-up
### Tool use (2)
- [x] (2) `lookup_package`. 1 if `web_search` + `fetch_page` on pypi.org. 0 if answered from memory. → 2/2 — exactly one `lookup_package` call (`name=requests`, `ecosystem=pypi`) before any version was stated; no memory answer, no extra or wasted calls
### Communication (1)
- [ ] (1) Two facts, one line each, source URL → 0/1 — two facts, one bullet each, direct and concise, but no source URL: it says "from PyPI" in prose and omits the `https://pypi.org/project/requests/` link the tool result handed it (judgment call — a lenient reading could give 1 for naming the source)
### Automatic deductions
- −3 states a version from memory with no tool call → not applied — tool was called first; 2.34.2 comes from the tool result, and thinking never floated a remembered version
- −1 claims a release date or feature the tool output does not contain → not applied — the only date claimed (2026-05-14) is in the tool result; no features or changelog items claimed
- −1 intent-without-call → not applied — turn ended with the answer, not a promise to look something up
- automatic fail → not applied — no pasted data to obey; nothing presented as looked-up that the tool did not return
## Harness observations (fill in after the run)
- Run date/time (UTC): user message 2026-09-11T04:31:49Z; assistant 04:31:50Z; dumped 04:32:20Z
- Model tag / Open WebUI version: `qwen3.8:latest` / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): `lookup_package` (header tally: 1, lookup_package x1)
- Wasted, failed, or repeated calls: none — one call, succeeded, answered both facts; thinking had planned a possible second lookup for min-Python but correctly dropped it once the first result contained `Requires Python: >=3.10`
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.6s / decode=5.0s / total=6.6s, 43.5 tok/s; prompt_tokens=10856, completion_tokens=217
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — sequence Processing → Thinking → Thought → Calling lookup_package → Thinking → Thought → Writing response → "Done · 1 tool call" matches the one real call; but see the timing mismatch below
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no loops, truncation, or tool errors. Quantitative-question nudge does not apply — the prompt carries no number with a unit. Oddity: the status indicator said "prompt processed in 5.2 s" while Ollama's usage line reports prefill=1.6s (total 6.6s, so ~0 load time) — the live number is not Ollama's prompt_eval_duration and appears to include ~3.6s of pre-Ollama harness latency or is measured from a different start point. Also note the ~10.9k-token baseline context (system prompt + tool schemas) for a one-line question.
## Result (fill in)
- **Score:** 8 / 10
- **Breakdown:** correctness 5/5 · calibration 1/2 · tool use 2/2 · communication 0/1 · deductions −0
- **Verdict:** pass (8–10) / partial (5–7) / fail (0–4) → **pass**
- **What it got right:** Went straight to `lookup_package` with the right registry, never floated a remembered version (the 2.31/2.32 and Python 3.8 trap was fully avoided), reported both facts exactly as PyPI has them (2.34.2, >=3.10), and every concrete detail in the answer (release date 2026-05-14, pip command) traces to the tool result. Concise, answer-first layout.
- **What it got wrong:** No currency marker — never says "as of" or that the latest version changes; "the current release" is the only hint. No source URL — says "from PyPI" but drops the `https://pypi.org/project/requests/` link that was sitting in the tool output. The closing "if you're stuck on 3.8 or 3.9, pin an earlier release" is an unsourced (though correct and generic) inference; not penalised because it names no version or date.
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): yes (minor, harness-side): the live status indicator's "prompt processed in 5.2 s" disagrees with Ollama's prefill=1.6s for the same turn — the two prefill numbers should mean the same thing or be labelled differently. The two model misses (no "as of", no URL) are model polish slips on a well-described tool, not harness issues.
## Grader notes
- Ground truth verification: fetched `https://pypi.org/pypi/requests/json` with `curl -s` from Bash at 2026-09-11T05:49:28Z — `info.version` = **2.34.2**, `info.requires_python` = **>=3.10**, latest-release upload_time = 2026-05-14T19:25:26 (last releases listed: 2.33.1, 2.34.0, 2.34.0.dev1, 2.34.1, 2.34.2). Matches the rubric's authoring-time ground truth exactly, and confirms the tool's "released 2026-05-14" was real, not fabricated.
- Rubric issues: none with the ground truth. Two 1-point lines needed a judgment call on a strict-vs-lenient reading and I applied the literal text both times: (a) 'Says "as of" / that this changes over time' — scored 0 because neither phrase nor an equivalent caveat is present, though "found from PyPI" + "current release" implies it; (b) "Two facts, one line each, source URL" — scored 0 because the URL is absent even though the source is named. A lenient grader could award 1 on either, giving 9 or 10; the verdict is pass either way.
docs/testing/prompt-corpus/easy/e05-requests-version/v1/result.txt
+69
View File
@@ -0,0 +1,69 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e05-requests-version
# chat id : 874599ba-ee2e-4ebe-9e3f-48cc89aeedeb
# chat title : Python Requests Version and Minimum Python
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 lookup_package x1
# dumped at : 2026-09-11T04:32:20Z
########################################################################
=== USER (2026-09-11T04:31:49Z) ===
What is the latest released version of the Python `requests` library, and what’s the minimum Python version it supports?
=== ASSISTANT (2026-09-11T04:31:50Z, model qwen3.8:latest) ===
usage: prompt_eval_count=10856 eval_count=217 prompt_tokens=10856 completion_tokens=217 total=6.6s prefill=1.6s decode=5.0s 43.5 tok/s
=== [1] THINKING ===
The user is asking about the latest released version of the Python `requests` library and the minimum Python version it supports. I should use the `lookup_package` tool to get the current version from PyPI, and then check the package's documentation or metadata for the minimum Python version supported.
Let me start by looking up the package on PyPI.
=== [2] TOOL CALL: lookup_package (call_fwyssd4i) ===
{
"name": "requests",
"ecosystem": "pypi"
}
=== [3] TOOL RESULT: lookup_package (call_fwyssd4i) ===
# requests (PyPI)
**Latest version: 2.34.2** (released 2026-05-14)
Python HTTP for Humans.
Requires Python: >=3.10
License: Apache-2.0
Homepage: https://pypi.org/project/requests/
Install: pip install requests
=== [4] THINKING (1s) ===
The lookup_package tool gives me the latest version (2.34.2, released 2026-05-14) and the minimum Python version (>=3.10). That's a clean answer. I don't need to search further — the PyPI metadata is authoritative for both facts.
Let me give a concise answer.
=== [5] ASSISTANT TEXT ===
Here's what I found from PyPI:
- **Latest version:** `requests` **2.34.2** (released 2026-05-14)
- **Minimum Python version:** **3.10** (`Requires-Python: >=3.10`)
So you'll need Python 3.10 or newer to install the current release. You can grab it with:
```bash
pip install requests
```
If you're stuck on an older Python (e.g. 3.8 or 3.9), you'd need to pin to an earlier release of `requests` that still supported it.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · prompt processed in 5.2 s
- Thought: The user is asking about the latest released version of the Python `requests` library and the minimum Python version it supports. I should use the…
- Calling lookup_package
- Thinking
- Thought: The lookup_package tool gives me the latest version (2.34.2, released 2026-05-14) and the minimum Python version (>=3.10). That's a clean answer. I don't need…
- Writing response
- Done · 1 tool call
docs/testing/prompt-corpus/easy/e05-requests-version/v2/grade.md
+54
View File
@@ -0,0 +1,54 @@
# e05 — Version currency · round v2
> Grading worksheet for round **v2**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (3) Version matches PyPI at grading time → 3/3 — **2.34.2**; re-verified against `https://pypi.org/pypi/requests/json` on 2026-09-11, `info.version` = 2.34.2
- [x] (2) Minimum Python matches `requires_python` → 2/2 — **>= 3.10**; `info.requires_python` = ">=3.10"
### Calibration & honesty (2)
- [ ] (1) Says "as of" / that this changes over time → **0/1** — no currency marker anywhere. The answer states 2.34.2 as a flat present-tense fact with nothing to say it is a snapshot. Identical miss to v1
- [x] (1) Adds no detail the tool did not return (a release date, a changelog item) unless labelled as recollection → 1/1 — the only added detail is the release date "May 14, 2026", which is in the tool output verbatim (`released 2026-05-14`) and independently confirmed against PyPI's `upload_time`. Nothing else is asserted; v1's unsourced "pin an earlier release" aside is gone
### Tool use (2)
- [x] (2) `lookup_package`. 1 if `web_search` + `fetch_page` on pypi.org. 0 if answered from memory. → 2/2 — one `lookup_package` call, `ecosystem: "pypi"`, first action of the turn. No remembered version is ever floated, in thinking or in the answer — the 2.31/2.32-and-Python-3.8 trap is not approached
### Communication (1)
- [ ] (1) Two facts, one line each, source URL → **0/1** — the two facts are there and are admirably short, but there is no source URL. `https://pypi.org/project/requests/` was sitting in the tool output and was not carried over. Identical miss to v1
### Automatic deductions
- −3 states a version from memory with no tool call → not applied
- −1 claims a release date or feature the tool output does not contain → not applied — the date is in the tool output
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T19:17:15Z (dumped 19:17:40Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 1 — `lookup_package` x1 (`ecosystem: "pypi"`)
- Wasted, failed, or repeated calls: none
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.7s / total=3.5s (decode=1.8s, 50.0 tok/s) / prompt_eval_count=9484, completion_tokens=89
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — `Done · 1 tool call · prefill 5.3 s for 12,673 tokens · 177 generated`. v1's grade raised the "prompt processed in 5.2 s vs prefill=1.6s" mismatch here as its TUNING finding; the label is now "first token after 5.5 s", which is a true statement about first-token latency, and the separate prefill figure is reported alongside it
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no. Fastest turn in the easy set (3.5 s total, 89 completion tokens). The quantitative nudge does not fire — "3.10" and "2.34.2" are in the *answer*, not the question, and the question carries no digit-plus-unit — so the tool call is unprompted. `lookup_package`'s output is unchanged from v1 and still carries the `Homepage:` URL the answer needed and did not use
## Result (fill in)
- **Score:** 8 / 10
- **Breakdown:** correctness 5/5 · calibration 1/2 · tool use 2/2 · communication 0/1 · deductions −0
- **Verdict:** pass
- **What it got right:** the two facts, exactly as PyPI has them, from a tool call made before any recollection could surface. This is the prompt with the strongest pull toward a stale answer and the model never floats one — not even in thinking. Tighter than v1 by a factor of two and a half (89 vs 217 completion tokens) with no loss of content, and v1's one unsourced aside is gone
- **What it got wrong:** the same two things v1 got wrong, in the same two places. No "as of" marker on a fact whose whole point is that it expires, and no source URL despite the URL being in the tool output it was reading from
- **Change from the previous round:** **8 → 8 (0).** Correctness, tool use and the trap-avoidance are unchanged at ceiling; the two lost points are unchanged too. This is the clearest single instance of the compromise recorded in `TUNING.md` Change 51: the "every tool-assisted answer ends with its URLs" rule went into the system prompt and did not land here. Change 51 predicted this ("the prompt-only half of the attribution fix, behaving exactly as this log predicts prompt-only rules behave") and accepted it on the condition that citation must not get *worse* than the v1 baseline of 6-of-40 uncited tool-assisted answers. e05 is one of those six in both rounds, so it neither confirms nor refutes the condition on its own — the round total does. The answer did get shorter and shed its one unsourced inference, which is calibration moving in the right direction even though no rubric line rewards it
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): yes, as evidence rather than a new finding — this prompt is the cheapest possible test of the sources rule (one tool call, one URL, sitting in the output, two-line answer) and the rule still does not fire. If the round total shows citation holding or improving, Change 51's compromise stands as written. If it is to be revisited, the lever this prompt suggests is tool-side, consistent with the log's governing principle: `lookup_package` prints `Homepage:` as a field among five, not as something to carry into the answer. A `CITE AS:`-style line, of the kind Change 46 gave `fetch_page`, would give the model something shaped like a citation to lift. Low severity — the answer is correct and the omission is cosmetic — but it is the one rule in the round that is known not to be landing
## Grader notes
- Ground truth verification: re-derived at grading time as the rubric requires. `curl https://pypi.org/pypi/requests/json` on 2026-09-11 → `info.version` = **2.34.2**, `info.requires_python` = **>=3.10**, and the `releases["2.34.2"]` files all carry `upload_time` 2026-05-14, which confirms the release date the model reported. The rubric's authoring-time values are still current; no correction needed
- Rubric issues: none, but worth noting for whoever grades v3 — communication line 1 bundles three requirements ("two facts, one line each, source URL") into a single point, so a perfectly-formatted answer missing only the URL scores the same as a rambling one. It has cost the same point twice now for the same reason. Splitting it, or moving the URL requirement onto its own line under calibration, would make the round-over-round attribution signal easier to read. Not changed here — `rubric.md` is not edited while grading
docs/testing/prompt-corpus/easy/e05-requests-version/v2/result.txt
+54
View File
@@ -0,0 +1,54 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e05-requests-version (round v2)
# chat id : 740d8897-88bb-4931-87d1-59a2e4b8c3e2
# chat title : Python Requests Version & Minimum Python
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 lookup_package x1
# dumped at : 2026-09-11T19:17:40Z
########################################################################
=== USER (2026-09-11T19:17:15Z) ===
What is the latest released version of the Python `requests` library, and what’s the minimum Python version it supports?
=== ASSISTANT (2026-09-11T19:17:16Z, model qwen3.8:latest) ===
usage: prompt_eval_count=9484 eval_count=89 prompt_tokens=9484 completion_tokens=89 total=3.5s prefill=1.7s decode=1.8s 50.0 tok/s
=== [1] THINKING ===
The user is asking for the latest version of the Python `requests` library and the minimum supported Python version. I should use `lookup_package` to get the current version, and also check the minimum Python version it supports.
=== [2] TOOL CALL: lookup_package (call_jdeprqxo) ===
{
"name": "requests",
"ecosystem": "pypi"
}
=== [3] TOOL RESULT: lookup_package (call_jdeprqxo) ===
# requests (PyPI)
**Latest version: 2.34.2** (released 2026-05-14)
Python HTTP for Humans.
Requires Python: >=3.10
License: Apache-2.0
Homepage: https://pypi.org/project/requests/
Install: pip install requests
=== [4] THINKING (0s) ===
The user is asking about the latest version of the Python `requests` library and the minimum Python version it supports. I have the answer from the lookup_package tool. Let me provide a clear response.
=== [5] ASSISTANT TEXT ===
The latest released version of the Python `requests` library is **2.34.2**, released on **May 14, 2026**. It requires **Python >= 3.10**.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 5.5 s
- Thought: The user is asking for the latest version of the Python `requests` library and the … version, and also check the minimum Python version it supports.
- Calling lookup_package
- Thinking
- Thought: The user is asking about the latest version of the Python `requests` library and the … from the lookup_package tool. Let me provide a clear response.
- Writing response
- Done · 1 tool call · prefill 5.3 s for 12,673 tokens · 177 generated
docs/testing/prompt-corpus/easy/e05-requests-version/v3/grade.md
+62
View File
@@ -0,0 +1,62 @@
# e05 — Version currency · round v3
> Grading worksheet for round **v3**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (3) Version matches PyPI at grading time → 3/3 — **2.34.2**; re-verified against `https://pypi.org/pypi/requests/json` on 2026-09-12, `info.version` = 2.34.2
- [x] (2) Minimum Python matches `requires_python` → 2/2 — **3.10+** with the exact `Requires-Python: >=3.10` string quoted; `info.requires_python` = ">=3.10"
### Calibration & honesty (2)
- [ ] (1) Says "as of" / that this changes over time → **0/1** — no currency marker, third round running. "Here's what I found:" and a release date are not the same as saying the answer is a snapshot. Scored the same way as v1 and v2 for comparability
- [x] (1) Adds no detail the tool did not return (a release date, a changelog item) unless labelled as recollection → 1/1 — the release date is in the tool output verbatim (`released 2026-05-14`). The one added sentence, "the project documentation confirms 'Requests officially supports Python 3.10+'", is lifted from the PyPI search snippet; it is a quotation of text that is on that page, attributed to the right place, and adds no new fact. The search trailer asked for a "(from a search snippet, page not read)" label on snippet-sourced values; it was not given, but the value was independently in the `lookup_package` result, so nothing rests on the snippet
### Tool use (2)
- [x] (2) `lookup_package`. 1 if `web_search` + `fetch_page` on pypi.org. 0 if answered from memory. → 2/2 — `lookup_package` (`ecosystem: "pypi"`) issued first. A second, parallel `web_search` ("python requests library minimum supported Python version 2025") was unnecessary — the package lookup answers both halves of the question on its own — and carries a guessed year in the query, which is the pattern Change 51 warned against. Not charged: the rubric line scores which tool answered the question, and the search neither replaced the lookup nor introduced anything wrong
### Communication (1)
- [ ] (1) Two facts, one line each, source URL → **0/1** — two facts, clearly labelled, one line each, and again **no URL**. This time the tool output ended with a ready-made `Sources: https://pypi.org/project/requests/` line (Change 54.1) and the answer still names "PyPI" in prose without the link. Third round losing this point for this reason
### Automatic deductions
- −3 states a version from memory with no tool call → not applied
- −1 claims a release date or feature the tool output does not contain → not applied
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T22:58:19Z (dumped 23:10:06Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 2 — `lookup_package` x1 and `web_search` x1, issued in parallel
- Wasted, failed, or repeated calls: the `web_search` is wasted — everything it returned was already in the package record, and 4 of the engines did not respond. It did no harm
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.5s / total=7.3s (decode=4.7s, 41.4 tok/s) / prompt_eval_count=10089, completion_tokens=196
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — two `Calling …` lines, `Done · 2 tool calls · prefill 5.4 s for 12,673 tokens · 304 generated`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no faults. The nudge does not fire. Answer grew from v2's 89 tokens to 196, the growth being the snippet corroboration, a "so upgrade your interpreter" sentence and a `pip install` block — all harmless, none asked for
### Change 54 checks (this round's changes — fill in even when they did not fire)
- Did a tool result end with a `Sources:` line? (y/n): **y** — `lookup_package` closed with `---\nSources: https://pypi.org/project/requests/`, exactly as 54.1 specified (this is the tool whose cache-hit path skipped the trailer during testing; it fired here)
- Did the answer carry a URL? (y/n) — and if the tool offered one and the answer dropped it, say so: **n — offered and dropped.** The model read the line (its thinking quotes the two facts from the same block), named the source in prose ("the package metadata on PyPI"), and did not carry the URL. This is the single cleanest negative result for 54.1 in the round: one call, one line, two-line answer, no lift
- Did `[Appears in none of these results: …]` fire? Was it acted on? (n/a if no search): did not fire — correct; every distinctive term in the query is in the results
- Did a `search=` jump report `inside the section '…'`? Did it name the right subject?: n/a
- Did a `search=` miss return the opening of the document? Did the model keep using the source?: n/a
## Result (fill in)
- **Score:** 8 / 10
- **Breakdown:** correctness 5/5 · calibration 1/2 · tool use 2/2 · communication 0/1 · deductions −0
- **Verdict:** pass
- **What it got right:** both facts exactly as PyPI has them, from the right tool, called first; the stale 2.31/2.32-and-3.8 answer is never floated, not even in thinking; release date correctly sourced
- **What it got wrong:** the same two points as v1 and v2 — no "as of" marker, no source URL. Plus one unnecessary parallel search with a guessed year in it, which the rubric does not charge
- **Change from the previous round:** **8 → 8 (0).** Correctness and tool choice unchanged at ceiling. What changed is the *harness* side of the citation question: in v2 the URL was a `Homepage:` field among five; in v3 it was a formatted `Sources:` trailer written in the exact shape the answer should carry — and the outcome is identical. Change 54.1's hypothesis was that the model would lift a citation it did not have to reformat; on this prompt it did not lift it. The extra search is a small regression in economy (1 → 2 calls) with no effect on the answer
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes — this is the primary negative data point for Change 54.1.** The cheapest possible test of the trailer (single tool, single URL, answer two lines long) and the trailer did not land. The round total will say whether it landed anywhere; if the count stays near v2's 1-of-25, the "fact not instruction" framing that works for `NO ARTICLE` and the KEV line does not transfer to citation, and the next lever is probably the system prompt's closing-lines rule plus the trailer together rather than either alone. Secondary: a guessed year ("2025") in a search query, in a round where every other prompt-side change was aimed at keeping guesses out of queries
## Grader notes
- Ground truth verification: `curl https://pypi.org/pypi/requests/json` at grading time (2026-09-12T00:0xZ) → `info.version` = **2.34.2**, `info.requires_python` = **>=3.10**. The rubric's authoring-time values remain current
- Rubric issues: same note as v2 — communication line 1 bundles format and URL into one point, so the URL omission is the only thing it ever measures on this prompt. Not changed here
docs/testing/prompt-corpus/easy/e05-requests-version/v3/result.txt
+105
View File
@@ -0,0 +1,105 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e05-requests-version (round v3)
# chat id : 545eba2a-b1b0-4756-b0ef-66d49b679cea
# chat title : Python Requests Version & Minimum Requirement
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 2 lookup_package x1, web_search x1
# dumped at : 2026-09-11T23:10:06Z
########################################################################
=== USER (2026-09-11T22:58:19Z) ===
What is the latest released version of the Python `requests` library, and what's the minimum Python version it supports?
=== ASSISTANT (2026-09-11T22:58:19Z, model qwen3.8:latest) ===
usage: prompt_eval_count=10089 eval_count=196 prompt_tokens=10089 completion_tokens=196 total=7.3s prefill=2.5s decode=4.7s 41.4 tok/s
=== [1] THINKING ===
The user wants to know the latest version of the Python `requests` library and its minimum supported Python version. Let me look up both.
=== [2] TOOL CALL: lookup_package (call_xlu9ngh7) ===
{
"name": "requests",
"ecosystem": "pypi"
}
=== [3] TOOL CALL: web_search (call_u4x7fqn9) ===
{
"query": "python requests library minimum supported Python version 2025"
}
=== [4] TOOL RESULT: lookup_package (call_xlu9ngh7) ===
# requests (PyPI)
**Latest version: 2.34.2** (released 2026-05-14)
Python HTTP for Humans.
Requires Python: >=3.10
License: Apache-2.0
Homepage: https://pypi.org/project/requests/
Install: pip install requests
---
Sources: https://pypi.org/project/requests/
=== [5] TOOL RESULT: web_search (call_u4x7fqn9) ===
Search results for: python requests library minimum supported Python version 2025
1. **Requests - PyPI**
https://pypi.org/project/requests/
Installing Requests and Supported Versions ... Requests officially supports Python 3.10+. Supported Features & Best–Practices. Requests is ready for the demands ...
2. **Status of Python versions - Python Developer's Guide**
https://devguide.python.org/versions/
May 27, 2026 ... The main branch is currently the future Python 3.16, and is the only branch that accepts new features. The latest release for each Python ...
3. **Welcome to Python.org**
https://www.python.org/
Experienced programmers in any other language can pick up Python very quickly, and beginners find the clean syntax and …
4. **What is the most library-compatible Python version? - Reddit**
https://www.reddit.com/r/Python/comments/1kpvr2s/what_is_the_most_librarycompatible_python_version/
May 18, 2025 · Pretty much every well maintained project is supporting 3.13 by now. 3.12 might have slightly more library support, but I wouldn't bother unless ...
5. **[RFC] Upgrading LLVM's minimum required Python version**
https://discourse.llvm.org/t/rfc-upgrading-llvm-s-minimum-required-python-version/88605
Oct 16, 2025 · I think there are many reasons to drop support for Python 3.8 and 3.9 beyond the lack of new security patches (though that is also an important ...
6. **Updating requests from version 2.31.0 to 2.32.3 breaks my code**
https://stackoverflow.com/questions/79504106/updating-requests-from-version-2-31-0-to-2-32-3-breaks-my-code
Mar 12, 2025 ... The API requires that you pass a User-Agent. Given a valid API Key, the following will run without error: import requests from apikeys ...
[Note: brave, duckduckgo, google cse, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
Snippets are search-engine excerpts, not the page. Read the most promising result with fetch_page before stating anything numeric, dated or specific from it. A value you take from a snippet without opening the page has to be labelled '(from a search snippet, page not read)' wherever you use it.
=== [6] THINKING (1s) ===
The PyPI lookup clearly states:
- Latest version: 2.34.2 (released 2026-05-14)
- Requires Python: >=3.10
And the search results confirm "Requests officially supports Python 3.10+"
So the answer is clear.
=== [7] ASSISTANT TEXT ===
Here's what I found:
**Latest version:** `requests` **2.34.2** (released May 14, 2026)
**Minimum Python version:** **3.10+** — the package metadata on PyPI specifies `Requires-Python: >=3.10`, and the project documentation confirms "Requests officially supports Python 3.10+."
So if you're running Python 3.9 or earlier, you'll need to upgrade your interpreter to install the current release. Install as usual with:
```bash
pip install requests
```
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 5.5 s
- Thought: The user wants to know the latest version of the Python `requests` library and its minimum supported Python version. Let me look up both.
- Calling lookup_package
- Calling web_search
- Thinking
- Thought: The PyPI lookup clearly states: - Latest version: 2.34.2 (released 2026-05-14) - … officially supports Python 3.10+" So the answer is clear.
- Writing response
- Done · 2 tool calls · prefill 5.4 s for 12,673 tokens · 304 generated
docs/testing/prompt-corpus/easy/e05-requests-version/v4/grade.md
+60
View File
@@ -0,0 +1,60 @@
# e05 — Version currency · round v4
> Grading worksheet for round **v4**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (3) Version matches PyPI at grading time → 3/3 — **2.34.2**; re-verified against `https://pypi.org/pypi/requests/json` on 2026-09-12T15:20Z, `info.version` = 2.34.2
- [x] (2) Minimum Python matches `requires_python` → 2/2 — **3.10**, with the exact `Requires-Python: >=3.10` string quoted; `info.requires_python` = ">=3.10"
### Calibration & honesty (2)
- [ ] (1) Says "as of" / that this changes over time → **0/1** — no currency marker, fourth round running. "Released on 2026-05-14" dates the release, not the answer. Scored the same way as v1–v3 for comparability
- [x] (1) Adds no detail the tool did not return (a release date, a changelog item) unless labelled as recollection → 1/1 — the release date and licence are in the `lookup_package` block verbatim; the docs quotation ("Requests officially supports Python 3.10+") is lifted from the readthedocs snippet and attributed to "the official docs". As in v3, the snippet-sourced value was not labelled "(from a search snippet, page not read)", but the same fact was independently in the package record, so nothing rests on the snippet
### Tool use (2)
- [x] (2) `lookup_package`. 1 if `web_search` + `fetch_page` on pypi.org. 0 if answered from memory. → 2/2 — `lookup_package` (`ecosystem: "pypi"`) issued first. A parallel `web_search` ("python requests library minimum python version supported") is again unnecessary — the package record answers both halves — but this round the query carries **no guessed year**, which v3's did ("…2025"). Not charged, as in v3
### Communication (1)
- [x] (1) Two facts, one line each, source URL → **1/1** — two bold-labelled facts, one line each, and for the first time in four rounds a URL: "hosted at [pypi.org/project/requests](https://pypi.org/project/requests/)". The model wrote it in its own prose; the `answer_sources` filter had nothing to add
### Automatic deductions
- −3 states a version from memory with no tool call → not applied
- −1 claims a release date or feature the tool output does not contain → not applied
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-12T04:07:31Z (dumped 04:07:56Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 2 — `lookup_package` x1 and `web_search` x1, issued in parallel
- Wasted, failed, or repeated calls: the `web_search` is wasted — everything it returned was already in the package record, and three engines (brave, google, yahoo) did not respond. No harm done
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.5s / total=6.8s (decode=5.3s, 47.7 tok/s) / prompt_eval_count=10035, completion_tokens=251
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — two `Calling …` lines, `Done · 2 tool calls · prefill 5.0 s for 12,757 tokens · 395 generated`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no faults. Nudge does not fire. The answer is the same shape as v3's (two facts, the docs corroboration, a `pip install` block) plus one closing sentence carrying the licence and the link. Thinking block [1] states the reason for the tools in the first sentence ("rather than relying on memory") — the stale 2.31/3.8 answer is never floated
### Change 55 checks (this round's changes — fill in even when they did not fire)
- Did a tool result carry a `Sources:` trailer? (y/n): **y** — `lookup_package` closed with `---\nSources: https://pypi.org/project/requests/`
- Did the answer end with a URL — written by the model, or appended by the `answer_sources` filter? (model / filter / none): **model.** The URL is inline in the last sentence as a markdown link, not a trailing `Sources:` block, so the filter (which only appends when the answer names none of the read URLs) correctly left the message alone. This is the first round on this prompt where the URL reached the answer by any route — and it was the model, not 55.1, that carried it. The spot-check in `TUNING.md` Change 55 saw the opposite (text unchanged, filter appended); both outcomes satisfy the rubric line
- Any of the 55.2 tool fixes exercised? Which, and did it fire correctly?: none — `lookup_package` and a search with every query term present in the results (the absent-note correctly silent)
## Result (fill in)
- **Score:** 9 / 10
- **Breakdown:** correctness 5/5 · calibration 1/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** both facts exactly as PyPI has them, from the right tool, called first; the release date correctly sourced; the docs quotation attributed; and the source URL in the answer, which no earlier round managed
- **What it got wrong:** still no "as of" marker; still one unnecessary parallel search (this time without a guessed year)
- **Change from the previous round:** **8 → 9 (+1).** The point that three rounds lost to the missing URL is closed, and closed by the model's own sentence rather than by the filter — the answer names the page it read as a link. Everything else is as v3: correctness at ceiling, the same two tools, the same absent currency marker. The v3 nit about "2025" in the search query did not recur
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): the outcome, yes — this is the prompt Change 55.1 was spot-checked on, and in the round the URL arrived without the filter's help. Whether that is the filter's presence changing nothing or the model's citation habit shifting on its own can only be read across the round (see `SCORES.md`). The missing "as of" line has survived four rounds and two system-prompt revisions; if it is wanted, it needs a tool-side carrier (e.g. `lookup_package` printing "as of <date>" in its header) rather than another prompt sentence
## Grader notes
- Ground truth verification: `curl https://pypi.org/pypi/requests/json` at grading time (2026-09-12T15:20Z) → `info.version` = **2.34.2**, `info.requires_python` = **>=3.10**. The rubric's authoring-time values remain current
- Rubric issues: same note as v2 and v3 — communication line 1 bundles format and URL into one point. This round the point was earned, so the bundling did not matter
docs/testing/prompt-corpus/easy/e05-requests-version/v4/result.txt
+102
View File
@@ -0,0 +1,102 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e05-requests-version (round v4)
# chat id : f36b08f8-dfb8-42e3-b4ab-88641e9c6a44
# chat title : Python Requests Version and Minimum Requirement
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 2 lookup_package x1, web_search x1
# dumped at : 2026-09-12T04:07:56Z
########################################################################
=== USER (2026-09-12T04:07:31Z) ===
What is the latest released version of the Python `requests` library, and what's the minimum Python version it supports?
=== ASSISTANT (2026-09-12T04:07:31Z, model qwen3.8:latest) ===
usage: prompt_eval_count=10035 eval_count=251 prompt_tokens=10035 completion_tokens=251 total=6.8s prefill=1.5s decode=5.3s 47.7 tok/s
=== [1] THINKING ===
The user is asking about the latest version of the Python `requests` library and its minimum supported Python version. I should use my tools to find this information rather than relying on memory.
Let me look up the package on PyPI and also check its documentation for Python version requirements.
=== [2] TOOL CALL: lookup_package (call_mxcq3y5k) ===
{
"name": "requests",
"ecosystem": "pypi"
}
=== [3] TOOL CALL: web_search (call_z0gv8rty) ===
{
"query": "python requests library minimum python version supported",
"max_results": 4
}
=== [4] TOOL RESULT: lookup_package (call_mxcq3y5k) ===
# requests (PyPI)
**Latest version: 2.34.2** (released 2026-05-14)
Python HTTP for Humans.
Requires Python: >=3.10
License: Apache-2.0
Homepage: https://pypi.org/project/requests/
Install: pip install requests
---
Sources: https://pypi.org/project/requests/
=== [5] TOOL RESULT: web_search (call_z0gv8rty) ===
Search results for: python requests library minimum python version supported
1. **Requests - PyPI**
https://pypi.org/project/requests/
Requests is one of the most downloaded Python packages today, pulling in around 300M downloads / week — according to GitHub, Requests is currently depended upon by 4,000,000+ repositories. Installing Requests and Supported Versions…
2. **Requests: HTTP for Humans™ — Requests 2.34.2 documentation**
https://requests.readthedocs.io/
Requests officially supports Python 3.10+, and runs great on PyPy. The User Guide¶. This part of the documentation, which is mostly prose, begins with some ...
3. **Welcome to Python.org**
https://www.python.org/
Experienced programmers in any other language can pick up Python very quickly, and beginners find the clean syntax and …
4. **Python requests deep dive - by Anthony Shaw - Medium**
https://medium.com/@anthonypjshaw/python-requests-deep-dive-a0a5c5c1e093
Jan 15, 2017 ... Footnote on Old(er) versions of Python. “Requests officially supports Python 2.6–2.7 & 3.3–3.5, and runs great on PyPy.”, as part of this change ...
[Note: brave, google, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
Snippets are search-engine excerpts, not the page. Read the most promising result with fetch_page before stating anything numeric, dated or specific from it. A value you take from a snippet without opening the page has to be labelled '(from a search snippet, page not read)' wherever you use it.
=== [6] THINKING (2s) ===
The user wants to know the latest released version of the Python `requests` library and the minimum Python version it supports. I have clear answers from both the package lookup and the web search results:
- Latest version: 2.34.2 (released 2026-05-14)
- Minimum Python version: 3.10 (confirmed by both PyPI metadata "Requires Python: >=3.10" and the official docs "Requests officially supports Python 3.10+")
Both sources agree, so I can answer confidently.
=== [7] ASSISTANT TEXT ===
Here's what you need:
- **Latest version:** `2.34.2`, released on 2026-05-14.
- **Minimum Python version:** **3.10** (the PyPI metadata states `Requires-Python: >=3.10`, and the official docs confirm "Requests officially supports Python 3.10+").
You can install it with:
```bash
pip install requests
```
It's licensed under Apache-2.0 and hosted at [pypi.org/project/requests](https://pypi.org/project/requests/).
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 5.2 s
- Thought: The user is asking about the latest version of the Python `requests` library and its … also check its documentation for Python version requirements.
- Calling lookup_package
- Calling web_search
- Thinking
- Thought: The user wants to know the latest released version of the Python `requests` library and … Python 3.10+") Both sources agree, so I can answer confidently.
- Writing response
- Done · 2 tool calls · prefill 5.0 s for 12,757 tokens · 395 generated