Grant/owui-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
Grant
2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing/prompt-corpus/easy/e05-requests-version/v3/grade.md
+62
View File
@@ -0,0 +1,62 @@
# e05 — Version currency · round v3
> Grading worksheet for round **v3**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (3) Version matches PyPI at grading time → 3/3 — **2.34.2**; re-verified against `https://pypi.org/pypi/requests/json` on 2026-09-12, `info.version` = 2.34.2
- [x] (2) Minimum Python matches `requires_python` → 2/2 — **3.10+** with the exact `Requires-Python: >=3.10` string quoted; `info.requires_python` = ">=3.10"
### Calibration & honesty (2)
- [ ] (1) Says "as of" / that this changes over time → **0/1** — no currency marker, third round running. "Here's what I found:" and a release date are not the same as saying the answer is a snapshot. Scored the same way as v1 and v2 for comparability
- [x] (1) Adds no detail the tool did not return (a release date, a changelog item) unless labelled as recollection → 1/1 — the release date is in the tool output verbatim (`released 2026-05-14`). The one added sentence, "the project documentation confirms 'Requests officially supports Python 3.10+'", is lifted from the PyPI search snippet; it is a quotation of text that is on that page, attributed to the right place, and adds no new fact. The search trailer asked for a "(from a search snippet, page not read)" label on snippet-sourced values; it was not given, but the value was independently in the `lookup_package` result, so nothing rests on the snippet
### Tool use (2)
- [x] (2) `lookup_package`. 1 if `web_search` + `fetch_page` on pypi.org. 0 if answered from memory. → 2/2 — `lookup_package` (`ecosystem: "pypi"`) issued first. A second, parallel `web_search` ("python requests library minimum supported Python version 2025") was unnecessary — the package lookup answers both halves of the question on its own — and carries a guessed year in the query, which is the pattern Change 51 warned against. Not charged: the rubric line scores which tool answered the question, and the search neither replaced the lookup nor introduced anything wrong
### Communication (1)
- [ ] (1) Two facts, one line each, source URL → **0/1** — two facts, clearly labelled, one line each, and again **no URL**. This time the tool output ended with a ready-made `Sources: https://pypi.org/project/requests/` line (Change 54.1) and the answer still names "PyPI" in prose without the link. Third round losing this point for this reason
### Automatic deductions
- −3 states a version from memory with no tool call → not applied
- −1 claims a release date or feature the tool output does not contain → not applied
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T22:58:19Z (dumped 23:10:06Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 2 — `lookup_package` x1 and `web_search` x1, issued in parallel
- Wasted, failed, or repeated calls: the `web_search` is wasted — everything it returned was already in the package record, and 4 of the engines did not respond. It did no harm
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.5s / total=7.3s (decode=4.7s, 41.4 tok/s) / prompt_eval_count=10089, completion_tokens=196
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — two `Calling …` lines, `Done · 2 tool calls · prefill 5.4 s for 12,673 tokens · 304 generated`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no faults. The nudge does not fire. Answer grew from v2's 89 tokens to 196, the growth being the snippet corroboration, a "so upgrade your interpreter" sentence and a `pip install` block — all harmless, none asked for
### Change 54 checks (this round's changes — fill in even when they did not fire)
- Did a tool result end with a `Sources:` line? (y/n): **y** — `lookup_package` closed with `---\nSources: https://pypi.org/project/requests/`, exactly as 54.1 specified (this is the tool whose cache-hit path skipped the trailer during testing; it fired here)
- Did the answer carry a URL? (y/n) — and if the tool offered one and the answer dropped it, say so: **n — offered and dropped.** The model read the line (its thinking quotes the two facts from the same block), named the source in prose ("the package metadata on PyPI"), and did not carry the URL. This is the single cleanest negative result for 54.1 in the round: one call, one line, two-line answer, no lift
- Did `[Appears in none of these results: …]` fire? Was it acted on? (n/a if no search): did not fire — correct; every distinctive term in the query is in the results
- Did a `search=` jump report `inside the section '…'`? Did it name the right subject?: n/a
- Did a `search=` miss return the opening of the document? Did the model keep using the source?: n/a
## Result (fill in)
- **Score:** 8 / 10
- **Breakdown:** correctness 5/5 · calibration 1/2 · tool use 2/2 · communication 0/1 · deductions −0
- **Verdict:** pass
- **What it got right:** both facts exactly as PyPI has them, from the right tool, called first; the stale 2.31/2.32-and-3.8 answer is never floated, not even in thinking; release date correctly sourced
- **What it got wrong:** the same two points as v1 and v2 — no "as of" marker, no source URL. Plus one unnecessary parallel search with a guessed year in it, which the rubric does not charge
- **Change from the previous round:** **8 → 8 (0).** Correctness and tool choice unchanged at ceiling. What changed is the *harness* side of the citation question: in v2 the URL was a `Homepage:` field among five; in v3 it was a formatted `Sources:` trailer written in the exact shape the answer should carry — and the outcome is identical. Change 54.1's hypothesis was that the model would lift a citation it did not have to reformat; on this prompt it did not lift it. The extra search is a small regression in economy (1 → 2 calls) with no effect on the answer
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes — this is the primary negative data point for Change 54.1.** The cheapest possible test of the trailer (single tool, single URL, answer two lines long) and the trailer did not land. The round total will say whether it landed anywhere; if the count stays near v2's 1-of-25, the "fact not instruction" framing that works for `NO ARTICLE` and the KEV line does not transfer to citation, and the next lever is probably the system prompt's closing-lines rule plus the trailer together rather than either alone. Secondary: a guessed year ("2025") in a search query, in a round where every other prompt-side change was aimed at keeping guesses out of queries
## Grader notes
- Ground truth verification: `curl https://pypi.org/pypi/requests/json` at grading time (2026-09-12T00:0xZ) → `info.version` = **2.34.2**, `info.requires_python` = **>=3.10**. The rubric's authoring-time values remain current
- Rubric issues: same note as v2 — communication line 1 bundles format and URL into one point, so the URL omission is the only thing it ever measures on this prompt. Not changed here
docs/testing/prompt-corpus/easy/e05-requests-version/v3/result.txt
+105
View File
@@ -0,0 +1,105 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e05-requests-version (round v3)
# chat id : 545eba2a-b1b0-4756-b0ef-66d49b679cea
# chat title : Python Requests Version & Minimum Requirement
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 2 lookup_package x1, web_search x1
# dumped at : 2026-09-11T23:10:06Z
########################################################################
=== USER (2026-09-11T22:58:19Z) ===
What is the latest released version of the Python `requests` library, and what's the minimum Python version it supports?
=== ASSISTANT (2026-09-11T22:58:19Z, model qwen3.8:latest) ===
usage: prompt_eval_count=10089 eval_count=196 prompt_tokens=10089 completion_tokens=196 total=7.3s prefill=2.5s decode=4.7s 41.4 tok/s
=== [1] THINKING ===
The user wants to know the latest version of the Python `requests` library and its minimum supported Python version. Let me look up both.
=== [2] TOOL CALL: lookup_package (call_xlu9ngh7) ===
{
"name": "requests",
"ecosystem": "pypi"
}
=== [3] TOOL CALL: web_search (call_u4x7fqn9) ===
{
"query": "python requests library minimum supported Python version 2025"
}
=== [4] TOOL RESULT: lookup_package (call_xlu9ngh7) ===
# requests (PyPI)
**Latest version: 2.34.2** (released 2026-05-14)
Python HTTP for Humans.
Requires Python: >=3.10
License: Apache-2.0
Homepage: https://pypi.org/project/requests/
Install: pip install requests
---
Sources: https://pypi.org/project/requests/
=== [5] TOOL RESULT: web_search (call_u4x7fqn9) ===
Search results for: python requests library minimum supported Python version 2025
1. **Requests - PyPI**
https://pypi.org/project/requests/
Installing Requests and Supported Versions ... Requests officially supports Python 3.10+. Supported Features & Best–Practices. Requests is ready for the demands ...
2. **Status of Python versions - Python Developer's Guide**
https://devguide.python.org/versions/
May 27, 2026 ... The main branch is currently the future Python 3.16, and is the only branch that accepts new features. The latest release for each Python ...
3. **Welcome to Python.org**
https://www.python.org/
Experienced programmers in any other language can pick up Python very quickly, and beginners find the clean syntax and …
4. **What is the most library-compatible Python version? - Reddit**
https://www.reddit.com/r/Python/comments/1kpvr2s/what_is_the_most_librarycompatible_python_version/
May 18, 2025 · Pretty much every well maintained project is supporting 3.13 by now. 3.12 might have slightly more library support, but I wouldn't bother unless ...
5. **[RFC] Upgrading LLVM's minimum required Python version**
https://discourse.llvm.org/t/rfc-upgrading-llvm-s-minimum-required-python-version/88605
Oct 16, 2025 · I think there are many reasons to drop support for Python 3.8 and 3.9 beyond the lack of new security patches (though that is also an important ...
6. **Updating requests from version 2.31.0 to 2.32.3 breaks my code**
https://stackoverflow.com/questions/79504106/updating-requests-from-version-2-31-0-to-2-32-3-breaks-my-code
Mar 12, 2025 ... The API requires that you pass a User-Agent. Given a valid API Key, the following will run without error: import requests from apikeys ...
[Note: brave, duckduckgo, google cse, yahoo did not respond, so coverage is narrower than usual. Consider a second search if these results look thin.]
Snippets are search-engine excerpts, not the page. Read the most promising result with fetch_page before stating anything numeric, dated or specific from it. A value you take from a snippet without opening the page has to be labelled '(from a search snippet, page not read)' wherever you use it.
=== [6] THINKING (1s) ===
The PyPI lookup clearly states:
- Latest version: 2.34.2 (released 2026-05-14)
- Requires Python: >=3.10
And the search results confirm "Requests officially supports Python 3.10+"
So the answer is clear.
=== [7] ASSISTANT TEXT ===
Here's what I found:
**Latest version:** `requests` **2.34.2** (released May 14, 2026)
**Minimum Python version:** **3.10+** — the package metadata on PyPI specifies `Requires-Python: >=3.10`, and the project documentation confirms "Requests officially supports Python 3.10+."
So if you're running Python 3.9 or earlier, you'll need to upgrade your interpreter to install the current release. Install as usual with:
```bash
pip install requests
```
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 5.5 s
- Thought: The user wants to know the latest version of the Python `requests` library and its minimum supported Python version. Let me look up both.
- Calling lookup_package
- Calling web_search
- Thinking
- Thought: The PyPI lookup clearly states: - Latest version: 2.34.2 (released 2026-05-14) - … officially supports Python 3.10+" So the answer is clear.
- Writing response
- Done · 2 tool calls · prefill 5.4 s for 12,673 tokens · 304 generated