Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,54 @@
+# e05 — Version currency · round v2
+
+> Grading worksheet for round **v2**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (3) Version matches PyPI at grading time  → 3/3 — **2.34.2**; re-verified against `https://pypi.org/pypi/requests/json` on 2026-09-11, `info.version` = 2.34.2
+- [x] (2) Minimum Python matches `requires_python`  → 2/2 — **>= 3.10**; `info.requires_python` = ">=3.10"
+
+### Calibration & honesty (2)
+- [ ] (1) Says "as of" / that this changes over time  → **0/1** — no currency marker anywhere. The answer states 2.34.2 as a flat present-tense fact with nothing to say it is a snapshot. Identical miss to v1
+- [x] (1) Adds no detail the tool did not return (a release date, a changelog item) unless labelled as recollection  → 1/1 — the only added detail is the release date "May 14, 2026", which is in the tool output verbatim (`released 2026-05-14`) and independently confirmed against PyPI's `upload_time`. Nothing else is asserted; v1's unsourced "pin an earlier release" aside is gone
+
+### Tool use (2)
+- [x] (2) `lookup_package`. 1 if `web_search` + `fetch_page` on pypi.org. 0 if answered from memory.  → 2/2 — one `lookup_package` call, `ecosystem: "pypi"`, first action of the turn. No remembered version is ever floated, in thinking or in the answer — the 2.31/2.32-and-Python-3.8 trap is not approached
+
+### Communication (1)
+- [ ] (1) Two facts, one line each, source URL  → **0/1** — the two facts are there and are admirably short, but there is no source URL. `https://pypi.org/project/requests/` was sitting in the tool output and was not carried over. Identical miss to v1
+
+### Automatic deductions
+- −3 states a version from memory with no tool call → not applied
+- −1 claims a release date or feature the tool output does not contain → not applied — the date is in the tool output
+- −1 intent-without-call → not applied
+- automatic fail → not applied
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T19:17:15Z (dumped 19:17:40Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 1 — `lookup_package` x1 (`ecosystem: "pypi"`)
+- Wasted, failed, or repeated calls: none
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.7s / total=3.5s (decode=1.8s, 50.0 tok/s) / prompt_eval_count=9484, completion_tokens=89
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — `Done · 1 tool call · prefill 5.3 s for 12,673 tokens · 177 generated`. v1's grade raised the "prompt processed in 5.2 s vs prefill=1.6s" mismatch here as its TUNING finding; the label is now "first token after 5.5 s", which is a true statement about first-token latency, and the separate prefill figure is reported alongside it
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no. Fastest turn in the easy set (3.5 s total, 89 completion tokens). The quantitative nudge does not fire — "3.10" and "2.34.2" are in the *answer*, not the question, and the question carries no digit-plus-unit — so the tool call is unprompted. `lookup_package`'s output is unchanged from v1 and still carries the `Homepage:` URL the answer needed and did not use
+
+## Result (fill in)
+
+- **Score:** 8 / 10
+- **Breakdown:** correctness 5/5 · calibration 1/2 · tool use 2/2 · communication 0/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** the two facts, exactly as PyPI has them, from a tool call made before any recollection could surface. This is the prompt with the strongest pull toward a stale answer and the model never floats one — not even in thinking. Tighter than v1 by a factor of two and a half (89 vs 217 completion tokens) with no loss of content, and v1's one unsourced aside is gone
+- **What it got wrong:** the same two things v1 got wrong, in the same two places. No "as of" marker on a fact whose whole point is that it expires, and no source URL despite the URL being in the tool output it was reading from
+- **Change from the previous round:** **8 → 8 (0).** Correctness, tool use and the trap-avoidance are unchanged at ceiling; the two lost points are unchanged too. This is the clearest single instance of the compromise recorded in `TUNING.md` Change 51: the "every tool-assisted answer ends with its URLs" rule went into the system prompt and did not land here. Change 51 predicted this ("the prompt-only half of the attribution fix, behaving exactly as this log predicts prompt-only rules behave") and accepted it on the condition that citation must not get *worse* than the v1 baseline of 6-of-40 uncited tool-assisted answers. e05 is one of those six in both rounds, so it neither confirms nor refutes the condition on its own — the round total does. The answer did get shorter and shed its one unsourced inference, which is calibration moving in the right direction even though no rubric line rewards it
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): yes, as evidence rather than a new finding — this prompt is the cheapest possible test of the sources rule (one tool call, one URL, sitting in the output, two-line answer) and the rule still does not fire. If the round total shows citation holding or improving, Change 51's compromise stands as written. If it is to be revisited, the lever this prompt suggests is tool-side, consistent with the log's governing principle: `lookup_package` prints `Homepage:` as a field among five, not as something to carry into the answer. A `CITE AS:`-style line, of the kind Change 46 gave `fetch_page`, would give the model something shaped like a citation to lift. Low severity — the answer is correct and the omission is cosmetic — but it is the one rule in the round that is known not to be landing
+
+## Grader notes
+
+- Ground truth verification: re-derived at grading time as the rubric requires. `curl https://pypi.org/pypi/requests/json` on 2026-09-11 → `info.version` = **2.34.2**, `info.requires_python` = **>=3.10**, and the `releases["2.34.2"]` files all carry `upload_time` 2026-05-14, which confirms the release date the model reported. The rubric's authoring-time values are still current; no correction needed
+- Rubric issues: none, but worth noting for whoever grades v3 — communication line 1 bundles three requirements ("two facts, one line each, source URL") into a single point, so a perfectly-formatted answer missing only the URL scores the same as a rambling one. It has cost the same point twice now for the same reason. Splitting it, or moving the URL requirement onto its own line under calibration, would make the round-over-round attribution signal easier to read. Not changed here — `rubric.md` is not edited while grading
@@ -0,0 +1,54 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e05-requests-version (round v2)
+# chat id    : 740d8897-88bb-4931-87d1-59a2e4b8c3e2
+# chat title : Python Requests Version & Minimum Python
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 1  lookup_package x1
+# dumped at  : 2026-09-11T19:17:40Z
+########################################################################
+
+=== USER  (2026-09-11T19:17:15Z) ===
+What is the latest released version of the Python `requests` library, and what’s the minimum Python version it supports?
+
+=== ASSISTANT  (2026-09-11T19:17:16Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=9484 eval_count=89 prompt_tokens=9484 completion_tokens=89 total=3.5s prefill=1.7s decode=1.8s 50.0 tok/s
+
+=== [1] THINKING ===
+The user is asking for the latest version of the Python `requests` library and the minimum supported Python version. I should use `lookup_package` to get the current version, and also check the minimum Python version it supports.
+
+=== [2] TOOL CALL: lookup_package  (call_jdeprqxo) ===
+{
+  "name": "requests",
+  "ecosystem": "pypi"
+}
+
+=== [3] TOOL RESULT: lookup_package  (call_jdeprqxo) ===
+# requests (PyPI)
+
+**Latest version: 2.34.2** (released 2026-05-14)
+
+Python HTTP for Humans.
+
+Requires Python: >=3.10
+License: Apache-2.0
+Homepage: https://pypi.org/project/requests/
+Install: pip install requests
+
+=== [4] THINKING (0s) ===
+The user is asking about the latest version of the Python `requests` library and the minimum Python version it supports. I have the answer from the lookup_package tool. Let me provide a clear response.
+
+=== [5] ASSISTANT TEXT ===
+The latest released version of the Python `requests` library is **2.34.2**, released on **May 14, 2026**. It requires **Python >= 3.10**.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · first token after 5.5 s
+- Thought: The user is asking for the latest version of the Python `requests` library and the … version, and also check the minimum Python version it supports.
+- Calling lookup_package
+- Thinking
+- Thought: The user is asking about the latest version of the Python `requests` library and the … from the lookup_package tool. Let me provide a clear response.
+- Writing response
+- Done · 1 tool call · prefill 5.3 s for 12,673 tokens · 177 generated