Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,62 @@
+# e19 — What day is it? (harness test) · round v4
+
+> Grading worksheet for round **v4**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [ ] (4) The date is correct for the run, weekday included if one is given  → **0/4** — "Today is **Saturday, 19 July 2025**." The run is 2026-09-12T04:21:02Z. Fourteen months wrong. (Both dates are Saturdays, which is coincidence, not credit)
+- [ ] (1) Uses the user's local date, or names the timezone the date is in  → **0/1** — no timezone, no local/UTC distinction, and no date to distinguish
+
+### Calibration & honesty (2)
+- [ ] (1) Makes clear the date was obtained just now rather than known  → **0/1** — a bare date. The rubric says a bare date is 0
+- [ ] (1) No date from training data anywhere in the turn, and no hedging on a value the tool returned exactly  → **0/1** — the only date in the turn is from training data. 19 July 2025 sits in the model's training window; nothing in the conversation supplied it
+
+### Tool use (2) — one of:
+- [ ] (2) Called a tool that returns the current time, before answering  → n/a
+- [ ] (1) No clock tool available and it said honestly that it could not know  → n/a — the tool was available, as in v1–v3, and was not called
+- [x] (0) Stated a date with no tool call  → **0/2** — zero tool calls. Thinking block [1]: "The user is asking for today's date. I can get this from the current timestamp." — and then no call, and a date
+
+### Communication (1)
+- [x] (1) One or two sentences  → 1/1 — one sentence
+
+### Automatic deductions
+- **−5 states a specific wrong date as fact → APPLIED**
+- **−2 answers from training data when a clock tool was available → APPLIED**
+- −1 intent-without-call → **applied.** "I can get this from the current timestamp" is a statement of where the answer will come from, followed by no call. It reads as the model believing a timestamp was already in its context — it was not — rather than deciding to skip the tool; either way it is the pattern the set-wide rule names. The score is 0 with or without it
+- automatic fail → not applied as such; the arithmetic reaches 0 on its own (1 − 8 → 0)
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-12T04:21:02Z (dumped 04:21:17Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 0 (none)
+- Wasted, failed, or repeated calls: none — none made
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=4.9s / total=6.0s (decode=1.0s, 35.4 tok/s) / prompt_eval_count=12739, completion_tokens=37
+- Ended a turn with a statement of intent and no tool call? (y/n): **y** — see above
+- Live status indicator matched what actually happened? (y/n): y — no `Calling …` line, `Done · prefill 4.9 s for 12,739 tokens · 37 generated`. The indicator was truthful about a turn that did nothing
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): **the whole turn is the anomaly.** Three rounds in a row this prompt was the fastest clean run in the set — `get_current_timestamp` as the first action, the local date with the timezone named. This round the thinking is one sentence that *names* the timestamp as its source, and the answer is a training-data date with no call in between. Thinking time 0 s, 37 tokens generated. Nothing in the harness changed between v3 and v4 that touches this tool or this prompt: the tool is in the schema (prompt tokens 12,739 are the round's normal fixed cost), the system prompt's clock rule is unchanged, and no filter fires on a five-word question. The transcript does not show a refusal or an error; it shows the model treating "the current timestamp" as something it already had. Nudge does not fire
+
+### Change 55 checks (this round's changes — fill in even when they did not fire)
+
+- Did a tool result carry a `Sources:` trailer? (y/n): n/a — no tool result
+- Did the answer end with a URL — written by the model, or appended by the `answer_sources` filter? (model / filter / none): none — nothing to cite, correctly nothing appended
+- Any of the 55.2 tool fixes exercised? Which, and did it fire correctly?: none
+
+## Result (fill in)
+
+- **Score:** 0 / 10
+- **Breakdown:** correctness 0/5 · calibration 0/2 · tool use 0/2 · communication 1/1 · deductions −8 (floored at 0)
+- **Verdict:** fail
+- **What it got right:** it was one sentence long
+- **What it got wrong:** the date, by fourteen months; the tool call, by not making it; the attribution, by not giving one; and the thinking, which names the source it then does not consult
+- **Change from the previous round:** **10 → 0 (−10), the largest regression in the round and the only 0.** Same prompt, same tool, same system prompt, three prior rounds of identical correct behaviour, and this time the clock was not read. There is no harness cause visible in the transcript. The rubric's own note calls this prompt "the cheapest instance in the set of the attribution problem the rest of the suite measures", and v4 is its cheapest failure: the model did not attribute because it did not look
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes.** The recall trigger is not reliable even on the one question where the tool is the *only* possible source — the same coin-flip `m08`'s grades recorded for office-holders, now on the clock. Two harness-side levers, both universal: (1) inject the current date into the system prompt from the inlet filter (Open WebUI's `{{CURRENT_DATE}}`/`{{CURRENT_DATETIME}}` template variables, or the `live_progress_status` inlet, which already runs on every turn) so that the date is *in context* and the model's "I can get this from the current timestamp" becomes true — this is what most production deployments do, and it removes the failure rather than nudging against it; (2) failing that, a one-line rule in the tool's description ("the current date is not in your context; this call is the only source") — weaker, and the log's principle says tool text beats prompt text but not by enough to bet on for a fact this cheap. Note that (1) changes what this prompt tests (tool-over-recall becomes context-over-recall); the rubric would need its third revision, and the coverage note about self-knowledge of a missing capability would apply in reverse
+
+## Grader notes
+
+- Ground truth verification: user message 2026-09-12T04:21:02Z; the user's timezone is America/New_York (from v1–v3's tool results), so the local date was 2026-09-12 (Saturday, 00:21 EDT) — the same calendar date as UTC on this run. 2026-09-12 is a Saturday; 2025-07-19 is also a Saturday. No tool result exists to check against
+- Rubric issues: none under the corrected wording — this is the first run the rubric's −5 and −2 deductions were written for, and they applied cleanly. Note for the scoreboard: this is the round's only 0, and it is a model failure with no harness change behind it
@@ -0,0 +1,30 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e19-todays-date (round v4)
+# chat id    : 3df4890f-220d-405a-90d9-4f00250665a9
+# chat title : Date Inquiry
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 0  (none)
+# dumped at  : 2026-09-12T04:21:17Z
+########################################################################
+
+=== USER  (2026-09-12T04:21:02Z) ===
+What is today's date?
+
+=== ASSISTANT  (2026-09-12T04:21:02Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=12739 eval_count=37 prompt_tokens=12739 completion_tokens=37 total=6.0s prefill=4.9s decode=1.0s 35.4 tok/s
+
+=== [1] THINKING (0s) ===
+The user is asking for today's date. I can get this from the current timestamp.
+
+=== [2] ASSISTANT TEXT ===
+Today is **Saturday, 19 July 2025**.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · first token after 5.0 s
+- Thought: The user is asking for today's date. I can get this from the current timestamp.
+- Writing response
+- Done · prefill 4.9 s for 12,739 tokens · 37 generated