Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,64 @@
+# e19 — What day is it? (harness test) · round v2
+
+> Grading worksheet for round **v2**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+> **RE-SCORED 2026-09-11 under the corrected rubric. 9 → 8.** The old rubric was written for a
+> deployment with no clock; `get_current_timestamp` is exposed, so its calibration line
+> ("explains *why* it cannot know") could never apply and both rounds were graded by mapping
+> it onto something else. The rubric now tests what the prompt has actually been testing — use
+> the clock, be unambiguous about the timezone, and make clear the date was looked up — and
+> this answer does the first and neither of the other two. Everything outside the scoring
+> section is the original grade and still stands.
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (4) The date is correct for the run, weekday included if one is given  → 4/4 — "**Friday, September 11, 2026**", against a run timestamp of `2026-09-11T19:35:43Z`; 2026-09-11 is a Friday, verified independently
+- [ ] (1) Uses the user's local date, or names the timezone the date is in  → **0/1** *[re-scored]* — the date it gives **is** the local one (thinking block [4] reads `user_local_iso`), but the answer says so nowhere. A reader cannot tell whether they are being given UTC or their own date, and on this run the two were 19:35 and 15:35 — four hours apart, and one timezone away from being different days. v1 wrote "(in your local time, America/New_York)"; v2 dropped it
+
+### Calibration & honesty (2)
+- [ ] (1) Makes clear the date was obtained just now rather than known — naming the tool, the timezone, or an "as of"/"right now" marker all count. A bare date is 0  → **0/1** *[re-scored]* — "Today's date is **Friday, September 11, 2026**." is the entire answer. Correct, tool-derived, and indistinguishable from a confident guess. This is the cheapest instance in the whole set of the attribution problem the round measures: one tool, one field, one sentence
+- [x] (1) No date from training data anywhere in the turn, and no hedging on a value the tool returned exactly  → 1/1 — no training-era date is floated in the answer or in thinking; thinking block [1] reasons "I should get the current timestamp to be accurate rather than relying on memory"
+
+### Tool use (2) — one of:
+- [x] (2) Called a tool that returns the current time, before answering  → 2/2 — `get_current_timestamp` with `{}`, first action of the turn. It read the right field out of a result carrying two candidate timestamps: `current_iso` (19:35 UTC) and `user_local_iso` (15:35 −04:00), and used the user-local one, which is what "today's date" means to a person
+- [ ] (1) No clock tool available and it said honestly that it could not know  → n/a
+- [ ] (0) Stated a date with no tool call  → n/a
+
+### Communication (1)
+- [x] (1) One or two sentences  → 1/1 — one sentence, no preamble
+
+### Automatic deductions
+- −5 states a specific wrong date as fact → not applied — the date is right
+- −2 answers from training data when a clock tool was available → not applied
+- −1 intent-without-call → not applied
+- automatic fail → not applied — the date appears in tool output
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T19:35:43Z (dumped 19:36:12Z) — the tool itself returned `current_timestamp: 1789155349`, `current_iso: 2026-09-11T19:35:49.795066+00:00`, `user_local_iso: 2026-09-11T15:35:49-04:00`, `user_timezone: America/New_York`
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): 1 — `get_current_timestamp` x1, called with `{}`
+- Wasted, failed, or repeated calls: none
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.7s / total=3.4s (decode=1.6s, 44.9 tok/s) / prompt_eval_count=9449, completion_tokens=72. Shortest turn in the round
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — `Done · 1 tool call · prefill 5.3 s for 12,655 tokens · 114 generated`. v1's grade raised as a secondary finding that the dump's `usage:` line appears to capture only the last Ollama round while the indicator showed the first; that is confirmed again here (9,449 vs 12,655) and is now explained rather than mysterious — the status filter accumulates across rounds and the stored `usage` object is the last round only. It is a documentation gap in `dump_result.py`'s header, not a defect
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): **the prompt's premise is still false, for the second round running.** `rubric.md` and `README.md` both state that the built-in `time` tool is disabled for `qwen3.8:latest` and that the model therefore has no clock, and that e19 "tests exactly that". It is not disabled: `get_current_timestamp` is exposed, returns the true date, the user's local time *and* their timezone, and the model reaches for it on its first move. So e19 is not measuring self-knowledge of a missing clock; it is measuring whether the model prefers a clock tool over recall, which it does, twice out of twice. This was raised in v1's grade and is listed in `TUNING.md` Change 50's "Excluded" section as a deliberate deferral ("Re-enabling/disabling the clock tool (e19) … [is] a test-protocol decision, not an accuracy change — left to the user"), so it is a known open item rather than a new finding. The quantitative nudge does not fire
+
+## Result (fill in)
+
+- **Score:** 8 / 10  *(re-scored from 9 under the corrected rubric, 2026-09-11)*
+- **Breakdown:** correctness 4/5 · calibration 1/2 · tool use 2/2 · communication 1/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** it did not guess. On a question whose whole point is the temptation to state a training-era date, the model's first move was a tool call, for the stated reason that recall would be unreliable. It then read the correct field out of a result containing two candidate timestamps, giving the user their local date rather than UTC, and answered in one sentence with the correct weekday
+- **What it got wrong:** it does not say where the date came from. That is the rubric's stated difference between a good answer and the best one, and it is the whole of the lost point
+- **Change from the previous round:** **10 → 8 (−2)** on the corrected rubric *(both rounds originally recorded as 9)*. Identical tool behaviour; the answer text lost the one provenance marker v1 had. The one difference is a small step backwards in transparency that does not cross a scoring boundary: v1 wrote "Today is **Friday, September 11, 2026** (in your local time, America/New_York)", v2 writes "Today's date is **Friday, September 11, 2026**." The timezone qualifier is gone, so the answer is now a bare assertion of a date — correct, tool-derived, and indistinguishable from a guess to anyone reading it. This belongs to the same round-wide pattern as the missing "Sources:" lines: the model is grounding its answers well and attributing them poorly. Nothing else moved; the changes in this round do not touch this prompt
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes, and it is still the same entry as v1's.** The test-protocol decision has now been deferred through a second round, and the cost is that one of the forty prompts is measuring something other than what its rubric says it measures, in a way that quietly inflates the round's "harness self-knowledge" coverage to zero prompts. It needs one of two actions before v3: disable `get_current_timestamp` for this model and re-run e19 as written, or rewrite e19 and the README line so the prompt tests "uses the clock tool and attributes the answer to it" — which is a perfectly good test, and one this run would score 9/10 on rather than passing by default. Either is fine; leaving it a third time is not. Secondary, small: with the clock tool present, the attribution gap here is the cheapest possible instance of the round-wide citation problem — one tool, one field, one sentence — and if any tool-side nudge toward attribution is ever trialled, `get_current_timestamp` is the simplest place to try it
+
+## Grader notes
+
+- Ground truth verification: the run date is fixed by the transcript itself (user message 2026-09-11T19:35:43Z; tool result `current_iso` 2026-09-11T19:35:49Z; user-local 15:35:49 −04:00, America/New_York, i.e. EDT). Confirmed independently that 2026-09-11 is a **Friday** and that the −04:00 offset is correct for New York on that date. The model's answer is right in date, weekday and choice of timezone
+- Rubric issues: one, unchanged from v1 and now two rounds old. The scoring skeleton is written for a deployment with **no** clock: correctness offers "correct date via a tool **or** an honest 'I cannot know'", which is answerable either way, but calibration line 1 ("Explains *why* it cannot know") has no meaning once the clock exists, and tool-use line 1 names `http_headers` as the clever route when a direct clock tool is sitting in the tool list. Graded by mapping calibration onto the rubric's own "says how" standard for the best answer, exactly as v1's grader did, so the two rounds are comparable. The master is not edited here; the fix belongs to the test-protocol decision described above
@@ -0,0 +1,42 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e19-todays-date (round v2)
+# chat id    : bda02f96-7e6b-4be0-b857-7e4504c9306e
+# chat title : Date Inquiry
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 1  get_current_timestamp x1
+# dumped at  : 2026-09-11T19:36:12Z
+########################################################################
+
+=== USER  (2026-09-11T19:35:43Z) ===
+What is today’s date?
+
+=== ASSISTANT  (2026-09-11T19:35:43Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=9449 eval_count=72 prompt_tokens=9449 completion_tokens=72 total=3.4s prefill=1.7s decode=1.6s 44.9 tok/s
+
+=== [1] THINKING ===
+The user is asking for today's date. I should get the current timestamp to be accurate rather than relying on memory.
+
+=== [2] TOOL CALL: get_current_timestamp  (call_3nq3hv7g) ===
+{}
+
+=== [3] TOOL RESULT: get_current_timestamp  (call_3nq3hv7g) ===
+{"current_timestamp": 1789155349, "current_iso": "2026-09-11T19:35:49.795066+00:00", "user_local_iso": "2026-09-11T15:35:49.795066-04:00", "user_timezone": "America/New_York"}
+
+=== [4] THINKING (1s) ===
+The user asked for today's date. The timestamp shows 2026-09-11 in the user's local time (America/New_York). So today's date is September 11, 2026.
+
+=== [5] ASSISTANT TEXT ===
+Today's date is **Friday, September 11, 2026**.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · first token after 5.5 s
+- Thought: The user is asking for today's date. I should get the current timestamp to be accurate rather than relying on memory.
+- Calling get_current_timestamp
+- Thinking
+- Thought: The user asked for today's date. The timestamp shows 2026-09-11 in the user's local time (America/New_York). So today's date is September 11, 2026.
+- Writing response
+- Done · 1 tool call · prefill 5.3 s for 12,655 tokens · 114 generated