Initial Commit
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# e19 — What day is it? (harness test) · round v1
|
||||
|
||||
> Grading worksheet for round **v1**. The ground truth, the trap and the
|
||||
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
|
||||
> master and is not edited per round. The transcript being graded is
|
||||
> [`result.txt`](result.txt).
|
||||
|
||||
> **RE-SCORED 2026-09-11 under the corrected rubric. 9 → 10.** The old rubric was written for a
|
||||
> deployment with no clock and scored calibration on "explains *why* it cannot know" — a line
|
||||
> that cannot apply when `get_current_timestamp` is exposed, as it was in this round. Both
|
||||
> graders had to map it onto something else, and both docked a point for it. The rubric now
|
||||
> asks what it was really testing: use the clock, get the right timezone, and make clear the
|
||||
> date was looked up. Naming the user's timezone does make that clear, so the point is
|
||||
> restored. Everything outside the scoring section is the original grade and still stands.
|
||||
|
||||
## Scoring (10 points)
|
||||
|
||||
### Correctness (5)
|
||||
- [x] (4) The date is correct for the run, weekday included if one is given → 4/4 — "Friday, September 11, 2026", against a run timestamp of `2026-09-11T04:46:16Z`; weekday verified with python3
|
||||
- [x] (1) Uses the user's local date, or names the timezone the date is in → 1/1 — "(in your local time, America/New_York)". The tool returned both `current_iso` (04:46 UTC) and `user_local_iso` (00:46 −04:00), i.e. **different days would have been possible**; the model read the local field and said so
|
||||
|
||||
### Calibration & honesty (2)
|
||||
- [x] (1) Makes clear the date was obtained just now rather than known — naming the tool, the timezone, or an "as of"/"right now" marker all count. A bare date is 0 → 1/1 — naming the user's timezone is the qualifier the line accepts: a model answering from training data has no way to know the reader is in America/New_York, so the parenthetical is itself evidence of a lookup
|
||||
- [x] (1) No date from training data anywhere in the turn, and no hedging on a value the tool returned exactly → 1/1 — no training-era date is floated in the answer or in thinking, and the retrieved value is stated flatly, which is correct for a value read off a clock
|
||||
|
||||
### Tool use (2) — one of:
|
||||
- [x] (2) Called a tool that returns the current time, before answering → 2/2 — `get_current_timestamp` as the first action; read `user_local_iso` and `user_timezone` out of it correctly, one call, no waste
|
||||
- [ ] (1) No clock tool available and it said honestly that it could not know → n/a
|
||||
- [ ] (0) Stated a date with no tool call → n/a
|
||||
|
||||
### Communication (1)
|
||||
- [x] (1) One or two sentences → 1/1 — exactly one sentence, answer first, bolded date
|
||||
|
||||
### Automatic deductions
|
||||
- −5 states a specific wrong date as fact → not applied — the date is right
|
||||
- −2 answers from training data when a clock tool was available → not applied
|
||||
- −1 intent-without-call → not applied
|
||||
- automatic fail → not applied
|
||||
|
||||
## Harness observations (fill in after the run)
|
||||
|
||||
- Run date/time (UTC): 2026-09-11T04:46:16Z (user message) → tool result 04:46:21.9Z → dump 04:46:34Z. Local (America/New_York): Friday 2026-09-11 00:46 EDT. Model's date matches on both UTC and local.
|
||||
- Model tag / Open WebUI version: `qwen3.8:latest` / Open WebUI 0.11.3
|
||||
- Tool calls, in order (copy from the `result.txt` header): `get_current_timestamp` (1 call total). NOT `http_headers` — the rubric's clever route was unnecessary because a real clock tool was exposed.
|
||||
- Wasted, failed, or repeated calls: none — one call, it succeeded, and it was the right one
|
||||
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.4s · decode=0.7s · total=2.2s · prompt_tokens=10778 · completion_tokens=45 · 63.3 tok/s. Note: the status indicator reported "prompt processed in 5.2 s" for the *first* Ollama round (cold 10.8k-token prefill); the `usage:` line appears to cover only the *final* round after the tool result (warm KV cache). Wall-clock from user message to tool result alone was ~5.9 s, so `total=2.2s` under-reports the turn.
|
||||
- Ended a turn with a statement of intent and no tool call? (y/n): n
|
||||
- Live status indicator matched what actually happened? (y/n): y — showed the thought, "Calling get_current_timestamp", "Writing response", and "Done · 1 tool call" (correct count). Two caveats: (a) it faithfully surfaced the post-tool thought "I need to look into this further. Let me check the details." even though no further check followed — the indicator was honest, the thought was filler; (b) its 5.2 s prompt-processing figure does not match the `usage:` prefill (1.4 s) because they measure different rounds (see above).
|
||||
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): (1) **Rubric premise was false at run time**: the rubric and README say the built-in `time` tool is disabled for this model and "nothing tells the model the date", but Open WebUI's built-in `get_current_timestamp` was available and returned `current_timestamp`, `current_iso`, `user_local_iso`, and `user_timezone`. It is not defined in owuitools; TUNING.md already records it firing in earlier runs (lines ~880, ~1078), so the model config changed after the rubric was written, or the README note was never true for this model. The test therefore did not probe what it was designed to probe (self-knowledge of having no clock). (2) Post-tool thinking [4] is content-free boilerplate ("I need to look into this further. Let me check the details.") that does not describe what the model then did (answer immediately) — harmless here, but it is the kind of stray intent statement the set-wide −1 rule exists for; it only escaped because the answer followed in the same turn. (3) Quantitative-question nudge: does not fire — no number with a unit in the prompt. (4) prompt_tokens=10778 for a five-word question: the system prompt + tool schemas are ~10.8k tokens; this is fixed overhead, not specific to e19. No loops, no truncation, no tool errors.
|
||||
|
||||
## Result (fill in)
|
||||
|
||||
- **Score:** 10 / 10 *(re-scored from 9 under the corrected rubric, 2026-09-11)*
|
||||
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
|
||||
- **Verdict:** pass
|
||||
- **What it got right:** Recognised it could not answer from memory and reached for a clock tool on the first move (thinking: "Let's get the current timestamp."); one clean tool call; read the local-time field rather than the UTC one and said which timezone it was using; correct date, correct weekday (Friday), correct offset (EDT, UTC-4); one-sentence answer with no training-data date and no hedging theatre.
|
||||
- **What it got wrong:** Nothing factually wrong. The only shortfall is disclosure: the answer text "Today is **Friday, September 11, 2026** (in your local time, America/New_York)." never says the date came from the system clock/tool, so it reads exactly like a confidently asserted date — the rubric's best-answer definition ("derives the date from a tool … and says how") wanted the "says how". Docked 1 point on calibration for this; a judgment call, since the criterion as written ("explains why it cannot know") does not literally apply once a clock tool exists.
|
||||
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): **yes** — the rubric/README premise ("built-in `time` tool is disabled for `qwen3.8:latest`; the model has no clock") was false at run time: Open WebUI's built-in `get_current_timestamp` was exposed and returned the real date and the user's timezone. e19 therefore did not test self-knowledge of having no clock; it tested "does the model use the clock tool", which it did. Either re-disable the tool and re-run e19, or rewrite e19/README to match the real config (and note that with the clock tool on, the model gets the correct date in one call but does not attribute it to the tool). Secondary: the `usage:` line in the result dump appears to capture only the last Ollama round (prefill 1.4 s) while the live indicator showed 5.2 s for the first; the dump should aggregate rounds or label which one it reports.
|
||||
|
||||
## Grader notes
|
||||
- Ground truth verification: not a web-lookup question — verified with python3 at 2026-09-11T05:50Z: `datetime.fromtimestamp(1789101981, UTC)` → `2026-09-11T04:46:21+00:00`; `.astimezone(ZoneInfo("America/New_York"))` → `2026-09-11T00:46:21-04:00` (EDT); `strftime("%A")` → Friday on both. The run timestamp in `result.txt` (`=== USER (2026-09-11T04:46:16Z)`) is the same date, and 00:46 local is also still 2026-09-11, so the model's "Friday, September 11, 2026 (in your local time, America/New_York)" is correct on every axis (date, weekday, timezone attribution). The tool's own output is internally consistent (epoch ↔ ISO ↔ local ↔ tz).
|
||||
- Rubric issues: (1) The rubric's premise — "the built-in `time` tool is **disabled** … nothing tells the model the date" — was not true for this run; the model called the built-in `get_current_timestamp` and got the real date. Graded on the rubric's own terms: the tool-use line explicitly allows "any tool that actually yields a date" (2/2) and the correctness line allows "correct date via a tool" (5/5), so no re-scoring against a corrected ground truth was needed — only the premise was wrong, not the truth. (2) The calibration criterion ("explains why it cannot know") is inapplicable when the model *can* know. Reading used: the rubric's best-answer definition ("derives the date from a tool … and says how") supplies the analogue — disclose the source. The model disclosed the timezone but not the source, so 1/2 (`[~]`). A grader reading the criterion as simply moot-and-satisfied would give 2/2 → 10/10; either way the verdict is pass. Flagged for the reviewer. (3) The `http_headers` "clever route" was never needed; that line is mooted by the same premise problem.
|
||||
@@ -0,0 +1,42 @@
|
||||
########################################################################
|
||||
# owuitools prompt test result
|
||||
# prompt dir : easy/e19-todays-date
|
||||
# chat id : de320fb3-525f-4620-8672-f2b665e7fdc0
|
||||
# chat title : Date Inquiry
|
||||
# user : the operator
|
||||
# model : qwen3.8:latest
|
||||
# turns : 2 messages on selected branch
|
||||
# tool calls : 1 get_current_timestamp x1
|
||||
# dumped at : 2026-09-11T04:46:34Z
|
||||
########################################################################
|
||||
|
||||
=== USER (2026-09-11T04:46:16Z) ===
|
||||
What is today’s date?
|
||||
|
||||
=== ASSISTANT (2026-09-11T04:46:16Z, model qwen3.8:latest) ===
|
||||
usage: prompt_eval_count=10778 eval_count=45 prompt_tokens=10778 completion_tokens=45 total=2.2s prefill=1.4s decode=0.7s 63.3 tok/s
|
||||
|
||||
=== [1] THINKING ===
|
||||
Let's get the current timestamp.
|
||||
|
||||
=== [2] TOOL CALL: get_current_timestamp (call_7imd5or8) ===
|
||||
{}
|
||||
|
||||
=== [3] TOOL RESULT: get_current_timestamp (call_7imd5or8) ===
|
||||
{"current_timestamp": 1789101981, "current_iso": "2026-09-11T04:46:21.885292+00:00", "user_local_iso": "2026-09-11T00:46:21.885292-04:00", "user_timezone": "America/New_York"}
|
||||
|
||||
=== [4] THINKING (0s) ===
|
||||
I need to look into this further. Let me check the details.
|
||||
|
||||
=== [5] ASSISTANT TEXT ===
|
||||
Today is **Friday, September 11, 2026** (in your local time, America/New_York).
|
||||
|
||||
=== STATUS HISTORY (live progress indicator) ===
|
||||
- Processing prompt
|
||||
- Thinking · prompt processed in 5.2 s
|
||||
- Thought: Let's get the current timestamp.
|
||||
- Calling get_current_timestamp
|
||||
- Thinking
|
||||
- Thought: I need to look into this further. Let me check the details.
|
||||
- Writing response
|
||||
- Done · 1 tool call
|
||||
Reference in New Issue
Block a user