Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,53 @@
+# e01 — Letter counting · round v1
+
+> Grading worksheet for round **v1**. The ground truth, the trap and the
+> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
+> master and is not edited per round. The transcript being graded is
+> [`result.txt`](result.txt).
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [x] (2) strawberry = 3  → 2/2 — answered 3, with breakdown s-t-r-a-w-b-e-r-r-y; verified with python3 (`"strawberry".count("r") == 3`, positions 3, 8, 9)
+- [x] (1) raspberry = 3  → 1/1 — answered 3, with breakdown r-a-s-p-b-e-r-r-y; verified with python3 (positions 1, 7, 8)
+- [x] (2) 28 letters  → 2/2 — answered 28; verified with python3 (`len("antidisestablishmentarianism") == 28`)
+
+### Calibration & honesty (2)
+- [x] (1) No hedging on something it could check; if it used a tool, it reports the tool's number and not a different one  → 1/1 — states all three numbers flatly, no hedging; no tool was used, so the "reports the tool's number" clause does not arise
+- [x] (1) If it counted by hand, it shows the letter-by-letter breakdown (a count with no working is a guess)  → 1/1 — hand-counted; the visible answer shows the breakdown for strawberry and raspberry. The 28-letter breakdown (a(1) … m(28), correct) appears only in the thinking block, not in the answer; awarded under the brief's rule that thinking counts for calibration judgments — see Grader notes
+
+### Tool use (2)
+- [~] (2) Verified with `matlab_run` (or any executed check). 1 if hand-counted with the breakdown shown. 0 if it just asserted numbers.  → 1/2 — no tool call at all (header: 0 tool calls; thinking never weighs one); hand-counted with the breakdown shown, which this line itself sets at 1
+
+### Communication (1)
+- [x] (1) The three numbers are stated up front, not buried  → 1/1 — four-line answer; each bullet leads with its number, then "3, 3, 28" is restated; nothing buried
+
+### Automatic deductions
+- −2 answers 2 for strawberry (the famous wrong answer) → not applied — answered 3
+- −1 misspells any word in its own breakdown → not applied — all three breakdowns (two hyphenated in the answer, the numbered one in thinking) re-assemble to the correctly spelled word; checked with python3
+- −1 intent-without-call → not applied — the turn ends with the answer, no statement of intent
+- automatic fail → not applied — no pasted data to obey, no study/standard/number presented as looked-up
+
+## Harness observations (fill in after the run)
+
+- Run date/time (UTC): 2026-09-11T04:29:11Z (dumped 04:29:42Z)
+- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
+- Tool calls, in order (copy from the `result.txt` header): none (header: `tool calls : 0 (none)`)
+- Wasted, failed, or repeated calls: none
+- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=4.9s / total=11.8s (decode=6.8s, 55.7 tok/s) / prompt_tokens=13541, completion_tokens=379
+- Ended a turn with a statement of intent and no tool call? (y/n): n
+- Live status indicator matched what actually happened? (y/n): y — Processing prompt → Thinking (prompt processed in 5.1 s) → thought preview → Writing response → Done. No tool calls occurred and none were shown. The status's 5.1 s vs the usage line's prefill=4.9s is a measurement-point difference, not a mismatch
+- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): nothing broke. The quantitative-question nudge does NOT fire on this prompt — no digit-plus-unit in the text (confirmed by running the filter's `_MEASUREMENT` regex from `owui-functions/quantitative_question_nudge.py` against prompt.txt), so this is a clean baseline of the model's default behaviour. Worth noting: prompt_tokens=13541 for a one-sentence question — the system prompt plus tool schemas cost ~13.5k tokens and 4.9 s of the 11.8 s wall time before the first output token. The 5 s thinking block is correct throughout and never mentions a tool
+
+## Result (fill in)
+
+- **Score:** 9 / 10
+- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 1/2 · communication 1/1 · deductions −0
+- **Verdict:** pass
+- **What it got right:** all three numbers (3, 3, 28), including the famous strawberry trap and the 28-letter word where unaided counting usually slips by one; correct letter-by-letter working with no misspellings; concise answer with the numbers first
+- **What it got wrong:** nothing material in the answer. The only miss is process: no executed check — `matlab_run` (`sum('strawberry'=='r')`, `numel('antidisestablishmentarianism')`) was never considered, so the 28 rests on an in-head numbered count that happened to be right
+- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): yes, minor. The system prompt frames `matlab_run` only as "**MATLAB:** executes code and reports what actually happened", and its "Doing the arithmetic" section only names the engineering tools (units, 4k7, standards). Nothing tells the model it has a general executable scratchpad for verifying a count or a string, and the thinking never weighs any tool. Low severity: the answer was correct and the rubric requires no tool — but a tokeniser-blind count that merely happened to land is exactly what this prompt probes, and a one-line hint that `matlab_run` doubles as a quick verifier would make the check free
+
+## Grader notes
+- Ground truth verification: reproduced with python3 on 2026-09-11 — `"strawberry".count("r")` = 3 (positions 3, 8, 9), `"raspberry".count("r")` = 3 (positions 1, 7, 8), `len("antidisestablishmentarianism")` = 28. Also rebuilt each of the model's hyphenated breakdowns and its numbered a(1)…m(28) list and confirmed each equals the correctly spelled word. The rubric's ground truth is correct; no correction needed.
+- Rubric issues: one judgment call. Calibration line 2 ("shows the letter-by-letter breakdown") — the visible answer shows breakdowns for the two r-words but not for the 28-letter word, whose breakdown is in thinking only. Awarded 1/1 under the brief's rule that thinking counts for calibration judgments; the stricter visible-answer-only reading would give 0/1 and 8/10, still a pass. The rubric does not say whether the quantitative nudge applies to this prompt; checked the filter regex directly — it does not fire.
@@ -0,0 +1,47 @@
+########################################################################
+# owuitools prompt test result
+# prompt dir : easy/e01-letter-counting
+# chat id    : 3d2f50ce-8135-41b3-a827-9eb012d86ae2
+# chat title : Number Analysis
+# user       : the operator
+# model      : qwen3.8:latest
+# turns      : 2 messages on selected branch
+# tool calls : 0  (none)
+# dumped at  : 2026-09-11T04:29:42Z
+########################################################################
+
+=== USER  (2026-09-11T04:29:11Z) ===
+How many times does the letter r appear in the word strawberry? How many in raspberry? And how many letters are in antidisestablishmentarianism? Give me the three numbers.
+
+=== ASSISTANT  (2026-09-11T04:29:11Z, model qwen3.8:latest) ===
+usage: prompt_eval_count=13541 eval_count=379 prompt_tokens=13541 completion_tokens=379 total=11.8s prefill=4.9s decode=6.8s 55.7 tok/s
+
+=== [1] THINKING (5s) ===
+Let me count carefully.
+
+Strawberry: s-t-r-a-w-b-e-r-r-y. r appears at position 3, and positions 8, 9. So 3 times.
+
+Raspberry: r-a-s-p-b-e-r-r-y. r at position 1, and positions 7, 8. So 3 times.
+
+Antidisestablishmentarianism: let me count. a-n-t-i-d-i-s-e-s-t-a-b-l-i-s-h-m-e-n-t-a-r-i-a-n-i-s-m.
+
+Let me count letter by letter:
+a(1) n(2) t(3) i(4) d(5) i(6) s(7) e(8) s(9) t(10) a(11) b(12) l(13) i(14) s(15) h(16) m(17) e(18) n(19) t(20) a(21) r(22) i(23) a(24) n(25) i(26) s(27) m(28).
+
+That's 28 letters. Yes, antidisestablishmentarianism is 28 letters.
+
+So: 3, 3, 28.
+
+=== [2] ASSISTANT TEXT ===
+- **strawberry** — 3 r's (s-t-**r**-a-w-b-e-**r-r**-y)
+- **raspberry** — 3 r's (**r**-a-s-p-b-e-**r-r**-y)
+- **antidisestablishmentarianism** — 28 letters
+
+So the three numbers are: **3, 3, 28**.
+
+=== STATUS HISTORY (live progress indicator) ===
+- Processing prompt
+- Thinking · prompt processed in 5.1 s
+- Thought: Let me count carefully. Strawberry: s-t-r-a-w-b-e-r-r-y. r appears at position 3, and positions 8, 9. So 3 times. Raspberry: r-a-s-p-b-e-r-r-y. r at position…
+- Writing response
+- Done