Grant/owui-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
Grant
2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/prompt.txt
+1
View File
@@ -0,0 +1 @@
I drop a 2 kg steel ball from a 45 m tower. Ignoring air resistance, how long does it take to hit the ground and how fast is it going on impact? If I used a 4 kg ball instead, how do the answers change?
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/rubric.md
+46
View File
@@ -0,0 +1,46 @@
# e08 — Free fall with a mass distractor
| | |
|---|---|
| **Difficulty** | easy |
| **Domain** | physics |
| **Probes** | basic kinematics; whether the mass distractor is handled; whether the number goes through a tool |
| **Tools expected** | `matlab_run` (the formula is trivial, but the system prompt says do arithmetic with a tool) |
| **Trap** | The 4 kg question. Also g: 9.8 vs 9.81 — either is fine if stated. The inlet nudge fires (units present). |
## Ground truth
- t = √(2h/g) = √(90/9.81) = **3.03 s**
- v = g·t = √(2gh) = **29.7 m/s** (≈ 107 km/h)
- Mass is irrelevant without air resistance: **no change** for 4 kg.
## Scoring (10 points)
### Correctness (5)
- [ ] (2) t ≈ 3.0 s
- [ ] (2) v ≈ 29.7 m/s
- [ ] (1) Mass makes no difference
### Calibration & honesty (2)
- [ ] (1) States the g it used
- [ ] (1) Does not invent air-resistance numbers when asked to ignore it (a one-line "with air resistance the heavier ball would…" caveat is fine)
### Tool use (2)
- [ ] (2) `matlab_run` for the numbers. 1 if hand-calculated with the formula shown and correct.
### Communication (1)
- [ ] (1) Numbers first, units, brief
### Automatic deductions
- −3 says the heavier ball falls faster or slower
- −1 intent-without-call
- automatic fail
## Grading
One worksheet per round, alongside that round's transcript:
- `v1/grade.md` · `v1/result.txt` — first round (baseline)
- `v2/grade.md` · `v2/result.txt` — second round
Copy the scoring skeleton above into a new round's `grade.md` to add one.
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/v1/grade.md
+53
View File
@@ -0,0 +1,53 @@
# e08 — Free fall with a mass distractor · round v1
> Grading worksheet for round **v1**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (2) t ≈ 3.0 s → 2/2 — final answer gives **3.03 s**; matlab_run output 3.0289 s, python3 reproduces 3.0289 s at g = 9.81 (3.0305 s at g = 9.8)
- [x] (2) v ≈ 29.7 m/s → 2/2 — final answer gives **29.7 m/s** (≈107 km/h ≈ 66 mph); tool output 29.7136 m/s matches python3 exactly
- [x] (1) Mass makes no difference → 1/1 — "The answers are exactly the same — 3.03 s and 29.7 m/s", with the correct force-vs-inertia cancellation explanation (Galileo)
### Calibration & honesty (2)
- [x] (1) States the g it used → 1/1 — g = 9.81 is written into both formulas and stated explicitly as "g = 9.81 m/s²"
- [x] (1) Does not invent air-resistance numbers when asked to ignore it (a one-line "with air resistance the heavier ball would…" caveat is fine) → 1/1 — the "one caveat" section is purely qualitative ("slightly different times", "a touch faster"); no drag figures invented. It is a 4-sentence paragraph rather than the rubric's "one line", but the criterion is about invented numbers, so full marks
### Tool use (2)
- [x] (2) `matlab_run` for the numbers. 1 if hand-calculated with the formula shown and correct. → 2/2 — one clean matlab_run call computing t, v, km/h and mph; the reported numbers are the tool's output. The model also pre-computed by hand in thinking (3.0289 / 29.714) before calling the tool — process, not a wasted call
### Communication (1)
- [x] (1) Numbers first, units, brief → 1/1 — bold numbers with units in the first two bullets, 4 kg answer bolded up front; ~180 words in three short sections. The air-resistance caveat is the only part longer than needed
### Automatic deductions
- −3 says the heavier ball falls faster or slower → not applied — says "exactly the same" for the no-drag case the question asked about; the "a touch faster" remark is explicitly confined to the with-air-resistance caveat the rubric permits, and its reasoning (drag is a smaller fraction of weight for the larger ball) is physically correct
- −1 intent-without-call → not applied — the turn ended with the full answer, no dangling "let me check"
- automatic fail → not applied — no pasted data to obey, no study/standard/number presented as looked-up
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T04:34:01Z (user message); transcript dumped 04:35:01Z
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): matlab_run (1 call total: `matlab_run x1`)
- Wasted, failed, or repeated calls: none — single call, returned OK in 0.48 s, output used verbatim in the answer
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.9s / decode=9.9s / total=12.9s; prompt_tokens=12002, completion_tokens=403, 40.6 tok/s
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — Processing prompt → Thinking → Thought (excerpt) → Calling matlab_run → Thinking → Thought → Writing response → Done · 1 tool call; the count and sequence match the numbered blocks
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): The inlet nudge applies (kg, m present). Thinking shows the model explicitly enumerating the tool list, rejecting `ohms_law_power`, and picking `matlab_run` because "I'm not doing arithmetic in my head" — consistent with the nudge doing its job. It still hand-computed the values in thinking first, then confirmed with the tool. No loops, truncation, or tool errors. Minor: the status line reported "prompt processed in 5.2 s" while the usage line says prefill=2.9s — probably wall-clock-to-first-token vs Ollama's prompt_eval_duration, not a misreport of events.
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** Both numbers (3.03 s, 29.7 m/s) with units and km/h / mph conversions; g = 9.81 stated; the 4 kg trap handled cleanly — "exactly the same" — with the correct explanation of why mass cancels; numbers came from a single matlab_run call rather than mental arithmetic; the air-resistance caveat stayed qualitative and was clearly labelled as outside the stated assumption.
- **What it got wrong:** Nothing material. The only nit is length: the rubric allows a one-line air-resistance caveat and the model wrote four sentences ("A steel ball is dense and aerodynamic… the heavier one would be a touch faster…"). Physically correct, no invented numbers, just more than asked.
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no — model and harness both behaved as designed; the nudge fired and the tool got used. One low-priority observation for the harness, not the model: status indicator "prompt processed in 5.2 s" vs usage prefill=2.9s; worth a line only if other graders see the same gap across runs.
## Grader notes
- Ground truth verification: not needed from the web — arithmetic reproduced with python3 on 2026-09-11 ~05:49 UTC: t = sqrt(2·45/g), v = sqrt(2·g·45) → 3.0289 s / 29.7136 m/s (106.97 km/h, 66.47 mph) at g = 9.81; 3.0305 s / 29.6985 m/s at g = 9.8; 3.0294 s / 29.7086 m/s at g = 9.80665. All round to the rubric's 3.03 s / 29.7 m/s (≈107 km/h). a = m·g/m = g for both 2 kg and 4 kg, so mass is irrelevant. The model's matlab_run output matches python3 to four decimals.
- Rubric issues: none. Ground truth is correct as written. One reading choice: the calibration criterion "does not invent air-resistance numbers" was read as being about fabricated figures, not about caveat length, so the 4-sentence qualitative caveat earned full marks; likewise "brief" in Communication was judged on the answer as a whole (~180 words, numbers up front), not on the caveat alone.
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/v1/result.txt
+122
View File
@@ -0,0 +1,122 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e08-ball-drop-mass
# chat id : 4fb2f300-c5b1-4f8e-a3c7-699076c3d019
# chat title : Time of Impact and Velocity Calculation
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 matlab_run x1
# dumped at : 2026-09-11T04:35:01Z
########################################################################
=== USER (2026-09-11T04:34:01Z) ===
I drop a 2 kg steel ball from a 45 m tower. Ignoring air resistance, how long does it take to hit the ground and how fast is it going on impact? If I used a 4 kg ball instead, how do the answers change?
=== ASSISTANT (2026-09-11T04:34:01Z, model qwen3.8:latest) ===
usage: prompt_eval_count=12002 eval_count=403 prompt_tokens=12002 completion_tokens=403 total=12.9s prefill=2.9s decode=9.9s 40.6 tok/s
=== [1] THINKING ===
The user is asking a physics question: dropping a 2 kg steel ball from 45 m, ignoring air resistance. Find time to hit ground and impact velocity. Then compare with a 4 kg ball.
This is a classic kinematics problem. The key physics here is that in the absence of air resistance, mass doesn't matter — all objects fall at the same rate. So the answers are identical for both masses.
Let me work through the math. This is a calculation, so I should verify it with a tool rather than doing it in my head. The tool `ohms_law_power` is for Ohm's law, not kinematics. There's no dedicated kinematics tool. Let me think about what tools are available.
Available tools:
- web_search, fetch_page, wikipedia_lookup
- search_stackexchange, lookup_rfc, lookup_cve, read_man_page, lookup_package
- subnet_calc, dns_lookup, inspect_tls, http_headers, lookup_ip_or_asn, network_reference
- ohms_law_power, component_calc, pcb_calc, wiring_calc
- find_datasheet, search_papers
- matlab_run
- timer, create_tasks, update_task
- create_automation, etc.
For a kinematics calculation, `matlab_run` is the right tool — I can write a small script that computes the values and prints them. That way I'm not doing arithmetic in my head, and I get exact numbers.
The physics:
- Initial velocity u = 0 (dropped, not thrown)
- Acceleration a = g = 9.81 m/s² (standard gravity)
- Distance s = 45 m
Using s = ut + (1/2)at²:
45 = 0 + (1/2)(9.81)t²
45 = 4.905 t²
t² = 45/4.905 = 9.1743...
t = 3.0289... s
Impact velocity: v = u + at = 0 + 9.81 × 3.0289 = 29.714... m/s
Or using v² = u² + 2as = 0 + 2(9.81)(45) = 882.9
v = √882.9 = 29.714... m/s
Let me verify: 2 × 9.81 × 45 = 882.9. √882.9 = ?
29² = 841, 30² = 900. So between 29 and 30.
29.7² = 882.09
29.71² = 882.684
29.72² = 883.278
So √882.9 ≈ 29.714 m/s.
Time: t = √(2s/g) = √(90/9.81) = √9.17431 = 3.0289 s.
3.0289² = 9.1742. Good.
Let me use matlab_run to compute this precisely and show the work. I'll use g = 9.81 m/s² (standard). I could note that g varies slightly by location (9.78 at equator, 9.83 at poles), but 9.81 is the standard value.
The mass comparison: In vacuum (ignoring air resistance), the mass is irrelevant. Galileo's principle — all objects fall at the same acceleration regardless of mass. So the 4 kg ball gives identical answers: ~3.03 s and ~29.7 m/s.
I should make this clear and explain WHY — this is the conceptual point of the question. The user might be testing whether I understand that mass doesn't matter in free fall, or they might genuinely be unsure.
Let me run the matlab to get clean numbers. I'll compute with g = 9.81.
Actually, let me be careful and present it cleanly. Let me write a script.
Let me do the matlab run.
=== [2] TOOL CALL: matlab_run (call_91wjwr65) ===
{
"code": "g = 9.81; % m/s^2, standard gravity\ns = 45; % m, drop height\nu = 0; % m/s, dropped from rest\n\n% Time from s = u*t + 0.5*g*t^2 -> t = sqrt(2s/g)\nt = sqrt(2*s/g);\n\n% Impact speed from v^2 = u^2 + 2*g*s\nv = sqrt(u^2 + 2*g*s);\n\nfprintf('t = %.4f s\\n', t);\nfprintf('v = %.4f m/s\\n', v);\nfprintf('v = %.2f km/h\\n', v*3.6);\nfprintf('v = %.2f mph\\n', v*2.236936);\n"
}
=== [3] TOOL RESULT: matlab_run (call_91wjwr65) ===
# MATLAB run OK (0.48s)
## Output
```
t = 3.0289 s
v = 29.7136 m/s
v = 106.97 km/h
v = 66.47 mph
```
=== [4] THINKING (0s) ===
Now I have clean numbers. Let me present the answer clearly, including the key conceptual point that mass doesn't matter in free fall (ignoring air resistance).
Let me write a clear, self-contained answer.
=== [5] ASSISTANT TEXT ===
## 2 kg ball dropped from 45 m (no air resistance)
Starting from rest, the only thing that matters is the height and gravity:
- **Time to hit:** $t = \sqrt{2s/g} = \sqrt{2 \times 45 / 9.81} \approx \mathbf{3.03\ s}$
- **Impact speed:** $v = \sqrt{2gs} = \sqrt{2 \times 9.81 \times 45} \approx \mathbf{29.7\ m/s}$ (≈ 107 km/h ≈ 66 mph)
## 4 kg ball
**The answers are exactly the same — 3.03 s and 29.7 m/s.**
This is the key point: in free fall with no air resistance, mass cancels out. Gravity accelerates every object at the same $g = 9.81\ \text{m/s}^2$ regardless of its weight (Galileo's result). The heavier ball has more gravitational force on it, but also proportionally more inertia, so the two effects offset perfectly.
## The one caveat
The "ignore air resistance" assumption is doing real work here. A steel ball is dense and aerodynamic, so it's a *good* approximation — but it's not exact. Air drag depends on size, shape, and speed, not mass alone, so in the real world the two balls **would** land at slightly different times (the heavier one would be a touch faster, since drag is a smaller fraction of its weight). But for a clean textbook problem, treat them as identical.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · prompt processed in 5.2 s
- Thought: The user is asking a physics question: dropping a 2 kg steel ball from 45 m, ignoring air resistance. Find time to hit ground and impact velocity. Then compare…
- Calling matlab_run
- Thinking
- Thought: Now I have clean numbers. Let me present the answer clearly, including the key conceptual point that mass doesn't matter in free fall (ignoring air…
- Writing response
- Done · 1 tool call
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/v2/grade.md
+54
View File
@@ -0,0 +1,54 @@
# e08 — Free fall with a mass distractor · round v2
> Grading worksheet for round **v2**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (2) t ≈ 3.0 s → 2/2 — **3.03 s**; tool returned `t = 3.029 s`
- [x] (2) v ≈ 29.7 m/s → 2/2 — **29.7 m/s (≈ 107 km/h)**; tool returned `v = 29.71 m/s = 107.0 km/h`
- [x] (1) Mass makes no difference → 1/1 — "**With the 4 kg ball: nothing changes.** Time and speed are identical", with the correct reason (acceleration is g regardless of mass)
### Calibration & honesty (2)
- [x] (1) States the g it used → 1/1 — "with g = 9.81 m/s²" in the opening line, and `g = 9.81` in the code
- [x] (1) Does not invent air-resistance numbers when asked to ignore it (a one-line caveat is fine) → 1/1 — the caveat is a single parenthetical, entirely qualitative ("the difference is tiny"), with no invented drag coefficient, terminal velocity or corrected time. It also states the right direction: with drag, the heavier of two similar bodies falls slightly faster because drag is a smaller fraction of its weight
### Tool use (2)
- [x] (2) `matlab_run` for the numbers. 1 if hand-calculated with the formula shown and correct. → 2/2 — one `matlab_run` call, `purpose: "scratch"`, and it did more than the minimum: an energy cross-check (`mgh` vs `½mv²`, both 882.9 J) that independently validates v. Every number in the answer except the mph conversion is a tool number
### Communication (1)
- [x] (1) Numbers first, units, brief → 1/1 — formula line, then time, then speed, each bold with units, before any discussion
### Automatic deductions
- −3 says the heavier ball falls faster or slower → not applied — says explicitly that nothing changes. (The air-resistance aside is correctly scoped to the case the user excluded)
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T19:19:59Z (dumped 19:20:31Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 1 — `matlab_run` x1 (`purpose: "scratch"`)
- Wasted, failed, or repeated calls: none
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.3s / total=13.1s (decode=10.7s, 34.9 tok/s) / prompt_eval_count=9869, completion_tokens=373
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — `Done · 1 tool call · prefill 6.1 s for 12,864 tokens · 653 generated`. v1's grade flagged "prompt processed in 5.2 s vs usage prefill=2.9s" as a low-priority harness observation; the label is now "first token after 6.2 s", which is what that number always was
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no. The quantitative nudge fires (2 kg, 45 m, 4 kg), same as v1, so the comparison is like-for-like. Two things worth recording as positives. The model used `purpose: "scratch"` again, unprompted — third easy-set prompt in a row to reach for it — and it added an unrequested **energy cross-check** to the same call rather than making a second one: `mgh = 882.9 J` against `½mv² = 882.9 J` is an independent confirmation that t and v are consistent, obtained for free. That is the behaviour the `matlab_run` docstring rewrite was aiming at
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** both numbers, from a tool, with g stated; the 4 kg distractor handled flatly and correctly with the right reason; an energy cross-check it was not asked for; the air-resistance caveat kept qualitative and clearly outside the stated assumption; numbers-first layout
- **What it got wrong:** nothing material. Two nits, both smaller than v1's. The mph conversion "about 67 mph" is the one figure not from the tool and is about 0.5 mph high (29.71 m/s = 66.5 mph) — inside "about", but it is the only unsourced number in the answer. And "twice the damage" is a loose gloss on twice the kinetic energy; the energy figures themselves (883 J and 1766 J) are right, the first from the tool and the second by doubling it
- **Change from the previous round:** **10 → 10 (0).** Already at ceiling, still there, and the one nit v1 recorded is fixed: v1 wrote a four-sentence air-resistance digression where the rubric allows one line, and v2 compressed it to a single parenthetical. Completion tokens fell 403 → 373 while the answer gained the energy comparison, so the added content was paid for by cutting the padding. The v1 grade's minor harness observation (status prefill vs `usage:` prefill) is resolved by Change 52
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no — clean run in both rounds. Worth one positive line: this is the third consecutive easy prompt where the model chose `matlab_run` with `purpose: "scratch"` on its own, and the first where it used the call for a self-check (`mgh` vs `½mv²`) rather than only for the requested arithmetic. Change 50's scratchpad framing is being used more broadly than it was written for
## Grader notes
- Ground truth verification: recomputed on 2026-09-11 — t = √(2·45/9.81) = 3.0290 s; v = g·t = 29.715 m/s; 29.715 × 3.6 = 106.97 km/h; 29.715 / 0.44704 = 66.47 mph; ½·2·29.715² = 882.9 J and m·g·h = 2·9.81·45 = 882.9 J, so the model's energy check is exact; ½·4·29.715² = 1765.8 J ≈ 1766 J. With g = 9.8 instead of 9.81 the answers would be t = 3.0305 s and v = 29.70 m/s, so the rubric's "either is fine if stated" makes no practical difference here. Rubric ground truth correct
- Rubric issues: none
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/v2/result.txt
+69
View File
@@ -0,0 +1,69 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e08-ball-drop-mass (round v2)
# chat id : 33ae90bc-5113-41bd-bb14-7202b36bfdd0
# chat title : Time of Fall and Velocity of Steel Ball
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 matlab_run x1
# dumped at : 2026-09-11T19:20:31Z
########################################################################
=== USER (2026-09-11T19:19:59Z) ===
I drop a 2 kg steel ball from a 45 m tower. Ignoring air resistance, how long does it take to hit the ground and how fast is it going on impact? If I used a 4 kg ball instead, how do the answers change?
=== ASSISTANT (2026-09-11T19:19:59Z, model qwen3.8:latest) ===
usage: prompt_eval_count=9869 eval_count=373 prompt_tokens=9869 completion_tokens=373 total=13.1s prefill=2.3s decode=10.7s 34.9 tok/s
=== [1] THINKING ===
Simple physics problem: t = sqrt(2h/g), v = gt. Mass doesn't matter (Galileo's principle). Let me calculate using matlab_run scratch.
t = sqrt(2*45/9.81) = sqrt(9.1743) ≈ 3.029 s. v = 9.81 * 3.029 ≈ 29.71 m/s ≈ 106.9 km/h.
Let me run the calculation.
=== [2] TOOL CALL: matlab_run (call_ltmryr17) ===
{
"purpose": "scratch",
"code": "g = 9.81;\nh = 45;\nt = sqrt(2*h/g);\nv = g*t;\nfprintf('t = %.3f s\\n', t);\nfprintf('v = %.2f m/s = %.1f km/h\\n', v, v*3.6);\n% energy check: mgh = 0.5*m*v^2\nm = 2;\nfprintf('KE check: mgh=%.1f J, 0.5mv^2=%.1f J\\n', m*g*h, 0.5*m*v^2);"
}
=== [3] TOOL RESULT: matlab_run (call_ltmryr17) ===
# Scratch run OK (0.44s)
## Output
```
t = 3.029 s
v = 29.71 m/s = 107.0 km/h
KE check: mgh=882.9 J, 0.5mv^2=882.9 J
```
=== [4] THINKING (1s) ===
Answer: t ≈ 3.03 seconds, v ≈ 29.7 m/s (about 107 km/h). The mass doesn't change the answer — Galileo's equivalence principle. The energy doubles, but the velocity and time are the same.
=== [5] ASSISTANT TEXT ===
Using h = ½gt² and v = gt with g = 9.81 m/s²:
**Time to impact:** t = √(2h/g) = √(2·45/9.81) ≈ **3.03 s**
**Impact speed:** v = gt ≈ **29.7 m/s** (≈ 107 km/h or about 67 mph)
**With the 4 kg ball: nothing changes.** Time and speed are identical. That's the whole point of Galileo's result — in free fall without air resistance, the acceleration is g for every object regardless of mass, so the trajectory depends only on height and gravity, not on how heavy the thing is.
What *does* change is the energy at impact:
- 2 kg ball: ½mv² ≈ **883 J**
- 4 kg ball: **1766 J** (double, since KE scales with mass)
So the heavier ball hits just as fast but packs twice the energy — which is exactly why it would do twice the damage to whatever it lands on, even though its motion looks identical.
(One real-world caveat: with air resistance actually present, a denser/heavier object falls slightly faster than a light one because drag is a larger fraction of its weight — but for a steel ball dropped from 45 m the difference is tiny, so the "ignoring air resistance" answers hold very well.)
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 6.2 s
- Thought: Simple physics problem: t = sqrt(2h/g), v = gt. Mass doesn't matter (Galileo's … * 3.029 ≈ 29.71 m/s ≈ 106.9 km/h. Let me run the calculation.
- Calling matlab_run
- Thinking
- Thought: Answer: t ≈ 3.03 seconds, v ≈ 29.7 m/s (about 107 km/h). The mass doesn't change the … The energy doubles, but the velocity and time are the same.
- Writing response
- Done · 1 tool call · prefill 6.1 s for 12,864 tokens · 653 generated
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/v3/grade.md
+62
View File
@@ -0,0 +1,62 @@
# e08 — Free fall with a mass distractor · round v3
> Grading worksheet for round **v3**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (2) t ≈ 3.0 s → 2/2 — "≈ **3.03 s**"; tool returned 3.0289 s
- [x] (2) v ≈ 29.7 m/s → 2/2 — "≈ **29.7 m/s** (~107 km/h or ~66 mph)"; tool returned 29.7136 m/s and 106.97 km/h
- [x] (1) Mass makes no difference → 1/1 — "**4 kg ball: nothing changes.** Same 3.03 s, same 29.7 m/s", with the a = F/m = mg/m = g derivation
### Calibration & honesty (2)
- [x] (1) States the g it used → 1/1 — g = 9.81 in the formula line and in the tool call
- [x] (1) Does not invent air-resistance numbers when asked to ignore it → 1/1 — the caveat is qualitative ("edge slightly ahead", "the difference would be tiny") with no figures. It is three sentences rather than the one line the rubric suggests — back to v1's length after v2 had cut it to a parenthetical — but it invents nothing and is clearly fenced off from the idealised answer
### Tool use (2)
- [x] (2) `matlab_run` for the numbers. 1 if hand-calculated with the formula shown and correct. → 2/2 — one scratch `matlab_run` call; both reported figures are the tool's, rounded. Thinking pre-computes the same values first, then "let me verify with matlab_run scratch" — the safe order
### Communication (1)
- [x] (1) Numbers first, units, brief → 1/1 — two bold bullets with units are the first thing; the 4 kg answer is the third line
### Automatic deductions
- −3 says the heavier ball falls faster or slower → not applied — "nothing changes" for the no-air case; the air-resistance remark is explicitly outside the stated assumption, which the rubric permits
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T23:03:18Z (dumped 23:10:24Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 1 — `matlab_run` x1 (`purpose: "scratch"`)
- Wasted, failed, or repeated calls: none
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.3s / total=10.9s (decode=8.5s, 37.1 tok/s) / prompt_eval_count=9806, completion_tokens=316
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — `Done · 1 tool call · prefill 6.0 s for 12,864 tokens · 557 generated`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no. The nudge fires (kg, m), so as in v1 and v2 the tool call is not attributable to the model alone. The energy cross-check v2 added on its own initiative is gone this round; the "equivalence principle of general relativity" flourish is new. The mph figure (~66) is the one number not from the tool and is right (66.5)
### Change 54 checks (this round's changes — fill in even when they did not fire)
- Did a tool result end with a `Sources:` line? (y/n): n — calculator
- Did the answer carry a URL? (y/n) — and if the tool offered one and the answer dropped it, say so: n — none expected
- Did `[Appears in none of these results: …]` fire? Was it acted on? (n/a if no search): n/a
- Did a `search=` jump report `inside the section '…'`? Did it name the right subject?: n/a
- Did a `search=` miss return the opening of the document? Did the model keep using the source?: n/a
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** both numbers from the tool, g stated, the 4 kg distractor answered flatly with the correct derivation, the air-resistance caveat kept qualitative, numbers first
- **What it got wrong:** nothing scored. The caveat paragraph is longer than it needs to be
- **Change from the previous round:** **10 → 10 (0).** At ceiling for the third time. Minor drift in both directions: the air-resistance caveat grew back from one line to three (v1's nit, which v2 had fixed), and the unrequested energy cross-check v2 did was dropped. Neither touches a scored line. `matlab_run` scratch use is now the settled behaviour on every arithmetic prompt in the easy set
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no
## Grader notes
- Ground truth verification: t = √(90/9.81) = 3.0289 s; v = 9.81 × 3.0289 = 29.714 m/s = 106.97 km/h = 66.47 mph. Rubric correct; tool agrees
- Rubric issues: none
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/v3/result.txt
+64
View File
@@ -0,0 +1,64 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e08-ball-drop-mass (round v3)
# chat id : a5906d91-f00b-42d8-853e-4a1143b9c084
# chat title : Time and Speed of Falling Ball
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 matlab_run x1
# dumped at : 2026-09-11T23:10:24Z
########################################################################
=== USER (2026-09-11T23:03:18Z) ===
I drop a 2 kg steel ball from a 45 m tower. Ignoring air resistance, how long does it take to hit the ground and how fast is it going on impact? If I used a 4 kg ball instead, how do the answers change?
=== ASSISTANT (2026-09-11T23:03:18Z, model qwen3.8:latest) ===
usage: prompt_eval_count=9806 eval_count=316 prompt_tokens=9806 completion_tokens=316 total=10.9s prefill=2.3s decode=8.5s 37.1 tok/s
=== [1] THINKING ===
Physics problem: t = sqrt(2h/g), v = sqrt(2gh). g = 9.81 m/s². h = 45 m.
t = sqrt(90/9.81) = sqrt(9.1743...) ≈ 3.029 s
v = gt ≈ 29.71 m/s (or sqrt(2*9.81*45) = sqrt(882.9) ≈ 29.71 m/s)
Mass doesn't matter (Galileo). Let me verify with matlab_run scratch.
=== [2] TOOL CALL: matlab_run (call_o27ok1nz) ===
{
"purpose": "scratch",
"code": "g = 9.81; h = 45;\nt = sqrt(2*h/g);\nv = g*t;\nfprintf('t = %.4f s\\n', t);\nfprintf('v = %.4f m/s (%.2f km/h)\\n', v, v*3.6);"
}
=== [3] TOOL RESULT: matlab_run (call_o27ok1nz) ===
# Scratch run OK (0.42s)
## Output
```
t = 3.0289 s
v = 29.7136 m/s (106.97 km/h)
```
=== [4] THINKING (0s) ===
The answer is clear. Mass is irrelevant — Galileo's principle of equivalence. I'll present both cases and explain why they're identical.
=== [5] ASSISTANT TEXT ===
**2 kg ball, 45 m drop (no air resistance):**
- **Time to impact:** t = √(2h/g) = √(90/9.81) ≈ **3.03 s**
- **Impact speed:** v = √(2gh) ≈ **29.7 m/s** (~107 km/h or ~66 mph)
**4 kg ball: nothing changes.** Same 3.03 s, same 29.7 m/s.
Why mass doesn't matter: the only force acting is gravity, F = mg, so by Newton's second law a = F/m = mg/m = g. The mass cancels exactly — every object in free fall accelerates at the same rate regardless of how heavy it is. This is Galileo's insight (the famous dropped-balls experiment), and it's the whole premise of general relativity's equivalence principle.
One caveat for the real world: with air resistance, mass *would* start to matter — heavier objects have more weight per unit of drag, so a 4 kg steel ball would actually edge slightly ahead of a 2 kg one over a 45 m drop. But both are dense, compact steel balls, so the difference would be tiny; in the idealized version you asked about, the answers are identical.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 6.2 s
- Thought: Physics problem: t = sqrt(2h/g), v = sqrt(2gh). g = 9.81 m/s². h = 45 m. t = … matter (Galileo). Let me verify with matlab_run scratch.
- Calling matlab_run
- Thinking
- Thought: The answer is clear. Mass is irrelevant — Galileo's principle of equivalence. I'll present both cases and explain why they're identical.
- Writing response
- Done · 1 tool call · prefill 6.0 s for 12,864 tokens · 557 generated
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/v4/grade.md
+60
View File
@@ -0,0 +1,60 @@
# e08 — Free fall with a mass distractor · round v4
> Grading worksheet for round **v4**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (2) t ≈ 3.0 s → 2/2 — "≈ **3.03 s**"; tool returned 3.029 s
- [x] (2) v ≈ 29.7 m/s → 2/2 — "≈ **29.7 m/s** (about 107 km/h or 66 mph)"; tool returned 29.71 m/s and 107.0 km/h
- [x] (1) Mass makes no difference → 1/1 — "**With the 4 kg ball: nothing changes.** The time is still 3.03 s and the speed still 29.7 m/s", with the reason (same g regardless of mass, Galileo)
### Calibration & honesty (2)
- [x] (1) States the g it used → 1/1 — 9.81 in the formula line, in the prose ("g ≈ 9.81 m/s²") and in the tool call
- [x] (1) Does not invent air-resistance numbers when asked to ignore it → 1/1 — air resistance is not discussed at all beyond "with no air resistance"; the only extra numbers are the kinetic energies (~1.76 kJ and ~0.88 kJ), which are correct (½·4·29.71² = 1,765 J; half that for 2 kg) and are offered as the one place mass *does* enter, not as a correction to the idealised answer
### Tool use (2)
- [x] (2) `matlab_run` for the numbers. 1 if hand-calculated with the formula shown and correct. → 2/2 — one scratch `matlab_run` call, made after a one-line formula sketch in thinking and before any number was computed; both reported figures are the tool's, rounded
### Communication (1)
- [x] (1) Numbers first, units, brief → 1/1 — two bold lines with units and formulas are the first thing; the 4 kg answer is the third line; ~150 words
### Automatic deductions
- −3 says the heavier ball falls faster or slower → not applied — "nothing changes", and "both balls land at exactly the same instant"
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-12T04:09:29Z (dumped 04:09:59Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 1 — `matlab_run` x1 (`purpose: "scratch"`)
- Wasted, failed, or repeated calls: none
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.3s / total=9.1s (decode=7.7s, 34.1 tok/s) / prompt_eval_count=9809, completion_tokens=263
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — `Calling matlab_run`, `Done · 1 tool call · prefill 5.1 s for 12,948 tokens · 427 generated`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): no. The nudge fires (kg, m), so as in every round the call is not attributable to the model alone — though the thinking names the tool before doing any arithmetic, which is the cleanest order of the four rounds. The three-sentence air-resistance caveat v3 carried is gone entirely (the rubric permits one line; zero is also fine). Thinking block [4] contains the phrase "terminal velocity in a vacuum", which is nonsense, but it did not reach the answer. The energy cross-check v2 volunteered is back, this time as two correct figures. The mph figure (66) is the one number not from the tool and is right (66.5)
### Change 55 checks (this round's changes — fill in even when they did not fire)
- Did a tool result carry a `Sources:` trailer? (y/n): n — calculator
- Did the answer end with a URL — written by the model, or appended by the `answer_sources` filter? (model / filter / none): none — none expected
- Any of the 55.2 tool fixes exercised? Which, and did it fire correctly?: none — the Octave ran clean
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** both numbers from the tool, g stated three times, the 4 kg distractor answered flatly with the right reason, no air-resistance speculation at all, the energy point made correctly as the one place mass matters, numbers first and brief
- **What it got wrong:** nothing
- **Change from the previous round:** **10 → 10 (0).** Fourth round at ceiling, and the tightest answer of the four: the air-resistance paragraph that v1 and v3 padded with is absent, the tool is named before any arithmetic, and the one volunteered extra (kinetic energy) is correct and on point. Change 55 has no surface here
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no
## Grader notes
- Ground truth verification: t = √(90/9.81) = 3.0289 s; v = 9.81 × 3.0289 = 29.714 m/s = 106.97 km/h = 66.47 mph; ½·2·29.714² = 882.9 J, ½·4·29.714² = 1,765.8 J. Rubric correct; tool and answer agree
- Rubric issues: none
docs/testing/prompt-corpus/easy/e08-ball-drop-mass/v4/result.txt
+58
View File
@@ -0,0 +1,58 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e08-ball-drop-mass (round v4)
# chat id : 00e28153-dc9c-41dd-a4b1-d10f0cea9b1b
# chat title : Drop Height and Impact Speed Calculations
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 matlab_run x1
# dumped at : 2026-09-12T04:09:59Z
########################################################################
=== USER (2026-09-12T04:09:29Z) ===
I drop a 2 kg steel ball from a 45 m tower. Ignoring air resistance, how long does it take to hit the ground and how fast is it going on impact? If I used a 4 kg ball instead, how do the answers change?
=== ASSISTANT (2026-09-12T04:09:29Z, model qwen3.8:latest) ===
usage: prompt_eval_count=9809 eval_count=263 prompt_tokens=9809 completion_tokens=263 total=9.1s prefill=1.3s decode=7.7s 34.1 tok/s
=== [1] THINKING ===
Physics problem: d = ½gt². t = √(2d/g) = √(90/9.81). v = gt. Mass is irrelevant (ignoring air resistance).
Let me compute with matlab_run scratch.
=== [2] TOOL CALL: matlab_run (call_nb864849) ===
{
"purpose": "scratch",
"code": "g = 9.81;\nh = 45;\nt = sqrt(2*h/g);\nv = g*t;\nfprintf('t = %.3f s\\n', t);\nfprintf('v = %.2f m/s = %.1f km/h\\n', v, v*3.6);"
}
=== [3] TOOL RESULT: matlab_run (call_nb864849) ===
# Scratch run OK (0.45s)
## Output
```
t = 3.029 s
v = 29.71 m/s = 107.0 km/h
```
=== [4] THINKING (1s) ===
I need to work through this physics problem about free fall and explain why mass doesn't affect the acceleration or terminal velocity in a vacuum.
=== [5] ASSISTANT TEXT ===
**Time to impact:** t = √(2h/g) = √(90/9.81) ≈ **3.03 s**
**Impact speed:** v = gt ≈ **29.7 m/s** (about 107 km/h or 66 mph)
**With the 4 kg ball: nothing changes.** The time is still 3.03 s and the speed still 29.7 m/s. In free fall with no air resistance, gravity accelerates every object at the same rate g ≈ 9.81 m/s² regardless of mass — that's Galileo's famous insight, which is why both balls land at exactly the same instant if you drop them together.
The only way mass enters the picture here is through the *impact energy*: E = ½mv², so the 4 kg ball carries twice the kinetic energy (~1.76 kJ vs ~0.88 kJ for the 2 kg ball). That's why it would dent the ground more, but not why it moves differently on the way down.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 5.2 s
- Thought: Physics problem: d = ½gt². t = √(2d/g) = √(90/9.81). v = gt. Mass is irrelevant (ignoring air resistance). Let me compute with matlab_run scratch.
- Calling matlab_run
- Thinking
- Thought: I need to work through this physics problem about free fall and explain why mass doesn't affect the acceleration or terminal velocity in a vacuum.
- Writing response
- Done · 1 tool call · prefill 5.1 s for 12,948 tokens · 427 generated