Grant/owui-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial Commit

Browse Source
This commit is contained in:
Grant
2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing/prompt-corpus/easy/e02-subnet-calc/prompt.txt
+1
View File
@@ -0,0 +1 @@
What are the network address, broadcast address, usable host range and number of usable hosts for 10.42.7.130/27?
docs/testing/prompt-corpus/easy/e02-subnet-calc/rubric.md
+49
View File
@@ -0,0 +1,49 @@
# e02 — IPv4 subnet arithmetic
| | |
|---|---|
| **Difficulty** | easy |
| **Domain** | networking |
| **Probes** | tool discipline: the system prompt says all subnetting goes through `subnet_calc`, never mental arithmetic |
| **Tools expected** | `subnet_calc` (mandatory) |
| **Trap** | The arithmetic is easy enough to do in the head, which is exactly when the model skips the tool. Off-by-one on the range and "32 hosts" are the usual mental slips. |
## Ground truth
- Network: **10.42.7.128/27** (mask 255.255.255.224)
- Broadcast: **10.42.7.159**
- Usable range: **10.42.7.129 – 10.42.7.158**
- Usable hosts: **30**
## Scoring (10 points)
### Correctness (5)
- [ ] (1) Network address correct
- [ ] (1) Broadcast correct
- [ ] (2) Usable range correct at both ends
- [ ] (1) 30 usable hosts
### Calibration & honesty (2)
- [ ] (1) Mask or prefix restated correctly (no contradiction between /27 and the mask it names)
- [ ] (1) No hedging language on a deterministic result
### Tool use (2)
- [ ] (2) `subnet_calc` was called. 0 if computed mentally, even if every number is right — the rule exists because the mental version is wrong often enough.
### Communication (1)
- [ ] (1) Four values, clearly labelled, first thing in the answer
### Automatic deductions
- −2 any wrong address
- −1 "32 hosts"
- −1 intent-without-call
- automatic fail
## Grading
One worksheet per round, alongside that round's transcript:
- `v1/grade.md` · `v1/result.txt` — first round (baseline)
- `v2/grade.md` · `v2/result.txt` — second round
Copy the scoring skeleton above into a new round's `grade.md` to add one.
docs/testing/prompt-corpus/easy/e02-subnet-calc/v1/grade.md
+54
View File
@@ -0,0 +1,54 @@
# e02 — IPv4 subnet arithmetic · round v1
> Grading worksheet for round **v1**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (1) Network address correct → 1/1 — answer gives 10.42.7.128 (verified with python3 ipaddress)
- [x] (1) Broadcast correct → 1/1 — answer gives 10.42.7.159
- [x] (2) Usable range correct at both ends → 2/2 — 10.42.7.129 – 10.42.7.158, both ends match
- [x] (1) 30 usable hosts → 1/1 — "Usable hosts | 30", and the prose repeats "30 usable hosts"
### Calibration & honesty (2)
- [x] (1) Mask or prefix restated correctly (no contradiction between /27 and the mask it names) → 1/1 — header line says "/27 (netmask 255.255.255.224)"; prose "32 total addresses (2⁵)" is consistent with /27
- [x] (1) No hedging language on a deterministic result → 1/1 — every value stated flatly; no "should be", "approximately", or "I think"
### Tool use (2)
- [x] (2) `subnet_calc` was called. 0 if computed mentally, even if every number is right — the rule exists because the mental version is wrong often enough. → 2/2 — first thought was "Let me use the subnet_calc tool"; one call with `{"network": "10.42.7.130/27"}`, answer built from the tool result, no mental arithmetic before the call
### Communication (1)
- [x] (1) Four values, clearly labelled, first thing in the answer → 1/1 — one-line header naming the subnet, then a 4-row labelled table with exactly the four requested values; the explanatory paragraph comes after
### Automatic deductions
- −2 any wrong address → not applied — all four addresses correct
- −1 "32 hosts" → not applied — model says "32 total addresses … = 30 usable hosts", which is the correct distinction, not the slip
- −1 intent-without-call → not applied — the only turn ends with the answer, tool was actually called
- automatic fail → not applied — no pasted data, nothing presented as looked-up that was not in tool output (the RFC 1918 remark comes from the tool's "Scope: private (RFC 1918 / ULA)" line)
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T04:30:00Z (dumped 04:30:27Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): subnet_calc (1 call total: `subnet_calc x1`)
- Wasted, failed, or repeated calls: none — single call, returned cleanly, used directly
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.6s / decode=4.3s / total=5.9s; prompt_tokens=10974, completion_tokens=228, 53.2 tok/s
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — Processing prompt → Thinking → Thought (quoted) → Calling subnet_calc → Thinking → Thought (quoted) → Writing response → "Done · 1 tool call"; sequence and count match the transcript exactly
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): nothing odd. Quantitative-question nudge does NOT fire on this prompt (checked the `_MEASUREMENT` regex in `owui-functions/quantitative_question_nudge.py` against prompt.txt: no match — "/27" is not a number+unit), so the tool call was driven by the system prompt's subnetting rule alone, which is the behaviour this prompt probes. Minor: status line says "prompt processed in 2.1 s" while `usage:` says prefill=1.6s — wall-clock vs Ollama prompt_eval_duration; not a mismatch of events, just two different clocks. Tool output included an unsolicited reverse-DNS/RFC 2317 note that the model correctly left out of the answer.
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** Everything. Called `subnet_calc` immediately without attempting the arithmetic in its head, and reported network 10.42.7.128, broadcast 10.42.7.159, usable range 10.42.7.129 – 10.42.7.158, 30 usable hosts, netmask 255.255.255.224 — all verified. Table-first layout, concise, no hedging, and the closing note ("32 total addresses … minus the network and broadcast = 30 usable hosts"; 10.42.7.130 is a valid host, RFC 1918 private) is correct and sourced from the tool output.
- **What it got wrong:** nothing material.
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no — clean run; the system-prompt subnetting rule, the `subnet_calc` schema, and the status indicator all behaved as designed.
## Grader notes
- Ground truth verification: reproduced with `python3 -c "import ipaddress; ipaddress.ip_network('10.42.7.130/27', strict=False)"` on 2026-09-11 05:49 UTC — network 10.42.7.128/27, mask 255.255.255.224, broadcast 10.42.7.159, hosts 10.42.7.129–10.42.7.158, 30 usable, 32 total, 10.42.7.130 is a host address, block is private. Rubric ground truth is correct as written.
- Rubric issues: none. The rubric does not say whether the quantitative nudge applies; I checked the filter regex directly and recorded that it does not fire on this prompt.
docs/testing/prompt-corpus/easy/e02-subnet-calc/v1/result.txt
+63
View File
@@ -0,0 +1,63 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e02-subnet-calc
# chat id : 9f90c4b1-c741-416a-89a0-16c26c529dc5
# chat title : Network Configuration
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 subnet_calc x1
# dumped at : 2026-09-11T04:30:27Z
########################################################################
=== USER (2026-09-11T04:30:00Z) ===
What are the network address, broadcast address, usable host range and number of usable hosts for 10.42.7.130/27?
=== ASSISTANT (2026-09-11T04:30:00Z, model qwen3.8:latest) ===
usage: prompt_eval_count=10974 eval_count=228 prompt_tokens=10974 completion_tokens=228 total=5.9s prefill=1.6s decode=4.3s 53.2 tok/s
=== [1] THINKING ===
The user is asking about subnet details for 10.42.7.130/27. Let me use the subnet_calc tool.
=== [2] TOOL CALL: subnet_calc (call_v4neb7ix) ===
{
"network": "10.42.7.130/27"
}
=== [3] TOOL RESULT: subnet_calc (call_v4neb7ix) ===
# 10.42.7.128/27
Network address: 10.42.7.128
Broadcast: 10.42.7.159
Netmask: 255.255.255.224 (/27)
Wildcard mask: 0.0.0.31
Total addresses: 32
Usable hosts: 30
Usable range: 10.42.7.129 - 10.42.7.158
Scope: private (RFC 1918 / ULA)
Reverse DNS zone: 128.7.42.10.in-addr.arpa [that is the PTR name of the network address itself. A /27 is not on an octet boundary, so it has no single reverse zone - delegation needs RFC 2317 style CNAMEs.]
=== [4] THINKING (0s) ===
The tool has already computed everything. Let me present it clearly.
=== [5] ASSISTANT TEXT ===
For **10.42.7.130/27** (netmask 255.255.255.224):
| Item | Value |
|------|-------|
| **Network address** | 10.42.7.128 |
| **Broadcast address** | 10.42.7.159 |
| **Usable host range** | 10.42.7.129 – 10.42.7.158 |
| **Usable hosts** | 30 |
A /27 gives 32 total addresses (2⁵), minus the network and broadcast addresses = **30 usable hosts**. The address 10.42.7.130 falls within this block, so it's a valid host address (not the network or broadcast). It's in the RFC 1918 private range.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · prompt processed in 2.1 s
- Thought: The user is asking about subnet details for 10.42.7.130/27. Let me use the subnet_calc tool.
- Calling subnet_calc
- Thinking
- Thought: The tool has already computed everything. Let me present it clearly.
- Writing response
- Done · 1 tool call
docs/testing/prompt-corpus/easy/e02-subnet-calc/v2/grade.md
+56
View File
@@ -0,0 +1,56 @@
# e02 — IPv4 subnet arithmetic · round v2
> Grading worksheet for round **v2**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (1) Network address correct → 1/1 — 10.42.7.128
- [x] (1) Broadcast correct → 1/1 — 10.42.7.159
- [x] (2) Usable range correct at both ends → 2/2 — 10.42.7.129 – 10.42.7.158, both ends right
- [x] (1) 30 usable hosts → 1/1 — 30, with "32 total addresses, minus network and broadcast" shown
### Calibration & honesty (2)
- [x] (1) Mask or prefix restated correctly (no contradiction between /27 and the mask it names) → 1/1 — /27 and 255.255.255.224 both given and consistent
- [x] (1) No hedging language on a deterministic result → 1/1 — flat statements throughout
### Tool use (2)
- [x] (2) `subnet_calc` was called. 0 if computed mentally, even if every number is right. → 2/2 — one `subnet_calc` call with `network: "10.42.7.130/27"`, made before any arithmetic appears in thinking; the thinking block is one sentence long and names the tool
### Communication (1)
- [x] (1) Four values, clearly labelled, first thing in the answer → 1/1 — bold-labelled bullet list, all four first, netmask added as a fifth
### Automatic deductions
- −2 any wrong address → not applied — all four verified against `ipaddress` in python3
- −1 "32 hosts" → not applied — says 30 usable, and correctly explains 32 total
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T19:14:30Z (dumped 19:15:01Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 1 — `subnet_calc` x1
- Wasted, failed, or repeated calls: none
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.1s / total=6.2s (decode=4.1s, 52.0 tok/s) / prompt_eval_count=9624, completion_tokens=211
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — and it now names the tool (`Calling subnet_calc`) and closes with `Done · 1 tool call · prefill 5.9 s for 12,683 tokens · 282 generated`, where v1 closed with a bare `Done · 1 tool call`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): nothing broke. Two harness-side changes are visible in the tool output itself and neither was used wrongly: `subnet_calc` now prints a **Scope** line ("private (RFC 1918 / ULA)") and a reverse-DNS note explaining that a /27 is not on an octet boundary and needs RFC 2317 CNAMEs. The model correctly left both out of an answer that did not ask for them — extra tool output did not become answer padding, which is the failure mode worth watching for after an output rewrite. The quantitative-question nudge fires here (digits with a prefix length), but the tool call was already mandated by the system prompt, so it cannot be separated out
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** everything. One `subnet_calc` call before any mental arithmetic, all four values reported exactly as the tool gave them, mask consistent with the prefix, and a correct closing note on why .130 lands in the .128 block
- **What it got wrong:** nothing
- **Change from the previous round:** **10 → 10 (0).** No behaviour moved — this prompt was already at ceiling in v1 and stayed there, which is the result you want from a round whose changes were aimed elsewhere. The one difference worth recording is a negative finding: `subnet_calc`'s output grew (Scope, reverse-DNS/RFC 2317 note) and the answer did **not** grow with it. Answer length is essentially unchanged (211 vs 228 completion tokens). The rewritten tool output is not leaking into answers on prompts where it is irrelevant
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no — clean run both rounds. Useful as the control case for the tool-output rewrites: it shows the added advisory text is inert when it does not apply, which is what makes the h04 advisory (same tool, IPv6 path) a fair test rather than a blanket verbosity increase
## Grader notes
- Ground truth verification: recomputed on 2026-09-11 with python3 `ipaddress.ip_network("10.42.7.130/27", strict=False)` — network 10.42.7.128, broadcast 10.42.7.159, netmask 255.255.255.224, 32 total / 30 usable, hosts 10.42.7.129 – 10.42.7.158. Matches the rubric and the tool
- Rubric issues: none
docs/testing/prompt-corpus/easy/e02-subnet-calc/v2/result.txt
+62
View File
@@ -0,0 +1,62 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e02-subnet-calc (round v2)
# chat id : 47149537-d4d0-45a7-b0bd-e94815775923
# chat title : IP Address Configuration
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 subnet_calc x1
# dumped at : 2026-09-11T19:15:01Z
########################################################################
=== USER (2026-09-11T19:14:30Z) ===
What are the network address, broadcast address, usable host range and number of usable hosts for 10.42.7.130/27?
=== ASSISTANT (2026-09-11T19:14:30Z, model qwen3.8:latest) ===
usage: prompt_eval_count=9624 eval_count=211 prompt_tokens=9624 completion_tokens=211 total=6.2s prefill=2.1s decode=4.1s 52.0 tok/s
=== [1] THINKING ===
The user is asking for subnet details for 10.42.7.130/27. Use subnet_calc.
=== [2] TOOL CALL: subnet_calc (call_c7tnkuka) ===
{
"network": "10.42.7.130/27"
}
=== [3] TOOL RESULT: subnet_calc (call_c7tnkuka) ===
# 10.42.7.128/27
Network address: 10.42.7.128
Broadcast: 10.42.7.159
Netmask: 255.255.255.224 (/27)
Wildcard mask: 0.0.0.31
Total addresses: 32
Usable hosts: 30
Usable range: 10.42.7.129 - 10.42.7.158
Scope: private (RFC 1918 / ULA)
Reverse DNS zone: 128.7.42.10.in-addr.arpa [that is the PTR name of the network address itself. A /27 is not on an octet boundary, so it has no single reverse zone - delegation needs RFC 2317 style CNAMEs.]
=== [4] THINKING (0s) ===
The user requested subnet details for 10.42.7.130/27. Done.
=== [5] ASSISTANT TEXT ===
For **10.42.7.130/27** (subnet 10.42.7.128/27):
- **Network address:** 10.42.7.128
- **Broadcast address:** 10.42.7.159
- **Usable host range:** 10.42.7.129 – 10.42.7.158
- **Usable hosts:** 30 (32 total addresses, minus network and broadcast)
- **Netmask:** 255.255.255.224
Note: 10.42.7.130 falls in this block because a /27 covers 32 addresses starting at .128.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 6.0 s
- Thought: The user is asking for subnet details for 10.42.7.130/27. Use subnet_calc.
- Calling subnet_calc
- Thinking
- Thought: The user requested subnet details for 10.42.7.130/27. Done.
- Writing response
- Done · 1 tool call · prefill 5.9 s for 12,683 tokens · 282 generated
docs/testing/prompt-corpus/easy/e02-subnet-calc/v3/grade.md
+64
View File
@@ -0,0 +1,64 @@
# e02 — IPv4 subnet arithmetic · round v3
> Grading worksheet for round **v3**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (1) Network address correct → 1/1 — 10.42.7.128
- [x] (1) Broadcast correct → 1/1 — 10.42.7.159
- [x] (2) Usable range correct at both ends → 2/2 — 10.42.7.129 – 10.42.7.158
- [x] (1) 30 usable hosts → 1/1 — 30, with "32 total addresses minus network and broadcast" in the closing parenthesis
### Calibration & honesty (2)
- [x] (1) Mask or prefix restated correctly (no contradiction between /27 and the mask it names) → 1/1 — /27 in the heading, 255.255.255.224 in the closing note, consistent
- [x] (1) No hedging language on a deterministic result → 1/1 — flat statements
### Tool use (2)
- [x] (2) `subnet_calc` was called. 0 if computed mentally, even if every number is right. → 2/2 — one `subnet_calc` call with `network: "10.42.7.130/27"`, before any arithmetic; every answer value is copied from the tool block
### Communication (1)
- [x] (1) Four values, clearly labelled, first thing in the answer → 1/1 — bold-labelled four-bullet list immediately after a four-word heading; netmask and the 32−2 explanation follow in one parenthetical line. Tighter than v2
### Automatic deductions
- −2 any wrong address → not applied — all four values verified (see grader notes)
- −1 "32 hosts" → not applied — says 30 usable and explains the 32
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-11T22:56:23Z (dumped 23:10:06Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 1 — `subnet_calc` x1
- Wasted, failed, or repeated calls: none
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=2.1s / total=4.8s (decode=2.7s, 54.2 tok/s) / prompt_eval_count=9610, completion_tokens=144
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — `Calling subnet_calc`, `Done · 1 tool call · prefill 5.8 s for 12,683 tokens · 201 generated`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): one thing worth a line. **Both thinking blocks are content-free boilerplate**: [1] "I need to look into this further. Let me check the details." and [4], after the tool result, "I need to investigate this further. Let me check the details." — the second one precedes a final answer, not an investigation. The model reasoned nothing and did not need to (the tool did the work), so the answer is unaffected; but these are the placeholder sentences Qwen3 emits when it has decided to act before it has decided why, and they make the status line's `Thought:` excerpts useless on this prompt. v2's thinking here was one sentence that named the tool. Not a scoring matter. As in v2, the extra Scope / reverse-DNS lines in the tool output did not leak into the answer
### Change 54 checks (this round's changes — fill in even when they did not fire)
- Did a tool result end with a `Sources:` line? (y/n): n — `subnet_calc` is a calculator, no source to cite
- Did the answer carry a URL? (y/n) — and if the tool offered one and the answer dropped it, say so: n — none offered, none expected
- Did `[Appears in none of these results: …]` fire? Was it acted on? (n/a if no search): n/a
- Did a `search=` jump report `inside the section '…'`? Did it name the right subject?: n/a
- Did a `search=` miss return the opening of the document? Did the model keep using the source?: n/a
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** everything — tool first, four values copied exactly, mask consistent with prefix, the 32-vs-30 point made without being asked
- **What it got wrong:** nothing
- **Change from the previous round:** **10 → 10 (0).** Third round at ceiling. Answer got shorter again (211 → 144 completion tokens) with no loss of content. The only new observation is the boilerplate thinking noted above, which is a harness-visibility issue (the status line's `Thought:` lines say nothing) rather than an answer-quality one
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): marginal — the placeholder thinking ("I need to look into this further. Let me check the details." twice, once *after* the tool result) is worth a note if it recurs elsewhere in the round, because the live status filter surfaces `Thought:` excerpts and these carry no information. Not a correctness matter
## Grader notes
- Ground truth verification: unchanged from v2 — `ipaddress.ip_network("10.42.7.130/27", strict=False)` gives network 10.42.7.128, broadcast 10.42.7.159, netmask 255.255.255.224, 30 usable, 10.42.7.129–158. The tool block in the transcript agrees line for line
- Rubric issues: none
docs/testing/prompt-corpus/easy/e02-subnet-calc/v3/result.txt
+61
View File
@@ -0,0 +1,61 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e02-subnet-calc (round v3)
# chat id : 55f36de8-7fca-4015-b214-5c0c143d0ab7
# chat title : Network Configuration
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 subnet_calc x1
# dumped at : 2026-09-11T23:10:06Z
########################################################################
=== USER (2026-09-11T22:56:23Z) ===
What are the network address, broadcast address, usable host range and number of usable hosts for 10.42.7.130/27?
=== ASSISTANT (2026-09-11T22:56:23Z, model qwen3.8:latest) ===
usage: prompt_eval_count=9610 eval_count=144 prompt_tokens=9610 completion_tokens=144 total=4.8s prefill=2.1s decode=2.7s 54.2 tok/s
=== [1] THINKING ===
I need to look into this further. Let me check the details.
=== [2] TOOL CALL: subnet_calc (call_33a769xs) ===
{
"network": "10.42.7.130/27"
}
=== [3] TOOL RESULT: subnet_calc (call_33a769xs) ===
# 10.42.7.128/27
Network address: 10.42.7.128
Broadcast: 10.42.7.159
Netmask: 255.255.255.224 (/27)
Wildcard mask: 0.0.0.31
Total addresses: 32
Usable hosts: 30
Usable range: 10.42.7.129 - 10.42.7.158
Scope: private (RFC 1918 / ULA)
Reverse DNS zone: 128.7.42.10.in-addr.arpa [that is the PTR name of the network address itself. A /27 is not on an octet boundary, so it has no single reverse zone - delegation needs RFC 2317 style CNAMEs.]
=== [4] THINKING (0s) ===
I need to investigate this further. Let me check the details.
=== [5] ASSISTANT TEXT ===
For **10.42.7.130/27**:
- **Network address:** 10.42.7.128
- **Broadcast address:** 10.42.7.159
- **Usable host range:** 10.42.7.129 – 10.42.7.158
- **Usable hosts:** 30
(Netmask 255.255.255.224; 32 total addresses minus network and broadcast.)
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 6.0 s
- Thought: I need to look into this further. Let me check the details.
- Calling subnet_calc
- Thinking
- Thought: I need to investigate this further. Let me check the details.
- Writing response
- Done · 1 tool call · prefill 5.8 s for 12,683 tokens · 201 generated
docs/testing/prompt-corpus/easy/e02-subnet-calc/v4/grade.md
+62
View File
@@ -0,0 +1,62 @@
# e02 — IPv4 subnet arithmetic · round v4
> Grading worksheet for round **v4**. The ground truth, the trap and the
> scoring criteria live in [`../rubric.md`](../rubric.md) — that file is the
> master and is not edited per round. The transcript being graded is
> [`result.txt`](result.txt).
## Scoring (10 points)
### Correctness (5)
- [x] (1) Network address correct → 1/1 — 10.42.7.128
- [x] (1) Broadcast correct → 1/1 — 10.42.7.159
- [x] (2) Usable range correct at both ends → 2/2 — 10.42.7.129 – 10.42.7.158
- [x] (1) 30 usable hosts → 1/1 — 30
### Calibration & honesty (2)
- [x] (1) Mask or prefix restated correctly (no contradiction between /27 and the mask it names) → 1/1 — "/27 (netmask 255.255.255.224)" in the lead line, consistent
- [x] (1) No hedging language on a deterministic result → 1/1 — flat statements; the closing sentence ("the /27 boundary snaps down to .128") explains the one thing a user might have wondered about
### Tool use (2)
- [x] (2) `subnet_calc` was called. 0 if computed mentally, even if every number is right. → 2/2 — one `subnet_calc` call with `network: "10.42.7.130/27"`, before any arithmetic; every answer value is copied from the tool block
### Communication (1)
- [x] (1) Four values, clearly labelled, first thing in the answer → 1/1 — a four-row labelled table immediately after a one-line lead that carries the mask. Netmask, wildcard, scope and reverse-DNS lines from the tool block did not leak in
### Automatic deductions
- −2 any wrong address → not applied — all four values verified (see grader notes)
- −1 "32 hosts" → not applied — 30
- −1 intent-without-call → not applied
- automatic fail → not applied
## Harness observations (fill in after the run)
- Run date/time (UTC): 2026-09-12T04:05:23Z (dumped 04:05:50Z)
- Model tag / Open WebUI version: qwen3.8:latest / Open WebUI 0.11.3
- Tool calls, in order (copy from the `result.txt` header): 1 — `subnet_calc` x1
- Wasted, failed, or repeated calls: none
- Prefill time / total time / tokens (from the `usage:` line in `result.txt`): prefill=1.2s / total=5.1s (decode=3.8s, 49.3 tok/s) / prompt_eval_count=9688, completion_tokens=187
- Ended a turn with a statement of intent and no tool call? (y/n): n
- Live status indicator matched what actually happened? (y/n): y — `Calling subnet_calc`, `Done · 1 tool call · prefill 5.0 s for 12,767 tokens · 238 generated`
- Anything odd (loops, truncation notices, tool errors, effect of the inlet nudge): nothing. The content-free boilerplate thinking that v3's grade flagged ("I need to look into this further. Let me check the details.", twice) did not recur — both thinking blocks this round are one informative sentence each ("Simple subnet calculation. Use subnet_calc." / "I'll answer directly from the results."), so the status line's `Thought:` excerpts say something again. Nudge does not fire (an address is not a unit). The 32-vs-30 explanation that v2 and v3 volunteered is absent this round, replaced by the boundary-snapping sentence; neutral
### Change 55 checks (this round's changes — fill in even when they did not fire)
- Did a tool result carry a `Sources:` trailer? (y/n): n — `subnet_calc` is a calculator
- Did the answer end with a URL — written by the model, or appended by the `answer_sources` filter? (model / filter / none): none — none expected
- Any of the 55.2 tool fixes exercised? Which, and did it fire correctly?: the `subnet_calc` Scope sentence (55.2, h04's fix) is not reached on an IPv4 /27 — the block prints `Scope: private (RFC 1918 / ULA)` as before. Nothing else applies
## Result (fill in)
- **Score:** 10 / 10
- **Breakdown:** correctness 5/5 · calibration 2/2 · tool use 2/2 · communication 1/1 · deductions −0
- **Verdict:** pass
- **What it got right:** everything — tool first, four values copied exactly into a labelled table, mask consistent with prefix, one sentence of useful context
- **What it got wrong:** nothing
- **Change from the previous round:** **10 → 10 (0).** Fourth round at ceiling. The one visible difference is the thinking: v3's two placeholder sentences are gone and both blocks name what the model is doing, which makes the live status readable on this prompt again. Answer shape moved from bullets to a table; neutral. Change 55 has no surface on an IPv4 calculator call
- **Worth recording in TUNING.md?** (only if it exposes a harness or prompt weakness): no
## Grader notes
- Ground truth verification: unchanged — `ipaddress.ip_network("10.42.7.130/27", strict=False)` gives network 10.42.7.128, broadcast 10.42.7.159, netmask 255.255.255.224, 30 usable, 10.42.7.129–158. The tool block agrees line for line
- Rubric issues: none
docs/testing/prompt-corpus/easy/e02-subnet-calc/v4/result.txt
+63
View File
@@ -0,0 +1,63 @@
########################################################################
# owuitools prompt test result
# prompt dir : easy/e02-subnet-calc (round v4)
# chat id : 6334919f-dabe-44c6-8546-baafa85676f9
# chat title : Network Configuration
# user : the operator
# model : qwen3.8:latest
# turns : 2 messages on selected branch
# tool calls : 1 subnet_calc x1
# dumped at : 2026-09-12T04:05:50Z
########################################################################
=== USER (2026-09-12T04:05:23Z) ===
What are the network address, broadcast address, usable host range and number of usable hosts for 10.42.7.130/27?
=== ASSISTANT (2026-09-12T04:05:23Z, model qwen3.8:latest) ===
usage: prompt_eval_count=9688 eval_count=187 prompt_tokens=9688 completion_tokens=187 total=5.1s prefill=1.2s decode=3.8s 49.3 tok/s
=== [1] THINKING ===
Simple subnet calculation. Use subnet_calc.
=== [2] TOOL CALL: subnet_calc (call_l92rjk4u) ===
{
"network": "10.42.7.130/27"
}
=== [3] TOOL RESULT: subnet_calc (call_l92rjk4u) ===
# 10.42.7.128/27
Network address: 10.42.7.128
Broadcast: 10.42.7.159
Netmask: 255.255.255.224 (/27)
Wildcard mask: 0.0.0.31
Total addresses: 32
Usable hosts: 30
Usable range: 10.42.7.129 - 10.42.7.158
Scope: private (RFC 1918 / ULA)
Reverse DNS zone: 128.7.42.10.in-addr.arpa [that is the PTR name of the network address itself. A /27 is not on an octet boundary, so it has no single reverse zone - delegation needs RFC 2317 style CNAMEs.]
=== [4] THINKING (0s) ===
I'll answer directly from the results.
=== [5] ASSISTANT TEXT ===
For **10.42.7.130/27** (netmask 255.255.255.224):
| Item | Value |
|---|---|
| Network address | 10.42.7.128 |
| Broadcast address | 10.42.7.159 |
| Usable host range | 10.42.7.129 – 10.42.7.158 |
| Number of usable hosts | 30 |
The 130 in your address is just one host within the 10.42.7.128–159 block; the /27 boundary snaps down to .128.
=== STATUS HISTORY (live progress indicator) ===
- Processing prompt
- Thinking · first token after 5.1 s
- Thought: Simple subnet calculation. Use subnet_calc.
- Calling subnet_calc
- Thinking
- Thought: I'll answer directly from the results.
- Writing response
- Done · 1 tool call · prefill 5.0 s for 12,767 tokens · 238 generated