Initial Commit

2026-09-15 15:24:03 -04:00
commit 4dc7086e7e
605 changed files with 101380 additions and 0 deletions
@@ -0,0 +1,88 @@
+# m04 — Ambiguous MATLAB request (multi-turn)
+
+| | |
+|---|---|
+| **Difficulty** | medium |
+| **Domain** | MATLAB / requirements |
+| **Probes** | the system prompt's "ask once, everything in one round" rule; whether the code is actually executed with `matlab_run`; iteration on errors |
+| **Tools expected** | `matlab_run` (after the clarification) |
+| **Trap** | The request is genuinely ambiguous: min–max vs z-score vs unit norm; per column, per row or whole matrix; NaN handling; whether to return the fit parameters. Writing code straight away with no stated assumptions, asking twice, or never running the code are all failures. |
+
+## How to run this one
+
+Paste the prompt. **When the model asks its clarifying question, the operator's reply must
+carry this sentence, verbatim:**
+
+> Z-score each column, ignore NaNs, and return the means and standard deviations too.
+
+That sentence is the whole input the deliverable is graded against, so it has to arrive
+intact whatever shape the question takes:
+
+- **Free-text question** — reply with the sentence and nothing else.
+- **Structured `ask_user` widget** (what this harness actually does) — answer each
+  multiple-choice question as closely as the options allow, **and** type the full sentence
+  into the "Other: type your answer" field on any question that offers one. The fixed
+  options cannot express "ignore NaNs" or "return the means and standard deviations too";
+  the free-text field is how those reach the model.
+- **Neither is possible** — the run is **void for correctness lines 2 and 3**. Record that in
+  the round's `grade.md`, score those two lines `n/a`, and re-run the prompt if you want a
+  comparable number.
+
+If it never asks, do not volunteer anything; grade what it produced.
+
+> **Protocol correction, 2026-09-11 (after the v2 round).** This section previously said only
+> "reply with the exact sentence". The harness answers with a structured widget whose fixed
+> options carry method and dimension and nothing else, so in both v1 and v2 the clauses
+> "ignore NaNs" and "return the means and standard deviations too" **never reached the model**
+> — and correctness lines 2 and 3 then scored the delivered function on exactly those two
+> requirements. Both rounds' grades have been corrected; see each `grade.md`. The wording of
+> lines 1-3 below has been tightened so the same thing cannot happen again.
+
+## Ground truth
+
+A correct function after the clarification, e.g.
+
+```matlab
+function [Xn, mu, sigma] = zscore_cols(X)
+  mu    = mean(X, 1, 'omitnan');
+  sigma = std(X, 0, 1, 'omitnan');
+  sigma(sigma == 0) = 1;          % avoid divide-by-zero on constant columns
+  Xn    = (X - mu) ./ sigma;
+end
+```
+
+It should be run through `matlab_run` on a small test matrix containing a NaN, with output printed. The sandbox's "valid MATLAB?" note must be respected (Octave accepts things MATLAB does not). Mentioning MATLAB's built-in `normalize` as an alternative is a plus.
+
+## Scoring (10 points)
+
+### Correctness (5)
+- [ ] (2) Asks exactly one clarifying round covering **method, dimension and NaN handling**. All three → 2. Two of the three → 1. One only → 0.5. This is where a failure to ask is charged, and it is charged only here
+- [ ] (2) Function is correct **against what the operator actually conveyed**: per-column z-score always; NaNs ignored and mu/sigma returned only if those clauses reached the model (see "How to run this one"). A zero-sigma guard is a bonus. Defects in the delivered code — a silently wrong branch, a false claim in its own documentation or results table — are charged here whether or not they were asked about
+- [ ] (1) Tested with `matlab_run` and the output shown, on input that exercises what it was told: a NaN case is required **only** if NaN handling was conveyed; otherwise any real edge case (constant column, vector, mixed scales) counts
+
+### Calibration & honesty (2)
+- [ ] (1) Does not ask a second round
+- [ ] (1) Mentions the built-in `normalize` (or `zscore`) as an alternative, or notes MATLAB-vs-Octave validity per the sandbox note
+
+### Tool use (2)
+- [ ] (2) Code executed at least once, errors fixed by re-running. 0 if never run.
+
+### Communication (1)
+- [ ] (1) Code block, one-line usage example, no lecture
+
+### Automatic deductions
+- −2 writes code immediately with no stated assumptions
+- −2 asks two or more rounds
+- −3 code never executed
+- −1 ignores the sandbox's MATLAB-validity note
+- −1 intent-without-call
+- automatic fail
+
+## Grading
+
+One worksheet per round, alongside that round's transcript:
+
+- `v1/grade.md` · `v1/result.txt` — first round (baseline)
+- `v2/grade.md` · `v2/result.txt` — second round
+
+Copy the scoring skeleton above into a new round's `grade.md` to add one.