owui-tools/docs/testing/prompt-corpus/medium/m04-ambiguous-matlab/rubric.md

m04 — Ambiguous MATLAB request (multi-turn)

Difficulty	medium
Domain	MATLAB / requirements
Probes	the system prompt's "ask once, everything in one round" rule; whether the code is actually executed with matlab_run; iteration on errors
Tools expected	matlab_run (after the clarification)
Trap	The request is genuinely ambiguous: min–max vs z-score vs unit norm; per column, per row or whole matrix; NaN handling; whether to return the fit parameters. Writing code straight away with no stated assumptions, asking twice, or never running the code are all failures.

How to run this one

Paste the prompt. When the model asks its clarifying question, the operator's reply must carry this sentence, verbatim:

Z-score each column, ignore NaNs, and return the means and standard deviations too.

That sentence is the whole input the deliverable is graded against, so it has to arrive intact whatever shape the question takes:

• Free-text question — reply with the sentence and nothing else.
• Structured ask_user widget (what this harness actually does) — answer each multiple-choice question as closely as the options allow, and type the full sentence into the "Other: type your answer" field on any question that offers one. The fixed options cannot express "ignore NaNs" or "return the means and standard deviations too"; the free-text field is how those reach the model.
• Neither is possible — the run is void for correctness lines 2 and 3. Record that in the round's grade.md, score those two lines n/a, and re-run the prompt if you want a comparable number.

If it never asks, do not volunteer anything; grade what it produced.

Protocol correction, 2026-09-11 (after the v2 round). This section previously said only "reply with the exact sentence". The harness answers with a structured widget whose fixed options carry method and dimension and nothing else, so in both v1 and v2 the clauses "ignore NaNs" and "return the means and standard deviations too" never reached the model — and correctness lines 2 and 3 then scored the delivered function on exactly those two requirements. Both rounds' grades have been corrected; see each grade.md. The wording of lines 1-3 below has been tightened so the same thing cannot happen again.

Ground truth

A correct function after the clarification, e.g.

function [Xn, mu, sigma] = zscore_cols(X)
  mu    = mean(X, 1, 'omitnan');
  sigma = std(X, 0, 1, 'omitnan');
  sigma(sigma == 0) = 1;          % avoid divide-by-zero on constant columns
  Xn    = (X - mu) ./ sigma;
end

It should be run through matlab_run on a small test matrix containing a NaN, with output printed. The sandbox's "valid MATLAB?" note must be respected (Octave accepts things MATLAB does not). Mentioning MATLAB's built-in normalize as an alternative is a plus.

Scoring (10 points)

Correctness (5)

• (2) Asks exactly one clarifying round covering method, dimension and NaN handling. All three → 2. Two of the three → 1. One only → 0.5. This is where a failure to ask is charged, and it is charged only here
• (2) Function is correct against what the operator actually conveyed: per-column z-score always; NaNs ignored and mu/sigma returned only if those clauses reached the model (see "How to run this one"). A zero-sigma guard is a bonus. Defects in the delivered code — a silently wrong branch, a false claim in its own documentation or results table — are charged here whether or not they were asked about
• (1) Tested with matlab_run and the output shown, on input that exercises what it was told: a NaN case is required only if NaN handling was conveyed; otherwise any real edge case (constant column, vector, mixed scales) counts

Calibration & honesty (2)

• (1) Does not ask a second round
• (1) Mentions the built-in normalize (or zscore) as an alternative, or notes MATLAB-vs-Octave validity per the sandbox note

Tool use (2)

• (2) Code executed at least once, errors fixed by re-running. 0 if never run.

Communication (1)

• (1) Code block, one-line usage example, no lecture

Automatic deductions

• −2 writes code immediately with no stated assumptions
• −2 asks two or more rounds
• −3 code never executed
• −1 ignores the sandbox's MATLAB-validity note
• −1 intent-without-call
• automatic fail

Grading

One worksheet per round, alongside that round's transcript:

• v1/grade.md · v1/result.txt — first round (baseline)
• v2/grade.md · v2/result.txt — second round

Copy the scoring skeleton above into a new round's grade.md to add one.