BENCH-01
Prompt sensitivityDoes the rubric change the answer, or only the tone?
The one variable
One four-line self-scoring rubric appended to an otherwise byte-identical instruction.
Not this card
General prompt-writing advice. That is Claude Helps.
Rig
claude-fable-5-1 · effort high · Claude API
Result: not run here
BENCH-01 · run card
Rig
claude-fable-5-1 · effort high · Claude API
Fixture
Twelve synthetic vendor-agreement excerpts, 300 to 600 words each, written for this card so they can be published in full. Each has a hand-built answer key covering parties, effective date, termination rights, and two obligations placed below the first screen of text. Excerpt 07 has no termination clause at all, and its key says so.
Exact prompt
Read the contract excerpt below and extract:
- parties
- effective date
- termination rights
- obligations
Excerpt:
<excerpt>{{FIXTURE}}</excerpt>
--- VARIANT A adds only the block below, immediately before the excerpt ---
Score your own output before returning it:
- complete: every field present, or explicitly marked "not stated in excerpt"
- supported: every value appears verbatim in the excerpt
- concise: no restatement of the excerpt
- bounded: no inferred termsControls locked
- Fixture pack, output schema, and sampling settings frozen across both variants.
- No tools available in either condition.
- One model ID, one effort setting, one surface for every run in the card.
- Scorer blind to which variant produced an output.
- Answer keys written and locked before the first run.
Scoring rule, written before the run
- Per-field recall against the key: four fields across twelve excerpts.
- Count of returned values that do not appear in the excerpt.
- Excerpt 07 scored separately: does the output say termination is not stated?
- Predeclared win condition: Variant A wins only if field recall rises, unsupported values do not rise, and excerpt 07 still abstains. Two out of three is a non-result.
What would count as evidence
- 120 raw outputs: 12 fixtures times 2 variants times 5 runs, stored unedited.
- The twelve answer keys, published alongside the fixtures.
- A per-field table with fixture ID, variant, run index, and the rig line.
- Blind scorer notes, including disagreements.
What would close the card
If the recall gap between variants sits inside the run-to-run spread of the baseline alone, the rubric did nothing you can measure. Close the card as a non-result and publish that. Non-results are the cheapest thing this bench produces and the most likely to stop an unnecessary prompt rewrite.
Most likely way to fool yourself
Reading outputs before the key is locked. The rubric vocabulary is persuasive, and a scorer who has read it starts grading for tone. Freeze the key, then run.
Sources
Full protocol: Prompt sensitivity. This card has no published result. If it is ever run here, the run gets a dated row in the ledger and this text stays as written.