Prompt Sensitivity Protocol

Question to Test

The prompt-sensitivity question is narrow: if every task input stays the same, does changing the instruction wording, order, or markup change the answer in a way your evaluator would care about? This is not a general prompt-engineering tutorial; broad advice belongs on Claude Helps. Curious Claude keeps the scope to experiment design.

Begin with one production-like task and one measurable output standard. Examples: extraction must return every named party from a contract; summarization must include all material risks; classification must use exactly one allowed label. The standard is written before any prompt variants are tried.

Variables and Controls

Hold the model, temperature or sampling settings, tool availability, source document, and output schema constant. Change one prompt property at a time: heading names, XML tags, example count, instruction order, or the presence of a brief rubric. If two properties move together, the result can show that a bundle helped, but it cannot say which property mattered.

For each variant, record the exact prompt text, model identifier, date, account surface, and any platform feature used. Anthropic documents current model families and platform surfaces in the models overview; experiments should cite that page or the relevant release note when model behavior is part of the claim.

• Control prompt: the existing production or baseline instruction.
• Variant A: a minimal wording change.
• Variant B: a structural change, such as XML sections or a numbered rubric.
• Blind scoring: hide variant names from the scorer when human review is involved.

Runbook

Create at least ten fixtures before looking at outputs. The fixtures should represent normal cases, edge cases, and one intentionally boring case. A boring case matters because a prompt that only shines on adversarial examples may add cost or verbosity to routine work.

Run each fixture against every prompt variant in the same session style. If the surface has conversation memory or hidden state, reset it between runs. Store raw outputs exactly as returned. A clean result table has rows for fixture, prompt variant, model, timestamp, pass/fail, rubric notes, and reviewer initials or automated score version.

Scoring

A variant wins only if it improves the predeclared metric without failing a guardrail. Guardrails can include output length, citation presence, JSON validity, refusal behavior, or latency budget. Record ties honestly; a non-result is valuable because it prevents unnecessary prompt churn.

If a model upgrade changes the result, do not backfill a conclusion. Create a new dated row in the experiment ledger. Model releases and system cards move over time, and a prompt result from one model generation should not be silently treated as a property of all Claude models.

Common Failure Modes

The most common false positive is judging a single eloquent answer as proof. The second is using the same examples in the prompt and the test fixture, which rewards memorization of the example pattern. The third is changing the model, prompt, and scoring standard together, then calling the whole bundle a prompt improvement.

When the question turns into broader prompt-writing advice, link out. When it turns into benchmark reporting, link to Claude Reports. This page owns the repeatable method, not the league table.