Long-Context Stress Protocol

Capacity Is Not the Result

A long-context experiment should separate four things: whether the request fits, whether the answer retrieves the right evidence, whether reasoning over that evidence is correct, and whether the cost or latency is acceptable. A model can fit a large request and still miss a buried clause. It can retrieve the clause and still reason from it incorrectly.

Capacity now differs across the lineup in ways that decide which model can even hold your fixture. Anthropic documents 1M-token context windows on Claude Fable 5.1, Claude Opus 5, and Claude Sonnet 5, with 200K on Claude Haiku 4.5, and maximum output of 128K tokens on the three larger models against 64K on Haiku 4.5. Output budget matters as much as input budget in extraction tasks, where a truncated answer looks like a retrieval failure.

This page stays inside experiment methodology. For broader context-engineering patterns, compaction strategies, and prompt-as-context design, route readers to Claude Context.

Fixture Design

Build a document pack with known answers, near misses, and irrelevant distractors. Put some answer-bearing passages early, some in the middle, and some late. Include one answer that requires combining two passages. Include one impossible question whose correct answer is that the source pack does not contain the requested fact.

Use the token counting endpoint or equivalent platform tooling before running the test. Record measured tokens for the exact payload, not an estimate copied from a previous model generation.

• Needle task: direct retrieval from one passage.
• Bridge task: combine two passages that are far apart.
• Refusal-to-infer task: answer "not in the materials" when evidence is absent.
• Distractor task: distinguish a similar but wrong passage from the answer passage.

Controls

Run the same questions against a short-context control pack that contains only the answer-bearing passages. If the model fails in the short pack, the long-context condition is not the cause. If it passes the short pack and fails the long pack, inspect distractor placement, instruction wording, and output budget.

Do not compare a Haiku 4.5 run at 200K against a Fable 5.1 or Sonnet 5 run at 1M without noting that model, output ceiling, and pricing differences change both behavior and economics. Cache economics moved too: Anthropic prices cache reads on Fable 5.1 well below the rest of the lineup, which can make a repeated-pack experiment cheaper on the larger model than intuition suggests. Cost comparisons themselves belong at Claude Reports; record your measured tokens here and let the analysis desk price them.

Metrics

Score three columns separately: evidence location found, final answer correct, and unsupported claims avoided. A concise answer that cites the wrong location fails evidence grounding even if the final sentence happens to be true. A verbose answer with correct citations but unsupported extra claims should be marked partial, not pass.

For cost-sensitive applications, add measured input tokens, output tokens, cache usage if applicable, and wall-clock time. Long context is a capability, not a free variable.

Freshness Rule

When Anthropic changes context limits, tokenizer behavior, or model availability, create a new dated entry in the ledger. Do not overwrite older results. Old results are still useful when they preserve model ID, date, fixture hash, and scoring rules. Knowledge cutoffs belong in the same record: a pack that tests whether the answer came from the documents rather than from memory behaves differently on Haiku 4.5, cut off in February 2025, than on Fable 5.1, cut off in June 2026.