Long-Context Stress Protocol

Capacity Is Not the Result

A long-context experiment should separate four things: whether the request fits, whether the answer retrieves the right evidence, whether reasoning over that evidence is correct, and whether the cost or latency is acceptable. A model can fit a large request and still miss a buried clause. It can retrieve the clause and still reason from it incorrectly.

This page stays inside experiment methodology. For broader context-engineering patterns, compaction strategies, and prompt-as-context design, route readers to Claude Context.

Fixture Design

Build a document pack with known answers, near misses, and irrelevant distractors. Put some answer-bearing passages early, some in the middle, and some late. Include one answer that requires combining two passages. Include one impossible question whose correct answer is that the source pack does not contain the requested fact.

Use the token counting endpoint or equivalent platform tooling before running the test. Record measured tokens for the exact payload, not an estimate copied from a previous model generation.

• Needle task: direct retrieval from one passage.
• Bridge task: combine two passages that are far apart.
• Refusal-to-infer task: answer "not in the materials" when evidence is absent.
• Distractor task: distinguish a similar but wrong passage from the answer passage.

Controls

Run the same questions against a short-context control pack that contains only the answer-bearing passages. If the model fails in the short pack, the long-context condition is not the cause. If it passes the short pack and fails the long pack, inspect distractor placement, instruction wording, and output budget.

Do not compare a 200k-token model run to a 1M-token model run without noting that model and tokenizer differences can change both behavior and economics. Anthropic release notes may also change token accounting guidance between model generations.

Metrics

Score three columns separately: evidence location found, final answer correct, and unsupported claims avoided. A concise answer that cites the wrong location fails evidence grounding even if the final sentence happens to be true. A verbose answer with correct citations but unsupported extra claims should be marked partial, not pass.

For cost-sensitive applications, add measured input tokens, output tokens, cache usage if applicable, and wall-clock time. Long context is a capability, not a free variable.

Freshness Rule

When Anthropic changes context limits, tokenizer behavior, or model availability, create a new dated entry in the ledger. Do not overwrite older results. Old results are still useful when they preserve model ID, date, fixture hash, and scoring rules.