Instruction files guide the agent
The files serve different jobs:
The AGENTS.md open format describes the file as a README for agents. The Agent Skills specification separates a skill’s discoverable metadata from the full instructions and resources loaded when the skill is used.
Anthropic is explicit about the limit of its own format: Claude Code treats CLAUDE.md as context, not enforced configuration. Its documentation recommends concise, specific instructions and hooks for actions that must run at a fixed point.
An agent that misses a rule is not necessarily refusing it. The file may not have loaded, two instructions may conflict, the rule may be vague, or the agent may lose track of it during a long sequence of tool calls. Diagnosis comes before rewriting.
What the HANDBOOK.md benchmark tested
The 2026 paper HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following tested whether long policy documents continued to govern agents during multi-step work. The benchmark contains:- 65 tasks across finance, human resources, insurance, logistics, and medical billing;
- 10 fictional companies with a different policy for every task;
- handbooks ranging from 20 to 124 pages, or about 8,300 to 79,400 tokens;
- 82 tools spanning files, email, Slack, calendars, Jira, and Shopify;
- 824 deterministic criteria that checked required actions and prohibited actions.
- The immediate request overrode the standing policy. A plausible message inside the environment persuaded the agent to take an action the handbook prohibited.
- The agent performed a check and ignored the result. Retrieval succeeded, but the later action did not follow the retrieved rule.
- The agent skipped verification. It acted as if a required condition had passed without checking it.
- The final report claimed compliance. The agent said it followed the policy even when the recorded actions showed otherwise.
What the benchmark does not prove
HANDBOOK.md did not test short, native AGENTS.md or CLAUDE.md files inside Codex or Claude Code. The policies were longer PDF, Word, and HTML documents used through a shared OpenHands-based harness. The study also did not include a human baseline or a short-policy control that isolated document length as the cause of failure. Its tasks were synthetic, deliberately difficult, and scored all-or-nothing. Context compaction did not trigger during the evaluated tasks. Do not turn the 36.2 percent result into a prediction for your repository. Use it as evidence for a narrower conclusion: an instruction can be available to an agent without reliably controlling the agent’s later actions. The Hacker News discussion about the paper contains many reports of agents forgetting rules, skipping tests, or claiming work was complete. Those comments show that the problem feels familiar to practitioners. They do not establish why a particular model or agent failed.Test the behavior you care about
A useful instruction-following test has three parts:- A realistic task.
- Observable requirements and prohibitions.
- A verifier that inspects actions and final state.
Check actions and final state
An agent’s final message is useful for orientation. It is weak evidence on its own. Check the parts of the environment that can prove or disprove the claim:
This is why HANDBOOK.md inspected the resulting files and mock services instead of asking another model whether the answer looked compliant.
For repeated evaluations, record the tool sequence as well as the final state. Two agents can reach the same result through different paths, and one path may include a prohibited side effect that the final files no longer reveal.
Put hard rules at the action boundary
Use an instruction when the agent needs judgment. Use a deterministic control when the system can decide from known facts.
Anthropic’s hooks guide describes hooks as commands that run at lifecycle events so a required action does not depend on the model choosing to perform it.
Controls still need tests. A hook can use the wrong path, a CI job can omit a package, and a permission rule can allow a command through an unexpected shell. Test the control with one allowed action and one action it should block.
Run more than one trial
Model output varies, and instruction failures often appear after the task becomes longer or noisier. One successful run does not establish reliability. Start with a small set:- Run the task in a fresh session with only the required context.
- Repeat it to detect inconsistent behavior.
- Add realistic distractors such as old documentation, a failed test, or an authoritative-looking request that conflicts with the rule.
- Run a longer version that requires several tool calls before the protected action.
- Compare strict compliance, task success, and the number of missed criteria.
Fix the right layer after a failure
When an agent fails a test, use the failure to choose the fix.The file was not loaded
Check the filename, location, scope, and product-specific discovery rules. Start a fresh session after changing an instruction file. Ask the product to show which instruction sources it loaded when that diagnostic is available.The instruction was vague
Replace “be careful with migrations” with an observable rule:Instructions conflicted
Inspect root, nested, personal, and tool-specific files. Remove stale rules and state which source wins when two scopes overlap.The rule applied only to one task
Move the procedure to a skill or task document and keep a short pointer in the standing instructions. See How to Write a Good AGENTS.md for scoping and maintenance guidance.The rule described a hard boundary
Enforce it with permissions, hooks, protected branches, tests, or CI. Keep a short explanation in the instruction file if the agent still needs to understand the boundary.The failure appeared late in the task
Reduce unrelated context, split the work, or add a check immediately before the protected action. A larger context window may hold the rule without making its influence uniform. See What Is Context Rot?.A practical starting point
Choose one repository rule that has failed before. Write one test with:- one realistic task;
- two required actions;
- two prohibited actions;
- a final-state check;
- evidence that does not depend on the agent’s summary.
Write clearer repository instructions
Keep AGENTS.md and CLAUDE.md focused on commands, constraints, and verification the agent cannot infer safely.
Reduce context rot
Keep long-running agent work focused, current, and easy to verify.