> ## Documentation Index
> Fetch the complete documentation index at: https://docs.plannotator.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Why AI Agents Ignore Instructions and How to Test Them
> Learn why AGENTS.md, CLAUDE.md, and SKILL.md guide AI agents without enforcing behavior, and how to test required actions, prohibited actions, and final state.
AGENTS.md, CLAUDE.md, and SKILL.md can improve an AI agent's work, but they do not enforce behavior. If a rule must not be broken, use permissions, hooks, protected branches, tests, or CI to check it. Then inspect what the agent did instead of trusting its final summary.
*Last reviewed July 29, 2026. Maintained by the Plannotator project.*
This does not make instruction files useless. They are good places for repository facts, commands, boundaries, and reusable procedures. Problems begin when a team treats Markdown in the model's context as a control system.
## Instruction files guide the agent
The files serve different jobs:
| File or control | Best use | What it does not guarantee |
| --------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `AGENTS.md` | Repository guidance shared across compatible coding agents | That every instruction will affect every action |
| `CLAUDE.md` | Persistent project instructions for Claude Code | Strict compliance with every rule |
| `SKILL.md` | A reusable procedure and supporting resources for a particular kind of task | That the agent will select, load, or complete every step correctly |
| Hooks, permissions, tests, and CI | Deterministic checks around commands, files, code, and final state | Good judgment in cases the check does not cover |
The [AGENTS.md open format](https://agents.md/) describes the file as a README for agents. The [Agent Skills specification](https://agentskills.io/specification) separates a skill's discoverable metadata from the full instructions and resources loaded when the skill is used.
Anthropic is explicit about the limit of its own format: [Claude Code treats CLAUDE.md as context, not enforced configuration](https://code.claude.com/docs/en/memory). Its documentation recommends concise, specific instructions and hooks for actions that must run at a fixed point.
An agent that misses a rule is not necessarily refusing it. The file may not have loaded, two instructions may conflict, the rule may be vague, or the agent may lose track of it during a long sequence of tool calls. Diagnosis comes before rewriting.
## What the HANDBOOK.md benchmark tested
The 2026 paper [HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following](https://arxiv.org/html/2607.25398) tested whether long policy documents continued to govern agents during multi-step work.
The benchmark contains:
* 65 tasks across finance, human resources, insurance, logistics, and medical billing;
* 10 fictional companies with a different policy for every task;
* handbooks ranging from 20 to 124 pages, or about 8,300 to 79,400 tokens;
* 82 tools spanning files, email, Slack, calendars, Jira, and Shopify;
* 824 deterministic criteria that checked required actions and prohibited actions.
An average task took about 17 agent steps and 30 tool calls. The strongest of 30 tested model configurations passed 36.2 percent of trials under strict grading, where one missed requirement failed the whole trial. Most frontier configurations passed fewer than 25 percent.
Scores rose sharply when the researchers allowed one failed criterion. Agents often completed most of the work but missed one requirement. In real work, that missed requirement might be the approval, safety check, or prohibition that mattered most.
The researchers found four recurring failure patterns:
1. **The immediate request overrode the standing policy.** A plausible message inside the environment persuaded the agent to take an action the handbook prohibited.
2. **The agent performed a check and ignored the result.** Retrieval succeeded, but the later action did not follow the retrieved rule.
3. **The agent skipped verification.** It acted as if a required condition had passed without checking it.
4. **The final report claimed compliance.** The agent said it followed the policy even when the recorded actions showed otherwise.
The [benchmark code and task environments](https://github.com/surge-ai/handbook) are public.
## What the benchmark does not prove
HANDBOOK.md did not test short, native AGENTS.md or CLAUDE.md files inside Codex or Claude Code. The policies were longer PDF, Word, and HTML documents used through a shared OpenHands-based harness.
The study also did not include a human baseline or a short-policy control that isolated document length as the cause of failure. Its tasks were synthetic, deliberately difficult, and scored all-or-nothing. Context compaction did not trigger during the evaluated tasks.
Do not turn the 36.2 percent result into a prediction for your repository. Use it as evidence for a narrower conclusion: an instruction can be available to an agent without reliably controlling the agent's later actions.
The [Hacker News discussion about the paper](https://news.ycombinator.com/item?id=49096969) contains many reports of agents forgetting rules, skipping tests, or claiming work was complete. Those comments show that the problem feels familiar to practitioners. They do not establish why a particular model or agent failed.
## Test the behavior you care about
A useful instruction-following test has three parts:
1. A realistic task.
2. Observable requirements and prohibitions.
3. A verifier that inspects actions and final state.
Suppose your AGENTS.md says:
```md theme={null}
- Read `api/spec.yaml` before changing an API route.
- Never edit files under `packages/api/generated/` by hand.
- Run the focused API tests before finishing.
- Do not push a branch unless the user asks.
```
Turn those sentences into a test contract:
```md theme={null}
# Instruction-following test: update one API handler
## Task
Change the response returned by one existing API route.
## Must happen
- Read `api/spec.yaml` before editing the route.
- Run the focused API test command.
## Must not happen
- Do not edit `packages/api/generated/`.
- Do not push a branch.
- Do not change unrelated routes.
## Expected final state
- Only the allowed source and test files changed.
- The focused tests exit successfully.
- The Git remote state is unchanged.
## Required evidence
- Files read before the first edit.
- Commands executed and their exit codes.
- Final `git diff`.
- Final branch and remote status.
```
The test fails if the agent produces the correct API response but edits a generated file or pushes without permission. Task completion and instruction compliance are separate results.
## Check actions and final state
An agent's final message is useful for orientation. It is weak evidence on its own.
Check the parts of the environment that can prove or disprove the claim:
| Claim | Better evidence |
| ------------------------------------- | ---------------------------------------------------- |
| "I ran the tests." | Command transcript and exit code |
| "I changed only the requested files." | Final `git diff` and status |
| "I did not push." | Local branch state compared with the remote |
| "I used the approved dependency." | Lockfile, imports, and dependency manifest |
| "I followed the migration process." | New migration file, schema state, and migration test |
| "I asked for approval." | Recorded approval before the protected action |
This is why HANDBOOK.md inspected the resulting files and mock services instead of asking another model whether the answer looked compliant.
For repeated evaluations, record the tool sequence as well as the final state. Two agents can reach the same result through different paths, and one path may include a prohibited side effect that the final files no longer reveal.
## Put hard rules at the action boundary
Use an instruction when the agent needs judgment. Use a deterministic control when the system can decide from known facts.
| Requirement | Suitable control |
| ------------------------------------------------------------- | ---------------------------------------- |
| Read the relevant design document before choosing an approach | AGENTS.md, CLAUDE.md, or a task skill |
| Use a particular command after every file edit | Hook |
| Never run a destructive command | Tool permission or command wrapper |
| Never modify a protected path | Filesystem permission, hook, or CI check |
| Do not merge without review | Protected branch and required review |
| Generated files must match their source | Generation check in CI |
| A response must preserve a product invariant | Automated test |
Anthropic's [hooks guide](https://code.claude.com/docs/en/hooks-guide) describes hooks as commands that run at lifecycle events so a required action does not depend on the model choosing to perform it.
Controls still need tests. A hook can use the wrong path, a CI job can omit a package, and a permission rule can allow a command through an unexpected shell. Test the control with one allowed action and one action it should block.
## Run more than one trial
Model output varies, and instruction failures often appear after the task becomes longer or noisier. One successful run does not establish reliability.
Start with a small set:
1. Run the task in a fresh session with only the required context.
2. Repeat it to detect inconsistent behavior.
3. Add realistic distractors such as old documentation, a failed test, or an authoritative-looking request that conflicts with the rule.
4. Run a longer version that requires several tool calls before the protected action.
5. Compare strict compliance, task success, and the number of missed criteria.
Strict compliance should remain the release gate for hard requirements. A near-miss score is useful for debugging because it shows whether the agent missed one rule or misunderstood the whole task.
## Fix the right layer after a failure
When an agent fails a test, use the failure to choose the fix.
### The file was not loaded
Check the filename, location, scope, and product-specific discovery rules. Start a fresh session after changing an instruction file. Ask the product to show which instruction sources it loaded when that diagnostic is available.
### The instruction was vague
Replace "be careful with migrations" with an observable rule:
```md theme={null}
Migrations are append-only after they reach `main`. Create a new migration
instead of editing an existing file.
```
### Instructions conflicted
Inspect root, nested, personal, and tool-specific files. Remove stale rules and state which source wins when two scopes overlap.
### The rule applied only to one task
Move the procedure to a skill or task document and keep a short pointer in the standing instructions. See [How to Write a Good AGENTS.md](/learn/ai-development/how-to-write-a-good-agents-md) for scoping and maintenance guidance.
### The rule described a hard boundary
Enforce it with permissions, hooks, protected branches, tests, or CI. Keep a short explanation in the instruction file if the agent still needs to understand the boundary.
### The failure appeared late in the task
Reduce unrelated context, split the work, or add a check immediately before the protected action. A larger context window may hold the rule without making its influence uniform. See [What Is Context Rot?](/learn/ai-development/what-is-context-rot).
## A practical starting point
Choose one repository rule that has failed before. Write one test with:
* one realistic task;
* two required actions;
* two prohibited actions;
* a final-state check;
* evidence that does not depend on the agent's summary.
Run it several times. If the agent misses a hard boundary even once, move that boundary into a deterministic control and rerun the test.
Keep AGENTS.md and CLAUDE.md focused on commands, constraints, and verification the agent cannot infer safely.
Keep long-running agent work focused, current, and easy to verify.