AGENTS.md: What Works, What Costs, and Best Practices

These files are an instruction layer that supplies the project-specific tooling and conventions raw source code often leaves out. Over 60,000 repositories have adopted the format, but adding one is not a guaranteed win — whether it helps comes down entirely to how it was written.

The data from ETH Zurich shows that human-written context files provide a 4% performance boost in task resolution across benchmarks. However, LLM-generated files often lower success rates by 0.5% to 2%. The failure mode is almost always redundancy: auto-generated files tend to repeat information already in the repository, which adds noise and confuses the agent's reasoning.

Treat these files as a deliberate engineering tradeoff. They improve adherence to specific tools, but they also impose a 20% "tax" on every execution step in higher inference costs. Unless you are prepared to manually curate these files to fill the "information gap" in niche repositories, they are more likely to bloat your CI budget than solve your bugs.

What is AGENTS.md and what problem does it solve?

Coding agents face a "context gap" when working on niche or internal repositories. Standard benchmarks like SWE-bench use popular, well-documented projects, but the ETH Zurich AGENTbench focuses on "less-popular" repositories where the code is often the only source of truth. In these cases, agents struggle with non-obvious tooling decisions, specialized CI setups, and project-specific conventions that aren't obvious from a raw file tree.

The core problem is that agents often hallucinate library usage or fail to run the correct test runner because the developer's intent isn't written down anywhere a machine can read it. AGENTS.md is a "README for agents," cutting tool errors and environment setup failures. It tells the agent exactly which commands are valid, preventing it from guessing whether to use pip, poetry, or uv.

These files only help when the information is additive. If a repository already has thorough documentation, an AGENTS.md often becomes a burden by repeating data the agent can already see. Its job is to bridge the gap for agents working where documentation is thin or non-standard.

How coding agents read and apply AGENTS.md

Technically, the file is injected directly into the agent's context window at the start of a session (as CLAUDE.md for Claude Code or AGENTS.md for Codex/Qwen). The agent treats these instructions as high-priority system rules. This works well for tool selection: mentioning the package manager uv in a context file increases its usage from near-zero to 1.6 times per task instance.

Agents map their internal tools—Read, Write, Grep, and Edit—to the instructions you provide. Give it an exact pytest command string, and the agent adapts its "Run Test" intent to match your project's environment.

The danger here is "Process over Outcome." Agents are instruction-following systems; if you give a detailed codebase overview, the agent will put following that "map" ahead of finding the bug. Data shows that giving overviews does not help the agent reach the target file faster; it just pushes the agent to go through more of the repository to follow the instructions, often at the cost of the actual fix.