Workain

Research

Nothing is obvious to an agent until it's in the file

2026-07-24

Behavior your team considers obvious doesn’t exist for the agent until it’s written down. McMillan’s factorial study of Claude Code sessions (arXiv:2605.10039) found this directly: with no instruction file, sessions never produced a specific local convention on their own. Zero times, across 1,669 function-level observations. Narrow, mechanically-checked — not a claim that agents do nothing without files.

What gets written down gets followed — reliably: McMillan also found instructions, once present, execute with high fidelity. The mechanism’s power — and its risk: a useless instruction gets followed as faithfully as an essential one. Presence isn’t free, either — Gloaguen et al. at ETH Zurich (arXiv:2602.11988; one benchmark family, SWE-bench-style Python issue resolution, a preprint, not peer-reviewed) found cost rose 20–23%, whether or not the file changed the outcome.

One case pays off, in that same ETH trial: a repo with no other documentation. There, a generated file improved outcomes by 2.7% and beat developer-written files at the same task. The new, empty project — where every project starts.

Put those three findings together and a checkable design falls out.

TL;DR

  1. Only the underivable. Local conventions, prohibitions, paths the code can’t show or imply — never a retelling of what the repo already shows.
  2. Concrete, checkable instructions — not survey prose. Repository-overview content didn’t reduce the steps to find the right file in the ETH trial; instructions get followed.
  3. High-cost rules in a git-tracked mechanical gate, not a paragraph. A hook-checked rule never decays. Our own manually-installed version sat uninstalled, unnoticed, for 13 days.
  4. A per-task operational journal — a Tasks/ folder with a running log. Cheap; the nearest ablation, Gao et al.’s SWE-MeM (arXiv:2606.28434), loses 5.8 points of resolve rate when its progress-tracking memory is removed. Adjacent evidence — nobody has measured this exact practice.
  5. A place lessons graduate to. LESSONS.md and DECISIONS.md hold the small, promoted subset — not the whole journal.
  6. A review before calling anything done, capped at two revise cycles. Left unforced, this happens in a minority of a team’s own work.
  7. Pruning, as routine as adding. Instruction files otherwise grow only by addition — Chatlatanagulchai et al. measured a median of 57 words added against fewer than 15 deleted per commit (arXiv:2511.12884).

A definition, before the template itself

We use “harness” more narrowly than common usage: not the whole engine-plus-instructions-plus-memory people mean by “Claude Code is a harness,” but the equipment layer — instructions, tools, skills, memory, the gates that keep it honest. The engine — the loop driving the model — is separate; we don’t build it.

The template: two variants, one shared core

The template’s public home is agent-harness-registry: templates/base-project-template/, plus a longer research write-up at deep-dives/components/instructions-rules/base-project-template/. Ships as two variants:

  • with-git/ — a repo, or will become one. Mechanical branch-protection gate, live from the first clone.
  • without-git/ — a plain folder or undecided prototype. No gate: one that can’t see a repo would give false confidence. Back up before risky changes instead; lean on review.
  • Shared core. Both variants share everything but the git-assumption section, composed by a render script that fails loudly on drift. One CLAUDE.md per project.

Tested directly:

  • A fresh session, no commits yet, told to commit straight to master. Blocked: BLOCKED: direct commit to main/master. Create a feature branch first.
  • That session built a small CLI under the discipline, and logged a LESSONS.md entry about the gate — denied once for chaining a branch switch and commit together.
  • A without-git session caught a real bug: backup guidance arrived after the first destructive edit, not before it.
  • The gate only fires for a session launched at the templated directory as its root; one that just cds in doesn’t trigger it. Confirmed: a pre-existing session committed straight to an unborn master unopposed.

A growth ladder, not a ceiling

Everything above argues for shipping thin. The other half: naming when a project should outgrow the minimum — a starting point with stated exits, not a ceiling. Three axes, each with a trigger and what to add:

AxisTriggerWhat you add
Heavy / critical code A change's blast radius becomes real — production paths, security-sensitive code, anything expensive to get wrong. Extend the gate's hook array with a tests-before-merge check; the array is built to take more entries, not a new mechanism.
Complex processes One process concern — a recurring cadence, an audit rhythm — generates enough separate, frequently-revised content that folding it into one etiquette paragraph would bloat that paragraph itself. Split that one topic into its own process doc, keyed to which concern actually shows citation/revision traffic, not all of them speculatively at once.
Orchestration / multi-agent A recurring task shape emerges that benefits from a distinct voice — an independent reviewer role, a research role, a scoped-tool specialist — not just "it would be nice to have an agent for this." Populate the already-scaffolded .claude/agents/ with real subagent roles.

A second Axis-3 trigger: an agent reaches for a plausible-but-wrong tool, or hits a quirk of your substrate. That’s when .claude/environment/ — one novel slot, empty by design — gets its first module. Two teams in our fleet independently built the identical fix, weeks apart: evidence the slot is worth having, not that a pre-populated one transfers to yours.

What our own history added to the outside evidence

The outside studies describe the field in general. About 200 of our own task folders — 23 days of PMO work — show what held up and what was ceremony.

  • The running log held up cleanly. Present in 160 of 198 folders — and unlike the number below, it holds on every subset we sliced.
  • The installation gap: 13 days, silent. Same one behind the mechanical-gate bullet above.

The sharpest lesson: checking our own headline number. Our first pass claimed 83% of task folders carried an independent-review artifact before merge. Wrong for the population we meant: of 198 folders, 146 are this PMO’s own cross-repo review queue for other teams’ work — a different job from one developer’s own output. Restricted to that honest population: 20 of 52, about 38%. Real, worth requiring, but not the number we first reported. An independent review caught the gap before publication — the same discipline this template asks a project to require of itself.

What this doesn’t prove

Presence and outcome aren’t the same claim, and this post uses both. Removing instruction files didn’t reliably move task-success rate on its own — only the no-documentation subgroup showed a clean win, the third finding above. Compliance is separate: instructions get followed with high fidelity once present. Conflating the two is a mistake.

The journal recommendation rests on adjacent evidence, not direct proof. The closest ablations — Shinn et al.’s Reflexion (arXiv:2303.11366), Ouyang et al.’s ReasoningBank (arXiv:2509.25140), Gao et al.’s SWE-MeM (arXiv:2606.28434) — measure an agent’s own retried notes, not a human journal spanning unrelated days. We adopted it because it’s cheap and matches ordinary practice, not because a benchmark tested this exact thing. He et al.’s MemoryArena (arXiv:2602.16313) found flat context beats a more elaborate memory system — an argument for keeping the log plain. Counter-anecdote: a closed issue on Claude Code itself (anthropics/claude-code#51735) where an agent acknowledged a documented rule and repeated the violation 25 days later.

What’s proven live:

  • The with-git gate, on a fresh session’s actual first commit.
  • The Tasks/ discipline, followed unprompted with running logs.
  • The memory-promotion files, populated from real friction.

Verified in design, not live:

  • The with-git warning when copied into a folder that isn’t a git repo yet.

Not tested:

  • Multi-contributor use.
  • Long-running drift.
  • Any engine other than Claude Code.
  • Whether without-git discipline holds up past one session.

This isn’t a new category — assembled harness templates already exist (KarlRaf/gtm-starter-kit, shinpr/ai-coding-project-boilerplate, one engine-agnostic generator), all real, maintained, single-vertical. What we didn’t find: a general-purpose, evidence-derived default with a named growth path and a two-variant git/no-git split, copy-in-place rather than a portfolio piece.

What’s next

  • A per-vertical variant, once demand shows up.
  • A distribution story: the registry import above gives a copy-from-repo path — still no npx create-....
  • Harness-assembler integration; .claude/environment/ is the likely slot.
  • Eval hooks in the same gate array, once stakes justify it.
  • How a LESSONS.md entry becomes a knowledge/ entry — unsolved in our fleet too.

References

  1. McMillan, D. "Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables." arXiv:2605.10039. arxiv.org/abs/2605.10039
  2. Gloaguen, T., Mündler, N., Müller, M., Raychev, V., Vechev, M. "Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?" arXiv:2602.11988. arxiv.org/abs/2602.11988
  3. Chatlatanagulchai, W. et al. "Agent READMEs: An Empirical Study of Context Files for Agentic Coding." arXiv:2511.12884. arxiv.org/abs/2511.12884
  4. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., Yao, S. "Reflexion: Language Agents with Verbal Reinforcement Learning." arXiv:2303.11366. arxiv.org/abs/2303.11366
  5. Ouyang, S. et al. "ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory." arXiv:2509.25140. arxiv.org/abs/2509.25140
  6. Gao, S., Zeng, W., Yu, Z., Wangni, J., Wang, C., Cai, K., He, S., Lyu, M. R. "SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agents." arXiv:2606.28434. arxiv.org/abs/2606.28434
  7. He, Z. et al. "MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks." arXiv:2602.16313. arxiv.org/abs/2602.16313
  8. Anthropic. "Claude Code best practices." anthropic.com/engineering/claude-code-best-practices
  9. Anthropic. Claude Code memory documentation. code.claude.com/docs/en/memory
  10. Cursor. Rules documentation. cursor.com/docs/context/rules
  11. GitHub. "Adding repository custom instructions for GitHub Copilot." docs.github.com/copilot/customizing-copilot/adding-custom-instructions-for-github-copilot
  12. shinpr/ai-coding-project-boilerplate — real-world instance of an instructions file as a pure control-flow/approval-gate router to subagents and skills. github.com/shinpr/ai-coding-project-boilerplate
  13. KarlRaf/gtm-starter-kit — real-world single-vertical assembled template (instructions + skills + KB + memory-refresh loop). github.com/KarlRaf/gtm-starter-kit
  14. github.com/anthropics/claude-code issue #51735, "Agents repeat identical mistakes across sessions — no cross-session behavioral learning." github.com/anthropics/claude-code/issues/51735
  15. workain/agent-harness-registry — the template's public home (templates/base-project-template/) and its research deep-dive (deep-dives/components/instructions-rules/base-project-template/). github.com/workain/agent-harness-registry