Documentation in the AI Era: Why My MEV Bot Has a Manual

The Coworker Who Forgets Everything by Morning

Every time I sit down to work on the bot, I am pairing with a coworker who has overnight amnesia. The AI agent helping me on the codebase does not remember yesterday's debugging session, last week's design decision, or the rejected approach from two months ago. As far as it is concerned, every morning is its first day on the job.

This is not a complaint. It is the working condition. And it has completely changed how I think about project documentation.

In the pre-AI days, documentation was the broccoli of software engineering. Everyone said you should eat it. Most of us shoved it to the side of the plate. The README got written once, the architecture diagram got drawn for the kickoff meeting, and the rest lived in the heads of whoever was around. If a new engineer joined, you sat down and walked them through the system over a coffee. Three months later they had absorbed enough tribal knowledge to operate without supervision. The documentation gap was annoying, but it was bounded — you only paid the onboarding cost once per new hire.

That math does not work when your collaborator resets to zero every single session. The onboarding cost is no longer paid once. It is paid every time. If your project has no written explanation of itself, you are running a one-person DMV: each session, you take a number, hand the AI your form, and wait while it asks the same questions in slightly different ways. The cost compounds.

For the MEV bot, that compounding is brutal. The codebase touches on-chain account layouts, custom transaction packing, scanning loops that need to clear in single-digit milliseconds, and DEX-specific quirks that are mostly undocumented anywhere outside our own notes. An AI that does not know any of that will burn an hour reinventing context I already paid for. So I started treating documentation the way a pilot treats its pre-flight checklist — boring, mandatory, and the difference between a smooth flight and a crash.

What the Research Actually Says

For a while I assumed the productivity gain from project context files was the kind of thing engineers like me felt in our bones but could not measure. Then the studies started showing up.

The most concrete one I have read is a peer-reviewed empirical paper that measured what happens when you hand an AI coding agent a project context file versus letting it go in cold. The researchers ran a paired within-task design across ten repositories and 124 pull requests, restricting tasks to roughly a hundred lines of code and at most five modified files. According to the arXiv paper "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents", the version with context completed tasks about 20.27% faster on wall-clock time, and the result was statistically significant at p<0.05. The same study found token usage dropped by 20.08% on output, with input tokens down nearly 10% as well. The mean task time fell from roughly 163 seconds to 130 seconds — a 33-second improvement per task that adds up fast when you are running hundreds of small tasks a week.

If those numbers feel modest, look at them through a cost-per-unit lens. Twenty percent fewer tokens means a fifth off your cloud bill on AI work. Twenty percent faster means the same engineer ships five tasks in the time that previously yielded four. Scale that across a team and you are running an entire extra developer-equivalent for free, with no new hire, no extra payroll, no team meeting over coffee.

A broader industry study cited by Upsun's analysis of context engineering pushes the headline number even higher. Citing GitHub and Accenture research, Upsun reports 55% faster task completion for developers using properly configured AI assistants, alongside a Microsoft study across 4,800 developers showing a 26% productivity gain. The same piece notes that 44% of AI code-quality issues trace back to missing context, and that proper architectural context cuts context-related errors by 65%.

The most damning line in that report is short: "AI models don't fail because they lack capability, they fail because they lack context." Andrej Karpathy is quoted there describing context engineering as "the careful art and science of filling the context window with just the right information for the next step." That sentence reframed the problem for me. I had been thinking of AI as a smart-but-stupid intern. The intern is not stupid. The intern is just blindfolded until you describe the room.

What Counts as AI-Ready Documentation

Not all documentation is created equal. A six-hundred-page enterprise wiki nobody updates is worse than a single page that is current and honest. I spent a long time figuring out what an AI agent actually needs from a project, and I have landed on three layers that each do something different.

Layer One: The Operating Manual

This is the layer most projects already have in some form, even if it is buried. It tells anyone — human or AI — what the project is and how to run it. Build commands. Test commands. Lint commands. The way directories are organized. The naming conventions. The frameworks you use and the ones you deliberately do not. The deployment story. Which environment variables matter. How to spin up a local instance.

The key is that the AI cannot infer this from the code alone in any reasonable time. It can guess, but guessing is exactly what burns the 20% efficiency tax the studies measure. Writing it down once turns a series of trial-and-error explorations into a single lookup.

A piece of practitioner data that surprised me here: shorter is often better. A blog post from EclipseSource on context files for AI coding agents describes a case where a manually refined 85-line context file outperformed an auto-generated 280-line version. The auto-generated version was longer but missed the implicit conventions only humans knew. The manually written one was tighter, and the AI behaved better. That matches my experience. Documentation written for AI is not a brain dump. It is a curated tour.

Layer Two: The Decision Log

This is the layer I missed for the longest time, and it turned out to be the most expensive miss. The decision log is where you record the architectural choices the project has already made and the reasoning behind them. Not what the code does, but why the code is the way it is.

Why this matters becomes obvious the first time you watch an AI agent re-litigate a settled question. A practitioner blog about Architecture Decision Records as AI Context describes a case where AI suggested the same "simpler" flat directory structure three separate times across a project's life, despite the team having explicitly chosen phase-based organization months earlier. Each time, the AI was being helpful in the way a Costco sample station is helpful: cheerful, identical, oblivious to the fact that you have already passed by twice.

The author's framing stuck with me: "AI operates on provided context. Without decision context, AI suggests changes that ignore past reasoning." And the kicker — "ADRs are not just documentation for future developers. They are governance mechanisms for AI-assisted development."

That second sentence is the part the engineering community is still catching up to. A decision record is no longer just for the next human engineer. It is the seatbelt that keeps an AI agent from accidentally driving you backwards through three months of progress.

For the MEV bot, my decision log includes things like why I chose a particular cycle-search approach over alternatives, why certain DEX integrations were deferred, why a specific failure mode is caught at one layer and not another, and the reasons we ruled out a tempting simplification that broke under live load. Each entry has the same three-part shape: the situation, the choice, the consequences. None of them is long. Together, they prevent an enormous amount of helpful nonsense.

Layer Three: The State of Play

The third layer is the smallest but the most actively maintained. It records what is happening right now: what I am working on, what is done, what is blocked, what the next planned step is. Think of it as the construction site whiteboard rather than the building's blueprint.

The AI uses this layer constantly. If I crash out of a session at midnight halfway through a fix, the state-of-play file lets the next session pick up at the right line of work instead of starting over from "what are we doing today." It is the difference between a relay race with a baton handoff and a relay race where every runner sprints back to the starting line first.

This layer breaks if it lies. A stale state file is worse than no state file, because it actively misleads. So I treat updating it the way pilots treat the pre-flight checklist — every time, no exceptions, even when it feels unnecessary.

The Mirror-and-Multiplier Effect

If you want a single sentence that explains why some teams get enormous lift from AI tools while others see almost nothing, the 2025 DORA report from Google has it: "AI doesn't fix a team; it amplifies what's already there." The DORA study, drawing on roughly five thousand respondents, found 90% AI adoption among software development professionals, with more than 80% reporting productivity improvements and 59% reporting positive effects on code quality.

But the more interesting finding is the variance. The report frames AI as a "mirror and multiplier": in cohesive, well-documented organizations it boosts efficiency, while in fragmented ones it highlights weaknesses. Connecting AI to internal documentation, codebases, and decision logs is called out explicitly as one of the key capabilities that drives output quality.

This is the part that should make every engineering manager pay attention. The default assumption a year ago was that AI tooling would close the gap between strong and weak teams by giving everyone the same lift. The data so far suggests the opposite. Strong teams — meaning teams that already document, decide deliberately, and keep state — get a much larger multiplier. Weak teams get amplified weakness. The same tool can be a force multiplier or a chaos multiplier depending on what it is multiplying.

There is also a humbling number in the report. Per DORA's findings, 64% of surveyed professionals already use AI for writing documentation. The tool that depends on documentation is now also the tool helping us write it. That loop can either compound the gains — better docs feed better AI feeds better docs — or compound the slop, if no human is checking the output. Which loop you end up in depends on how seriously you take the manual review pass.

The Living-Document Problem

The single fastest way to ruin everything I described above is to write a great set of documents once and let them rot. Documentation that lies is worse than documentation that does not exist.

A Yelp review from 2015 will tell you a restaurant is great. The restaurant has changed owners three times since then. The review is still there, still cheerful, still wrong. The same dynamic destroys project documentation. Last spring's architecture overview, written when the system had two services, is now actively misleading because the system has six. An AI agent reading it will confidently propose changes based on a layout that does not exist.

The EclipseSource piece I cited earlier puts a name on the discipline this requires: treat context files as living documents. Update them whenever the AI agent consistently struggles with something that should be obvious. That "consistently struggles" signal is gold. If an AI keeps making the same wrong assumption, the bug is almost always in the documentation, not in the AI. The fix is one or two lines added to the right place, and the recurring failure stops.

This is the loop I have settled into for the MEV bot. When I catch myself re-explaining the same thing across sessions, I stop, write the explanation into the appropriate document layer, and never have to repeat it. The friction of writing it down is real, but it is paid once. The friction of re-explaining it is paid forever.

There is an industry trend pushing in the same direction. A 2026 trends roundup from Document360 describes a shift from documentation that is "written for humans" to documentation that "actively thinks, adapts, and keeps humans in the loop." The piece also flags a Gartner-cited forecast that 75% of developers will use AI tool connectors to their documentation by 2026. The plumbing is getting better. The discipline of keeping the source material honest still falls on the human.

The Trust Gap and Why Documentation Closes It

One of the most useful numbers in the DORA data is how cautious developers still are about AI output. Per the same Google DORA report, only 24% of respondents trust AI "a lot" or "a great deal," with just 4% in the top "great deal" bucket. About 30% trust it "a little" or "not at all." Eighty percent say it boosts productivity, but a small minority would call it a trusted partner.

That gap — productive but not trusted — is a documentation problem in disguise. The reason I do not fully trust AI output is that I cannot quickly verify it against the project's actual rules. If I had to choose between an AI that produced perfect-looking code that violated three of my project's conventions and an AI that produced slightly clumsier code that respected all of them, I would take the second one every time. Trust is not about confidence in the model. It is about confidence in the constraints.

Documentation is how you encode the constraints. The conventions, the rejected approaches, the things-we-do-this-way-because-we-tried-the-other-way — written down, those become guardrails that turn a clever-but-unaccountable collaborator into a clever-and-accountable one. The AI is the same model either way. What changes is the rails it runs on.

For an MEV bot specifically, the trust question has a sharp edge. A bad code suggestion in a normal product gets caught in code review. A bad code suggestion in a bot that sends signed transactions to a public ledger gets caught by losing real money. The cost of an unverified change is asymmetric, which means the cost of weak documentation is asymmetric. I would rather slow down to write a decision record than speed up into a transaction I cannot defend later. The Solana MEV arena runs on tip-priority competition, where small advantages in execution quality decide who wins a block — and where weak documentation directly degrades execution quality.

What I Stopped Doing

Learning to document for AI also meant unlearning a few habits. Three in particular have been worth giving up.

The first is writing documentation that is mostly history. Old docs were narratives — "we built this in 2024, then we added this, then we refactored that." Useful for a memoir, useless for an agent that needs to know what is true right now. I rewrote everything in present tense, with the current state at the top of each file and the timeline below. The compiled-truth-on-top, history-below pattern is the single best structural change I made.

The second is treating documentation as a single document. The instinct to put everything in one giant README dies hard. Splitting the operating manual, the decision log, and the state of play into separate files lets the AI pull only the layer it needs for the current task. The cost of one big file is that the AI has to chew through irrelevant context every time. The cost of three smaller files is one extra path to remember. Worth it.

The third is auto-generating context files and calling it done. Auto-generated docs are useful as a starting point, the same way a Roomba is useful as a starting point for cleaning your kitchen — better than nothing, not the same as actually cleaning your kitchen. The 85-versus-280-line example from the EclipseSource piece I cited earlier matches exactly what I see. Manual refinement matters because the things that hurt most in AI output are exactly the things auto-generation cannot see: the unwritten rules, the implicit reasons, the cases where the obvious answer is wrong.

What the Numbers Mean for a Solo Project

Most of the studies I have cited describe team environments — 4,800 developers at Microsoft, 5,000 respondents in the DORA survey, 124 pull requests across organizational repositories. The MEV bot is a small project, not an enterprise. Does the same lift apply?

In my experience, it applies more, not less. A solo project has no other humans absorbing the tribal knowledge. There is no senior engineer to ask, no Slack channel where the right answer surfaces, no veteran teammate who corrects the AI's drift in code review. The documentation is the entire institutional memory. The 20% efficiency tax from missing context, measured in the arXiv study, is something I pay alone if I do not write things down.

And the asymmetric-cost point cuts even harder for a one-person bot. If I introduce a bug in a feature, I find it. If I introduce a bug in execution logic that touches real chain state, I find out the expensive way. Anything that reduces the rate at which AI introduces avoidable bugs has compounding value. The 65% reduction in context-related errors cited in the Upsun piece I referenced earlier is not a productivity number for me. It is a survival number.

How to Tell Your Documentation Is Working

There is a simple test. Open a new session with a fresh AI agent, ask it to do something nontrivial, and watch what happens in the first five minutes.

If it asks questions you have already answered, your documentation is incomplete. If it makes assumptions that contradict your conventions, the conventions are not written down clearly enough. If it proposes a change you rejected three months ago, your decision log has a gap. If it operates from an outdated picture of the system, your state file is stale.

If, on the other hand, it gets going immediately, references the right conventions, avoids the rejected paths, and produces a draft that is roughly in the shape your project would expect — you have crossed a real threshold. Everything that used to require a verbal walkthrough now happens via the documents.

I was deeply skeptical of the "around a 90% success rate" anecdote in the EclipseSource piece when I first read it. It sounded like marketing. But after refining the three layers for the MEV bot over several months, I get it. There is a phase change. Below a certain documentation quality, working with AI feels like supervising a confused intern. Above it, the same tool feels like working with a colleague who happens to have read the entire codebase in one sitting. Same model. Different rails.

What This Means Going Forward

The industry conversation around AI coding tools has spent most of its time on the model — bigger, smaller, faster, smarter, cheaper. The data I have read this past year suggests the conversation is shifting toward the substrate. The model is the engine. The documentation is the road. You can put a Formula 1 engine on a gravel path and it will not go faster than a Honda Civic on the highway.

For anyone building something nontrivial, the practical implication is uncomfortable: writing the manual is now part of the engineering work, not a chore to be deferred. The teams and projects that internalize this will get a multiplier that compounds with every session. The ones that do not will keep paying the onboarding tax forever, every morning, to a coworker who cannot remember yesterday.

For my own project, the bot is not done. The cycles I am tightening, the integrations I am adding, the failure modes I am still learning — those are not finished. But the manual that explains them to my forgetful AI coworker is something I keep current as I go. It is the only thing that lets a one-person project work at the pace of a team.

Key Takeaways

• AI coding tools fail more from missing context than from missing capability. A peer-reviewed study measured roughly 20% faster task completion and 20% fewer output tokens when AI agents had access to project context files, statistically significant at p<0.05.
• Documentation in the AI era has three useful layers: the operating manual (how the project runs), the decision log (what was chosen and why), and the state of play (what is happening right now). Each layer answers a different question the AI cannot infer from code alone.
• The DORA 2025 report frames AI as a "mirror and multiplier" — it amplifies cohesive, well-documented teams and equally amplifies fragmented ones. The differentiator between getting lift and getting noise is the quality of what AI is being multiplied against.
• Shorter, manually refined documentation beats longer auto-generated documentation. Documentation must be living — updated whenever the AI consistently struggles with something obvious — because stale docs are worse than no docs.
• Trust in AI output is structurally limited (only 24% of professionals trust AI "a lot or a great deal" per DORA). Documentation closes that trust gap by encoding the project's constraints into guardrails the AI runs against, turning a clever collaborator into an accountable one.