Why I Keep a Failure Log While Building a Solana MEV Bot

I Keep Almost Solving the Same Bug Twice

I am sitting in front of a terminal at 1 AM, staring at a stack trace that feels uncannily familiar. The error is in a code path I touched maybe three weeks ago. Same module. Similar symptom. I know — somewhere deep in my notes — that I solved this once. I just cannot remember how, or why, or even whether I solved it cleanly or just papered over it.

This is the moment that justifies every minute I spend writing things down. Not because the writing itself is glamorous, and not because I enjoy the act of documenting my own confusion. But because, without those notes, I am about to spend another two hours rediscovering what a past version of me already paid the price to learn.

What Started Out as a Personal Habit

When I started this MEV bot project, I had no intention of running anything resembling a postmortem culture. That phrase belongs to organizations with on-call rotations, P0 incident channels, and whole teams of SREs. I am one person at one keyboard, working with an AI agent and a pile of unfamiliar protocols.

But within a few weeks, I notice something uncomfortable. The bot's bug stack is not really a stack. It is a wheel. The same categories of mistakes keep cycling back: a slippage assumption that was wrong the first time I made it; a misread of how an instruction's accounts are ordered; an off-by-one in a fee calculation; a forgotten edge case in a token's transfer behavior. Each time I fix one, I tell myself, "this is the last time I will trip on this." Each time, I am wrong.

So I start keeping a log. Not a beautifully formatted document — just a flat file where I write the date, what broke, what I thought was happening, what was actually happening, and what I changed. Over time it stops being a log and starts being a tool. When I see an error that smells familiar, the first thing I do is search the log.

This article is about why that habit pays for itself, and what serious engineering organizations have figured out about the same problem at much larger scale.

What Google Already Learned About Forgetting

Google runs an internal tool called Requiem that holds thousands of incident postmortems dating back to 2009, according to the Google SRE Workbook. The same source describes "FixIt weeks" twice a year where teams compete on action-item closure, a Wheel of Misfortune game where engineers reenact old incidents to internalize lessons, and a Postmortem of the Month newsletter.

This sounds like ceremony. It is not. It is infrastructure for refusing to repeat mistakes.

The piece of evidence that lands hardest for me, sitting alone in front of my bot, is buried in a case study in the SRE Workbook. Google describes an outage that took out roughly 60% of normal global query traffic from the affected satellites. Recovery took on the order of 36 hours to restore half of edge serving capacity, plus additional time to fully rebalance. The traffic team spent approximately 48 hours in all-hands reconstruction mode. Three years later, a similar incident occurred — the same general class of failure, the same systems. But because the action items from the first postmortem had been implemented, Google reports that the second outage's blast radius and rate were "dramatically reduced." The wheel did not get to make a full rotation. That is what the practice buys you.

The principle that makes the system work is blameless culture. The Google SRE book quotes engineer Devin Carraway summing up the cultural attitude as "The cost of failure is education." When the focus shifts from "who broke it" to "what condition allowed this to break," people stop hiding their mistakes — and the organization stops paying for the same mistake repeatedly.

Ben Treynor Sloss, Google's VP for 24/7 operations, puts the operational bite on it in the same source: "a postmortem without subsequent action is indistinguishable from no postmortem." Writing the document is not the point. The action items are. If nothing changes, the writing was a ritual.

I Am Not Google, But I Have the Same Problem

The single-person version of postmortem culture feels absurd at first. I am not afraid of blaming a teammate, because there are no teammates. I do not need anonymous voting on action items. I am the commander, the on-call, the post-incident reviewer, and the implementer all rolled into one person at one keyboard.

But the underlying problem — the human brain's tendency to forget specifics quickly — does not care about org chart size. Solo developers have their own version of institutional amnesia. I just call it "what did I do last month?"

Paul I Taylor, a practitioner writing about institutional forgetting, describes maintaining a 12-year photographic work log that he reviews almost daily. Even with that discipline, he discovered that one problem in his work had been attempted at least five separate times over 10 years, with each attempt failing to reference the previous failures. Five attempts. One person. A real, documented log. And the wheel still spun, because the log was not searched at the right moment.

Taylor quotes Arnold Kransdorff: "Institutional 'forgetting' is the single biggest constraint to decision-making excellence." Substitute "personal" for "institutional" and the sentence still holds.

I find this oddly reassuring. The problem is not that I am uniquely sloppy. The problem is that human memory is bad at retaining the specific shape of a bug after the dopamine of fixing it has worn off. The mistake feels obvious in the moment, then it dissolves. Two weeks later, the shape of the new bug feels novel — but it is not. It is a cousin of something I have already paid for.

The Curse of Knowledge, Right After You Solve It

There is a clean cognitive science explanation for why I keep skipping documentation, and it shows up across multiple sources. It is called the curse of knowledge, traced in part to Baruch Fischhoff's 1975 work on hindsight bias. The Wikipedia summary describes it as the bias where, once you have acquired a piece of knowledge, you have trouble reasoning about what it was like not to have it.

The implication for failure logging is brutal. The moment after I solve a hard bug is the moment when the solution feels most obvious — and therefore the moment when I am least motivated to write it down. "Of course this is how the instruction layout works," I think. "Obviously the swap direction matters for that pool type. I will remember this." I will not.

This is exactly the trap. The window of maximum motivation to log is the window when the lesson feels least worth logging. Past me, three weeks ago, thought the same thing. Past-past me, two months ago, also thought the same thing. The log I never wrote is a tax future me pays, in 2 AM debugging time, every single time.

The fact that the cumulative damage is invisible makes it worse. As one developer documentation overview puts it, the damage done by not documenting technical decisions is cumulative — like tech debt. There is no alarm that goes off the day you skip a log entry. There is just a steady accumulation of pain that arrives, untraceable to its root cause, weeks later. Tech debt at least shows up on a dashboard somewhere. Memory debt does not.

The Failure Rate Paradox

There is a counterintuitive piece of math that I keep returning to. It comes from Lucian Radu Teodorescu, a staff engineer who wrote up 20 years of software engineering lessons in the ACCU Overload Journal.

He notes that a typical engineer writes about 300 lines of code per day, but the actual typing represents less than five minutes — roughly 1% of the working time. The other 99% is problem-solving. Code is mostly thinking, and only a little finger movement.

Then he runs a thought experiment on failure rates. Imagine two engineers. Engineer A ships 100 features per year with a 5% failure rate. Engineer B ships 10 features per year with a 1% failure rate. By the end of the year, A has delivered 95 working features. B has delivered 9.9. The "low failure rate" engineer is dramatically worse on net output.

Teodorescu's takeaway, which I have written on a sticky note near my monitor, is that mistakes accelerate learning when their impact remains controlled.

For my MEV bot, this reframes what failure even means. A failure that costs me nothing — a transaction that gets rejected before any value moves, a strategy I test in simulation and discard — is essentially free education. A failure that costs me real money is expensive education. The job is not to avoid failures. The job is to keep their cost survivable and their lessons cumulative. The failure log is what makes the lessons cumulative. Without it, the cost is paid and the education is forgotten.

This is why I do not feel guilty when I look at my list of unsuccessful experiments. I feel guilty when I find one I clearly attempted before and never wrote up.

What Goes in the Log — and What Stays Out

I am deliberately not going to describe the exact structure of my failure log. The format is the smallest part of this. What matters is what kind of content actually survives contact with future-me.

A few things I have learned through trial and error about what is worth capturing:

The symptom, in my own words first. Not the framework's error message. Not the stack trace's top line. The thing that made me notice something was wrong. "Bot stopped including this DEX in routes for 30 minutes around 11 PM." "Profit estimate is high, on-chain settlement is failing." The reason matters: when I search the log later, I will be searching with the same kind of vague description, because that is what failures feel like in the moment they happen again.

What I believed was happening, before I had evidence. This is the most embarrassing part of the log and the most useful. The hypothesis I held that turned out to be wrong is exactly the hypothesis I will form again next time, because my mental model has not changed. Writing it down forces me to confront the gap.

What was actually happening, with the specific observation that proved it. Not "it was a race condition." That tells me nothing. Instead: "the account ordering depended on a flag that flips when the pool's reserve crosses a threshold, and my code assumed the flag was static." That kind of sentence is searchable. The phrase "race condition" is not.

What I changed, and what I deliberately did not change. I have learned that the second half is as important as the first. The fixes I chose not to make — because they were out of scope, because they would have hidden the root cause, because they would have created new failure modes — are the breadcrumbs that prevent me from undoing my own decisions later.

The cross-reference. If this is the second or third time I have seen something in this family, I link the earlier entries. The point is to make the wheel visible. If the same module shows up in five entries, I have evidence that the design is wrong, not that the patches are wrong. That distinction matters enormously.

Notice what is not on this list: not the actual fix, not the diff, not the line numbers. Those live in source control. The log is for the parts that source control cannot capture — the reasoning, the failed hypotheses, the observations that closed the gap.

What the Practice Buys a Solo Builder

The benefit I expected when I started doing this was efficiency: less time wasted re-deriving things. That benefit is real, but it is not the largest one.

The biggest payoff is that the log changes how I make decisions in the moment, not just how I debug later. When I know that a bad decision today will end up in a written entry where I will have to face it with my own name on it, I think harder before making the bad decision. The log is a mirror.

There is a related effect at the design level. When the same module shows up in five log entries over three months, I cannot pretend the problem is "tricky bugs." The data says the abstraction is wrong. I would not see that pattern without writing each instance down at the time, because in the moment each failure looks novel. The log aggregates them into a signal.

A community of practitioners writing about postmortem culture frames this as a cultural signal. Thiago Avelino's post on postmortem culture puts the motto as: "we don't waste pain here." For an organization, the audience for that signal is everyone. For a solo developer, the audience is the only person who matters in this codebase — me, three weeks from now. The signal still works. I am telling future-me that the pain we paid was worth something.

Project management research backs the broader pattern. The PMI Learning Library notes that organizations without a habit of documenting lessons learned before project closure "tend to repeat the same mistakes over and over again." The phrasing is almost too on the nose. The repetition is not bad luck. It is the predictable outcome of not writing things down.

What a Failure Log Is Not

A failure log is not a substitute for fixing the underlying design. If five entries point to the same module, the answer is to redesign the module, not to add a sixth entry. The log is diagnostic, not therapeutic. It tells me where the wheel is spinning. Stopping the wheel is a separate act.

It is also not a substitute for tests. A passing test catches a class of failures before they happen. A failure log catches the next instance of a failure that already happened. Both are necessary. Neither replaces the other.

And — this matters in the AI-assisted era — a failure log is not something I can offload onto an AI agent. The agent can help me write a particular entry once I have decided what to say. It cannot decide that an experience was worth writing down, because the value of that decision depends on patterns I notice across my own work. The judgment is mine. The transcription can be shared.

A log is also not, by itself, an action item tracker. This is the trap Ben Treynor Sloss warned about: writing without follow-through is theater. If a log entry surfaces a needed change to my code or my process, that change has to land. Otherwise the entry becomes a sad museum exhibit — proof that I noticed the problem, and proof that I did nothing about it.

What It Means Going Forward

Building this MEV bot with an AI agent has changed the velocity at which I can produce code. It has not changed the rate at which I produce mistakes. Arguably it has increased that rate, because I am attempting more, sooner, with less personal mastery of every layer. That is not a complaint — the velocity is the point — but it does mean the cost of forgetting is higher than ever. The faster I move, the more entries the wheel can fit into a month.

The practice that scales with that velocity is not faster typing. It is more diligent recording of what I have already learned. A two-minute log entry written tonight is worth an hour of debugging next month. That ratio gets better, not worse, as the project grows and the surface area expands.

I am still bad at this. I still skip entries when I am tired. I still find old bugs that I clearly solved once and forgot. The discipline is not solved, only practiced. But I have learned to trust the practice enough to keep doing it on the nights when it feels like a waste of time — because those are the nights when, statistically, I am about to make a future-me very grateful.

The wheel keeps spinning whether I write things down or not. The log is the only thing that lets me see it spinning. And seeing it is the first step toward making it slow down.

Key Takeaways

• Memory is not a tool. The moment after you fix a bug is when its specifics start dissolving, and the curse of knowledge makes the lesson feel "too obvious to write down" exactly when it is most worth writing down.
• Postmortem culture scales down to one person. Google's three-year recurrence case study, where action items from the first incident dramatically reduced the blast radius of a similar later incident, demonstrates the value of the practice at scale — but the same logic applies to a solo project log.
• Failure rate is not the right metric. A controlled failure rate at high velocity produces more net output than a near-zero failure rate at low velocity, per Teodorescu's thought experiment. The log is what keeps failures controlled, meaning their lessons accumulate instead of evaporating.
• The format does not matter; the habit does. The content that survives contact with future-you is the hypothesis you got wrong, the observation that closed the gap, and the cross-reference to the last time this happened. The exact file format is the smallest part.
• A log without action items is theater. Surfacing a repeated mistake is only half the work. If the design does not change, the wheel keeps turning — and the log just becomes a more detailed record of the same revolutions.