Log Analysis — What the Data Tells You

The bot is running. Pools are being scanned. Cycles are being detected. Transactions are being built. Bundles are being submitted to Jito. The system is alive in every observable sense — processes are active, logs are scrolling, the CPU is busy. The question I keep asking myself is a simple one: is it working?

And I realize I do not actually know.

There is a difference between "running" and "working." A car engine can run all day in a driveway. It is burning fuel, generating heat, spinning the crankshaft, doing everything an engine does. But it is not taking anyone anywhere. The engine is running. The car is not working. The distinction requires a definition of what "working" means, and that definition requires measurement.

I have been operating on intuition. The bot starts. I see log lines scrolling past. Opportunities are detected. Bundles are submitted. The system feels active. And from that feeling, I construct a narrative: things are probably fine. There are probably some successes mixed in with the failures. The bot is probably capturing some value, even if I cannot point to a specific instance right now. The word "probably" is doing an extraordinary amount of heavy lifting in that sentence, and I have not noticed until now.

This is the moment where I stop trusting how things feel and start measuring what things are.

The Annual Physical You Have Been Skipping

Everyone has had the experience of going to the doctor for an annual physical after putting it off for a couple of years. You feel fine. You exercise occasionally. You eat reasonably well. You sleep enough, mostly. Nothing hurts. Nothing is visibly wrong. You walk into the office confident that the visit will be a formality — a handshake, a stethoscope, a rubber stamp.

Then the bloodwork comes back. Your cholesterol is higher than you expected. Your blood sugar is in a range you did not know existed between "normal" and "diabetic." Your vitamin D is low. Your blood pressure, which you assumed was fine because you feel fine, is elevated enough that the doctor uses the phrase "we should keep an eye on this."

None of this information was available through intuition. You cannot feel your cholesterol. You cannot sense your blood sugar. Your body does not send you a notification when your vitamin D drops below the recommended range. The only way to know these things is to measure them, and the only way to measure them is to draw blood and run the numbers.

The bot is the body. The logs are the bloodwork. And I have been walking around assuming everything is fine because nothing hurts, when the reality is that I have never actually looked at the numbers.

Building the Dashboard

The first step is making sure the bot records everything. Not just errors. Not just successes. Everything. Every scan, every cycle detected, every opportunity evaluated, every attempt made, every outcome observed. Each event needs to be structured — not a free-form text string that a human can read, but a structured record that a program can aggregate.

This is the difference between a shoebox full of receipts and a spreadsheet. Both contain financial information. Both record transactions. But the shoebox requires a human to sort through it, read each receipt, mentally categorize each expense, and attempt to hold a running total in their head. The spreadsheet allows formulas. It allows sums, averages, counts, filters. It allows you to ask "how much did I spend on groceries in March" and get an answer in milliseconds instead of hours.

The bot's previous logging is the shoebox. It records things. It writes lines to a file. A human — me — can scroll through and read them. But asking a quantitative question requires reading every line, which is not feasible when the bot generates thousands of events per hour. The structured logging is the spreadsheet. Each event has fields. Each field has a type. Each record can be filtered, grouped, counted, averaged.

The events I need to capture fall into natural categories. There is the scanning phase — how many pools are being monitored, how frequently the on-chain state is being read, how fresh the data is when it arrives. There is the detection phase — how many potential arbitrage cycles are identified, what their estimated profit looks like before execution, which pools are involved. There is the attempt phase — how many opportunities survive filtering and reach the point of bundle submission. And there is the outcome phase — what happened after submission, whether the bundle landed on-chain, whether it achieved the expected result.

Each phase generates events. Each event carries data. The data accumulates. And for the first time, I can ask questions and get answers that are not based on vibes.

The Weight Tracker That Does Not Lie

There is a specific kind of disappointment that comes from stepping on a scale after a month of what you believed was disciplined eating. You cut out the obvious junk food. You started eating salads for lunch. You switched from regular soda to diet. You made what felt like meaningful changes, and you expected the numbers to reflect that effort.

The scale says you have not lost a pound. Maybe you have gained one.

The scale is not being cruel. The scale is being accurate. What happened is that the salads had ranch dressing and croutons and cheese. The "diet" was accompanied by larger portions at dinner because you felt virtuous from the salad. The soda switch saved a couple hundred calories a day, but the evening snacking added them back and then some. Each individual decision felt like progress. The aggregate was stasis, or worse. And the only way to know that was to step on the scale.

The first time I aggregate the bot's structured logs, I experience the scale moment. I have been running the bot for a stretch of time. I have watched log lines scroll past. I have seen opportunities detected, bundles submitted, the machinery of arbitrage churning away. In my mental model, the bot is capturing some value. Not a lot — I know the system is still being tuned — but some. Surely some.

The numbers say otherwise. The attempt count is substantial. The bot is trying. It is detecting opportunities, evaluating them, constructing transactions, submitting bundles. The machinery works. But the success count — the number of bundles that actually land on-chain and produce the expected result — is effectively zero. Not low. Not disappointing. Zero.

This is not a rounding error. This is not a statistical fluke. This is a fundamental reality that was invisible to me because I was watching log lines instead of counting outcomes. The scrolling text created an illusion of activity that I interpreted as progress. The data says there is activity without progress. The engine is running. The car is not moving.

The Report Card Breakdown

When a student brings home a report card with a low overall grade, the first question a parent asks is not "how do we fix your grade" but "where are you losing points." The overall GPA is a summary statistic. It tells you there is a problem but not what the problem is. The breakdown — grades by subject, test scores versus homework scores, participation marks, specific teacher comments — is where the actionable information lives.

A student might have a low GPA because they are failing one subject and acing the rest. Or because they are getting C-minuses across the board. Or because their test scores are fine but they never turn in homework. Each of these scenarios produces the same summary statistic but requires a completely different intervention. Tutoring helps the first case. General study habits help the second. An organizational system helps the third. You cannot prescribe the treatment without the diagnosis, and you cannot make the diagnosis from the summary alone.

The bot's zero success rate is the low GPA. It tells me something is wrong but not what. The breakdown — classifying each failure by its cause — is where the actionable information lives.

So I start classifying. Every failed attempt gets a category. Why did this specific bundle not land? Why did this specific opportunity not convert into profit? The categories emerge from the data, not from my preconceptions. I do not decide in advance what the failure modes should be. I read the failures and let them sort themselves.

The categories that emerge are instructive. There is a category for stale data — opportunities that looked profitable based on the on-chain state the bot observed, but by the time the bundle reached the validator, the state had changed. Someone else traded against the same pool. A price moved. The opportunity evaporated between observation and execution. This is the temporal gap problem, the lag between seeing the world and acting on it.

There is a category for competition — bundles that were technically valid, targeting real opportunities, but lost the Jito auction to another searcher who offered a higher tip or submitted faster. The opportunity was real. The bot's response was correct. But someone else got there first.

There is a category for AMM calculation errors — the bot's math predicted a profitable trade, but the actual on-chain execution would have produced a different result. Rounding differences. Fee calculations that do not perfectly match the AMM's implementation. State transitions that the bot's model does not capture with full fidelity.

There is a category for insufficient resources — attempts that fail because the wallet does not have enough of a particular token, or because the transaction exceeds compute limits, or because an account is in an unexpected state.

And there are edge cases that do not fit neatly into any category — unusual AMM states, unexpected program behavior, network conditions that the bot was not designed to handle.

The Budget Spreadsheet Revelation

Anyone who has ever created a household budget knows the moment of revelation that comes from actually categorizing expenses. You know you spend money. You know roughly how much comes in and roughly how much goes out. You have a vague sense that you spend "too much" on eating out or "a lot" on subscriptions. But "too much" and "a lot" are feelings, not numbers.

Then you sit down with three months of bank statements and a spreadsheet. You create columns: rent, utilities, groceries, restaurants, subscriptions, transportation, entertainment, miscellaneous. You go through every transaction and assign it to a column. You sum the columns.

The results are never what you expected. The category you thought was your biggest expense turns out to be second or third. The category you never thought about — maybe it is the "miscellaneous" catch-all, maybe it is the sum of all the small daily purchases — turns out to be enormous. You discover that you spend more on weekday lunches bought near the office than you spend on the weekend dinners you feel guilty about. You discover that the streaming services you use add up to more than your electric bill. The numbers rearrange your mental model of your own spending.

The failure category breakdown does the same thing to my mental model of the bot's performance. Before I look at the data, I have assumptions about where the problems are. I assume that competition is the dominant factor — that the bot is finding real opportunities but losing them to faster, better-funded competitors. This assumption feels right because it is flattering. It implies that my bot is correct in its analysis and merely slow in its execution. The fix, under this assumption, is to be faster. Optimize the latency. Submit quicker. The problem is a speed problem.

The data tells a different story. Competition losses are present, but they are not the dominant category. The largest category, by a significant margin, is stale data. The bot is making decisions based on state that has already changed by the time the bundle is submitted. This is not a speed problem in the traditional sense — it is not that the bot is too slow to win a race. It is that the bot is racing toward a finish line that has already moved.

The second largest category is AMM math discrepancies — cases where the bot's calculation of expected profit does not match what the on-chain program would actually produce. This is an accuracy problem, not a speed problem. The bot is not losing to competitors; it is losing to its own miscalculations. The opportunities it targets are phantoms — artifacts of imperfect modeling.

The proportions matter. If competition losses dominated — say, making up the vast majority of all failures — the right investment would be latency optimization. Faster data feeds. Faster transaction construction. Faster submission. If stale data dominates, the right investment is data freshness — more frequent state polling, better staleness detection, faster invalidation of outdated state. If math errors dominate, the right investment is calculation accuracy — auditing the AMM math, fixing rounding differences, aligning the bot's model more precisely with on-chain reality.

Without the data, I would have invested in speed. The data says to invest in accuracy and freshness. These are completely different engineering priorities. The budget spreadsheet just saved me from renovating the kitchen when the real problem was the car payment.

The Patterns You Cannot See Without Numbers

Raw counts tell part of the story. Distribution over time tells the rest.

When I start plotting failure events against time-of-day and day-of-week, patterns emerge that were invisible in the scrolling log lines. There are periods when the stale data problem intensifies — times of day when on-chain activity spikes and the bot's state snapshots become outdated faster. There are periods when competition losses cluster — hours when other MEV searchers are most active. There are periods when certain AMM math errors appear more frequently, correlating with specific market conditions that stress the bot's calculations.

These temporal patterns are the equivalent of discovering that you always overspend on Fridays. The individual Friday purchases do not look unusual. A dinner here, a purchase there. But when you aggregate across many Fridays, you see a consistent spike that does not appear on Tuesdays. The spike has a cause — maybe it is payday, maybe it is the psychological release of the work week ending, maybe it is social plans that always seem to land on Friday. The point is that the pattern only becomes visible through aggregation. A single Friday looks normal. Twenty Fridays reveal a trend.

For the bot, the temporal patterns point toward environmental factors that I was not previously considering. The Solana network has rhythms. Validator schedules create patterns. Market activity in different time zones creates waves. Other bots and searchers have their own schedules, their own peak activity periods. My bot operates in this environment, and the environment's patterns shape its outcomes. Without the data, I am oblivious to these rhythms. With the data, I can start adapting to them.

What Gets Measured Gets Fixed

There is a management cliché that says "what gets measured gets managed." The cliché is overused, but it contains a kernel of truth that applies here with unusual precision. Before I built the measurement system, I was managing nothing. I was running a bot and hoping. Hope is not a strategy, but it was functioning as one.

The measurement system changes my relationship with the bot in a fundamental way. Before, I interact with it as a creator — I build features, fix bugs, deploy code, and observe behavior qualitatively. After, I interact with it as an analyst — I ask quantitative questions, receive quantitative answers, and make decisions based on evidence rather than instinct.

The first decision the data drives is a reordering of priorities. My to-do list before the analysis was organized by what felt most important. Latency optimization was near the top because being fast feels important in a competitive environment. After the analysis, the to-do list reorganizes itself around what the data says will have the most impact. Improving data freshness moves to the top because stale data is the largest failure category. Auditing AMM math moves to second because calculation accuracy is the second largest. Latency optimization moves down — not off the list, but down — because competition losses, while real, are a smaller portion of the total than I assumed.

This is exactly the experience of someone who goes to their annual physical expecting to hear "eat less red meat and exercise more" and instead hears "your thyroid levels are off." The expected advice addresses a problem that exists but is not primary. The actual finding addresses the thing that is actually dragging the numbers down. You do not know what is actually dragging the numbers down until you look at the numbers.

The Spreadsheet Habit

Building the analysis was a one-time effort. Using it is a daily practice.

I start every session now by looking at the aggregated data before touching any code. What happened overnight? How many attempts? How many successes? What is the failure distribution? Have the proportions shifted? Did yesterday's code change move any needle?

This is the habit of the person who has finally started tracking their spending. For the first month, they check the spreadsheet compulsively. Every purchase gets entered. Every category gets reviewed. Slowly, the compulsive checking settles into a routine — a weekly review, a monthly summary, a quarterly adjustment. The habit becomes part of the operating rhythm, as natural as checking the weather before getting dressed.

The data review habit transforms debugging from an emotional process into an analytical one. Before the data, a bad day feels like a bad day — the bot did not perform, something is probably broken, time to dig through code and look for issues. After the data, a bad day is a data point. What changed? Did a specific failure category spike? Did a new failure mode appear? The data does not eliminate the emotional response — it is still frustrating when the bot underperforms — but it channels the response into productive analysis instead of unfocused anxiety.

Trusting the Uncomfortable Truths

The hardest part of the data-driven approach is accepting what the data says when it contradicts what I believe.

I believe the bot's AMM math is good. I spent significant time implementing and testing the calculations. I verified them against known inputs and outputs. I am confident in the math. And the data says the math is producing a nontrivial number of phantom opportunities — cases where the bot thinks a trade is profitable but the on-chain reality disagrees. The data does not care about my confidence. The data counts the discrepancies.

I believe the data pipeline is fast enough. The bot polls on-chain state at regular intervals. The intervals feel short. The data says a substantial portion of all failures trace back to stale state — information that was accurate when received but outdated by the time it was used. The data does not care about my belief in the polling frequency. The data measures the gap between observation and action.

This is the scale again. You believe the diet is working. The scale says it is not. You can argue with the scale. You can blame water weight, or muscle gain, or the inaccuracy of home scales, or the phase of the moon. Or you can accept the number, adjust the approach, and measure again next week. The scale will still be there, still measuring, still indifferent to your feelings about what it should say.

I choose to trust the data. Not because it is comfortable — it is frequently uncomfortable — but because the alternative is to continue operating on intuition, and intuition gave me a mental model in which the bot was "probably" working when it was producing zero results. Intuition is the friend who tells you what you want to hear. Data is the friend who tells you what you need to hear. The second friend is less fun at parties, but more useful in practice.

The Numbers Are the Starting Point

The data does not fix anything by itself. Knowing that stale data is the primary failure category does not make the data fresher. Knowing that AMM math discrepancies are the second category does not correct the calculations. The data is diagnostic, not therapeutic. It tells you where to look, not what to do when you get there.

But knowing where to look is the difference between systematic improvement and random flailing. Without the data, I would be changing things based on gut feel — spending a week optimizing latency because speed feels important, then spending a week adjusting parameters because tuning feels productive, then spending a week refactoring code because cleanliness feels virtuous. Each change might or might not address the actual problem. I would not know, because I would not be measuring.

With the data, every change has a hypothesis and a measurement. If I improve the data pipeline, the stale data category should shrink. If the stale data category does not shrink, the improvement did not work, and I need to try something else. The feedback loop is tight and honest. The data does not flatter. The data does not equivocate. The data counts what happened and reports it without editorial.

This is the lesson of the annual physical, the budget spreadsheet, the bathroom scale, the student's report card. The numbers are not the answer. The numbers are the question, stated precisely enough that an answer becomes possible. "How is the bot performing?" is not a question that can be answered productively. "What percentage of failures are caused by stale data, and has that percentage changed since I modified the polling interval?" is a question that the data can answer in seconds.

I decide to only trust what the data says. Not because data is infallible — it can be measured wrong, aggregated poorly, interpreted incorrectly. But because the alternative, trusting my intuition about a complex system running thousands of operations per hour in a competitive environment, has already proven itself unreliable. The intuition said "probably working." The data said zero. Between the two, I know which one I am going to believe going forward.