WebSocket Data Feeds — The Trap of Real-Time

The arbitrage bot needs to know the state of liquidity pools. Not what they looked like five seconds ago, not what they looked like one second ago, but what they look like right now. In the world of on-chain MEV, "right now" is the only moment that matters. A pool's reserves shift with every trade. A price ratio that represents a profitable cycle at timestamp T might be a losing proposition at timestamp T plus 300 milliseconds. The bot's entire value proposition depends on the freshness of its data, and the freshness of its data depends on how that data arrives.

There are two ways to get real-time blockchain state: polling and subscribing. Polling means asking. Subscribing means listening. The difference is the difference between calling someone every thirty seconds to ask if anything has changed, and asking them to call you whenever something changes. One of these approaches is obviously more efficient than the other, and that is why I am building the data pipeline around WebSocket subscriptions.

Calling Every Thirty Seconds

HTTP polling is the straightforward approach. The bot sends an RPC request — "tell me the current state of this account" — and the server responds with the data. Simple. Reliable. Stateless. Each request is independent. If one fails, you send another. There is no connection to maintain, no session to manage, no state to track. It is the postal service of data retrieval: you send a letter, you get a letter back, and neither party needs to remember the other exists between exchanges.

The problem is volume. The bot monitors hundreds of liquidity pools. Each pool has on-chain accounts whose data determines the pool's reserves, fee structure, and price curve. To keep the bot's internal model accurate, each of these accounts needs to be checked regularly. If I poll every account every second, that is hundreds of HTTP requests per second. If I poll every 500 milliseconds — which is closer to what competitive MEV requires — that number doubles. Each request costs compute units against the RPC provider's rate limit. Each request adds network round-trip latency. Each request competes with every other request for bandwidth and processing time.

It is like calling a friend every thirty seconds to ask "anything new?" Even if the friend is patient, even if they answer every time, the sheer volume of calls becomes absurd. Most of the time, nothing has changed. You call, they say "nope," you hang up, and thirty seconds later you call again. The information-to-effort ratio is abysmal. You are spending enormous resources to learn, most of the time, that nothing has happened.

WebSocket subscriptions solve this. Instead of calling every thirty seconds, you establish a persistent connection and say: "let me know when something changes." The server pushes updates to you as they happen. No wasted calls. No polling overhead. No rate limit pressure from hundreds of redundant requests. The data arrives exactly when it changes, which is exactly when the bot needs it. The information-to-effort ratio approaches perfection. Every message carries genuine new information.

This is the theory. The theory is elegant, efficient, and correct. The practice is a nightmare.

The Line That Keeps Going Dead

The WebSocket connection drops.

Not immediately. Not predictably. Not with a helpful error message explaining what went wrong. The connection establishes cleanly. Data flows. Pool states update in real time. The bot's internal model stays fresh. Everything works exactly as designed for a while — sometimes minutes, sometimes hours, sometimes most of a day. And then, without warning, the connection goes silent. No close frame. No error event. No graceful shutdown. The data simply stops arriving, and the bot continues operating on increasingly stale state, making decisions based on information that is drifting further from reality with every passing second.

It is like an apartment building intercom system that works perfectly for the first three months after installation. Visitors buzz, residents answer, doors open. Then one morning, a visitor buzzes apartment 4B and gets nothing. No answer, no static, no indication that the signal was received at all. The visitor tries again. Nothing. They try the building manager. Nothing. The intercom is not broken in any visible way — the buttons still light up, the speaker still looks functional, the wiring is still connected. It has simply decided, for reasons known only to itself, to stop carrying signals.

I implement reconnection logic. When the connection drops, the bot detects the silence, tears down the dead connection, and establishes a new one. The new connection works. Data flows again. The pool states refresh. The world is right for another few hours, and then the connection drops again. I reconnect again. It drops again. I reconnect. It drops. The cycle establishes itself with the regularity of a metronome, and each drop represents a window of time during which the bot is operating blind.

The reconnection logic itself introduces complexity. When a new connection is established, the bot needs to resubscribe to all the accounts it was monitoring. It needs to request a fresh snapshot of each account's state, because the state may have changed during the disconnection window. It needs to reconcile any subscription IDs that changed between the old and new connections. Each reconnection is not just a technical event — it is a small operational disruption that ripples through the entire data pipeline.

I am spending more time managing the WebSocket connection than I am spending on the actual arbitrage logic. The connection management code grows. Error handling branches multiply. Retry timers, backoff strategies, health checks, keepalive timers — the infrastructure required to maintain a "simple" persistent connection begins to rival the complexity of the trading logic it serves.

The Paradox of the Heartbeat

Every WebSocket tutorial, every library's documentation, every best-practice guide says the same thing: use heartbeats. A heartbeat is a periodic signal sent between the client and server to confirm that the connection is still alive. The client sends a PING frame, the server responds with a PONG frame, and both sides know the connection is healthy. If a PING goes unanswered, the client knows the connection has died and can reconnect immediately instead of waiting for the TCP timeout, which can take minutes.

The logic is sound. Heartbeats are the equivalent of texting someone to make sure they are still there. You send a message — "hey, still around?" — and they respond. If they do not respond, you know the conversation is over without waiting indefinitely for a reply that will never come. It is a simple, well-understood mechanism. The WebSocket RFC defines it. Libraries implement it. Documentation recommends it.

I enable heartbeats. I set the interval to 20 seconds — send a PING every 20 seconds, expect a PONG within that interval. This is a standard configuration. Conservative, even. The library handles the PING/PONG frames automatically. I do not need to write any protocol-level code. I just set heartbeat=20 and the library takes care of the rest.

The connection drops increase.

Not by a small margin. Not by a statistically ambiguous amount. The drops increase dramatically. Connections that previously lasted hours now last minutes. The reconnection logic, which was already working overtime, begins triggering so frequently that the bot spends more time reconnecting than processing data. The data pipeline, which was unreliable before, becomes essentially unusable.

I double-check my configuration. The heartbeat interval is reasonable. The library is a widely used, well-maintained package. The code is straightforward. Nothing is obviously wrong. I increase the heartbeat interval to 30 seconds — maybe 20 seconds is too aggressive. The drops continue. I try 60 seconds. Still dropping. I try 10 seconds, thinking maybe the connection is dying between heartbeats and I need to detect it faster. The drops get worse.

This is the moment that separates "following documentation" from "understanding systems." The documentation says to enable heartbeats. The heartbeats are making things worse. Both of these things are true. They are not contradictory, but understanding why they are not contradictory requires looking beyond what the documentation says and examining what the server actually does.

Texting Someone Who Never Replies

Here is what is actually happening.

The heartbeat mechanism works as follows: the client sends a PING frame to the server every N seconds. The client then waits for a PONG frame in response. If no PONG arrives within the expected window, the client concludes that the server is unresponsive and closes the connection. This is the correct behavior according to the WebSocket specification. The client is doing exactly what the RFC says it should do. The library is implementing the specification faithfully. The configuration is correct.

The server, however, does not respond to PING frames.

Not because the server is broken. Not because the connection is dead. Not because there is a network issue. The server simply does not implement PONG responses to client-initiated PINGs. The WebSocket RFC says servers MUST respond to PING frames with PONG frames. A certain provider's WebSocket endpoint does not do this. The server receives the PING, ignores it, and continues sending data as normal. From the server's perspective, everything is fine. Data is flowing, the connection is alive, there is no problem.

From the client's perspective, the situation looks very different. The client sends a PING. It waits for a PONG. No PONG arrives. The client, following its programming, concludes: the server is not responding, the connection must be dead. The client closes the connection and initiates reconnection. The reconnection succeeds. The client sends another PING. No PONG. The client closes the connection again. Reconnect. PING. Silence. Close. Reconnect.

It is exactly like texting someone who never replies. You send a message: "Hey, are you there?" No response. You wait a reasonable amount of time. Still nothing. So you conclude the person is ignoring you, or their phone is dead, or something has gone wrong. You hang up and try calling again. They answer. Great, the line is working. You text again: "Still there?" No response. So you hang up and call again.

The problem is not the connection. The connection is fine. The problem is that your method of checking whether the connection is fine is itself causing the disconnection. You are testing liveness with a mechanism the other party does not participate in, and interpreting the lack of participation as evidence of death. The patient is alive and well; you are pulling the plug because the heart monitor is unplugged.

This is the heartbeat paradox. Enabling the health check mechanism makes the system less healthy. The feature designed to improve reliability destroys reliability. The documentation's recommended configuration is the exact configuration that causes the problem.

Disabling the Safety Net

The fix is counterintuitive and slightly uncomfortable: disable heartbeats entirely. Set the heartbeat parameter to None. Do not send PINGs. Do not expect PONGs. Do not use the WebSocket protocol's built-in liveness detection mechanism at all.

This feels wrong. It feels like removing the smoke detector because it keeps going off. Surely the correct response to a malfunctioning safety system is to fix the safety system, not to remove it. And in many contexts, that intuition would be correct. But in this specific context, the safety system is malfunctioning because the environment it is designed for does not match the environment it is deployed in. The smoke detector is going off because it is installed in a kitchen, and kitchens produce smoke as part of normal operation. The correct response is not to fix the smoke detector — the smoke detector is working correctly. The correct response is to recognize that this particular kitchen needs a different approach to fire safety.

I set heartbeat=None. The connection stabilizes. Not partially, not somewhat — the drops that were occurring every few minutes stop occurring. Connections last for hours. Data flows continuously. The reconnection logic, which was firing constantly, goes quiet. The improvement is not incremental. It is categorical.

Of course, disabling heartbeats means I need an alternative mechanism for detecting genuinely dead connections. If the server actually does go down, or if a network partition actually does occur, I will not detect it through PING/PONG. I implement application-level liveness detection instead. If no data messages arrive for a configurable timeout period — significantly longer than the heartbeat interval I was using before — the bot treats the connection as potentially dead and reconnects. This is a coarser mechanism. It detects dead connections more slowly. But it does not cause false positives, which is the critical difference.

A FedEx "attempted delivery" notice is a familiar frustration for anyone who has ever waited for a package. You are home all day. You are sitting in the living room. The doorbell never rings. And yet, when you check the tracking, there it is: "Delivery attempted — recipient not available." They did not ring the bell. They did not knock. They walked up to the door, decided without evidence that nobody was home, stuck the notice on the door, and left. The system designed to deliver the package has instead created additional delay by falsely concluding that delivery is impossible. Heartbeats on a non-compliant server are the FedEx driver who never rings the bell. The solution is not to ring the bell harder. The solution is to stop using the bell and check for the package yourself.

The Same Protocol, Different Planets

The deeper lesson is about provider differences. The WebSocket protocol is standardized. RFC 6455 defines exactly how connections should be established, how frames should be formatted, how PING/PONG should work, how connections should be closed. Two compliant implementations should behave identically. The protocol is the protocol, the same way English is English regardless of who speaks it.

Except it is not. Different RPC providers implement the WebSocket protocol differently. Some respond to PINGs faithfully. Some ignore them. Some send their own server-initiated PINGs and expect client-initiated PONGs. Some have idle timeouts that close connections after periods of inactivity. Some have message rate limits that throttle subscriptions. Some support certain subscription methods that others do not. Some send subscription data in one format and others send it in a slightly different format. The protocol is the same. The behavior is not.

This is not unique to blockchain RPC providers. It is a general truth about protocols and implementations. HTTP is standardized, but web servers handle edge cases differently. SMTP is standardized, but email servers have different policies about message size, rate limiting, authentication, and header parsing. SQL is standardized, but every database engine has its own dialect, its own performance characteristics, its own interpretation of ambiguous specification language.

The protocol is a contract, but contracts have margins of interpretation. Two parties can agree to the same contract and still behave differently in the spaces between the explicit clauses. The WebSocket RFC says servers MUST respond to PINGs with PONGs. A certain provider does not. Is this a bug? A deliberate choice? An oversight? An optimization? It does not matter. The behavior is what it is, and the client must deal with the behavior as it exists, not as the specification says it should exist.

The Documentation Is a Map, Not the Territory

This experience changes how I approach documentation. Before this, I treat documentation as authoritative. If the docs say to enable heartbeats, I enable heartbeats. If the library's README shows a configuration parameter, I use that parameter with the recommended value. Documentation is the instruction manual, and instruction manuals are to be followed.

Now I understand that documentation describes intended behavior, not actual behavior. The WebSocket specification describes how a compliant server should handle PINGs. The library documentation describes how the client will behave when heartbeats are enabled. Both descriptions are accurate. Neither description tells you what will actually happen when this specific client connects to that specific server, because the interaction between the two depends on both implementations, not just one.

The map is not the territory. The menu is not the meal. The documentation is not the system. A map of Montana does not tell you about the pothole on Route 93 that appeared after last winter. The menu at a restaurant does not tell you that the kitchen is out of salmon tonight. And the WebSocket documentation does not tell you that a specific provider's endpoint ignores PING frames and will cause your heartbeat mechanism to destroy your own connection.

The only way to know what a system does is to observe the system. Not read about the system. Not reason about the system from first principles. Not extrapolate from what similar systems do. Observe this system, in this environment, with this configuration, connected to this server, under these conditions. The documentation is where you start. Observation is where you learn.

Building for Reality

I restructure the WebSocket management code around observed behavior rather than specified behavior. The connection configuration is provider-aware. Different providers get different heartbeat settings, different timeout values, different reconnection strategies. The code does not assume that all WebSocket servers behave the same way, because they do not. The abstraction layer that made all providers look identical was a leaky abstraction — it concealed differences that materially affected behavior.

The reconnection logic becomes more sophisticated, not by adding complexity but by removing false assumptions. The old logic assumed that a failed heartbeat meant a dead connection. The new logic recognizes that a failed heartbeat might mean a dead connection, or might mean a non-compliant server, or might mean temporary network congestion, or might mean any number of things that are not "the connection is dead." The new logic is more conservative about declaring connections dead and more aggressive about verifying liveness through actual data flow rather than protocol-level signals.

I add monitoring. Not just "is the connection open or closed" monitoring, but "when was the last data message received" monitoring. A connection can be technically open — the TCP socket is connected, the WebSocket handshake completed, the subscription is active — and still be effectively dead if no data is flowing. The monitoring tracks data flow, not connection state, because data flow is what the bot actually needs.

This is the voicemail problem. You call someone and it goes to voicemail. Is their phone off? Are they in a meeting? Are they screening your calls? Are they in an area with no signal? The voicemail tells you nothing about why they are not answering — it only tells you that they are not answering. If you need to reach them, you need a different strategy: text them, email them, try again in an hour, call their office line. The voicemail itself is not the solution. It is the signal that you need a different approach.

The WebSocket heartbeat is the voicemail. It tells you the other end is not responding to your specific query. It does not tell you whether the connection is alive, whether data is flowing, whether the system is functional. For that, you need to look at what actually matters: is the data arriving? Are pool states being updated? Is the bot's internal model staying fresh? These are the questions that determine whether the system is working, and they cannot be answered by sending PING frames.

What Real-Time Actually Costs

There is a broader lesson about real-time systems that I am absorbing through this experience. "Real-time" sounds like a feature. It sounds like the premium tier, the upgrade, the thing you want. And it is — real-time data is genuinely more valuable than delayed data for MEV arbitrage. But "real-time" is not a feature you turn on. It is a property you maintain, and maintaining it has costs that are not obvious from the outside.

Real-time data requires persistent connections, and persistent connections require management. They need to be established, monitored, reconnected, and reconciled. They need error handling for conditions that do not exist in request-response architectures. They need state tracking — which subscriptions are active, which accounts are being monitored, what was the last known state of each account before the connection dropped. They need graceful degradation — what does the bot do during the window between a connection drop and successful reconnection? Does it halt? Does it fall back to polling? Does it operate on stale data and accept the risk?

HTTP polling is like going to the grocery store. You leave the house, you buy what you need, you come home. It takes time, but each trip is self-contained. If the store is closed, you go to a different store. If your car breaks down, you take a cab. Each trip is independent of every other trip. The failure modes are simple and the recovery strategies are obvious.

WebSocket subscriptions are like having groceries delivered by a service with a standing order. When it works, it is far more efficient. Food appears at your door without you doing anything. But when the delivery service has a problem — the driver gets lost, the warehouse is out of stock, the app crashes, the account gets suspended — you have no food, and the failure mode is opaque. You do not know if the driver is five minutes away or if the entire service is down. You do not know if your order was processed or lost. You do not know if you should wait or go to the store yourself. The convenience of the working case comes at the cost of complexity in the failure case.

I do not regret choosing WebSockets. The efficiency gains are real and necessary. HTTP polling at the frequency MEV requires would exhaust rate limits and add intolerable latency. But I have stopped thinking of WebSockets as a simple upgrade over HTTP. They are a different category of infrastructure with different failure modes, different operational requirements, and different maintenance costs. The word "real-time" implies immediacy and simplicity. The reality is persistence and complexity.

Trust the Wire, Not the Manual

The heartbeat incident is a small thing, technically. One configuration parameter. One line of code changed from heartbeat=20 to heartbeat=None. The diff is trivial. But the lesson embedded in that trivial diff is one of the more important things I have learned in this project.

Documentation tells you how things are supposed to work. Observation tells you how things actually work. When these two sources agree, the documentation is useful. When they disagree, the observation is correct and the documentation is wrong — not wrong in the sense of containing false information, but wrong in the sense of being incomplete. The documentation describes the specification. The observation describes the implementation. And in software, the implementation is the truth. The specification is just the implementation's press release.

I catch myself now, in other parts of the codebase, asking the question differently. Not "what does the documentation say this function does?" but "what does this function actually do when I call it with these inputs?" Not "what should the API return according to the spec?" but "what does the API actually return when I make this request?" The shift is subtle but significant. It is the difference between trusting the map and trusting the terrain, and when the map says "flat road ahead" but you can see a hill with your own eyes, you gear down for the hill.

The WebSocket connection is stable now. Data flows. Pool states update in real time. The bot's internal model stays fresh. The solution was not more code, more complexity, more defensive programming. The solution was less — less heartbeat, less assumption, less trust in documentation over evidence. Sometimes the best fix is to stop doing the thing that is causing the problem, and the hardest part is realizing that the thing causing the problem is the thing you added specifically to prevent problems.