Cross-Validating With On-Chain Data: When Code Lies, Bytes Don't

When the Code and the Chain Disagree

I keep finding the same bug, and it keeps embarrassing me in the same way. I read my Rust source, I read the IDL, I read the comments I wrote two weeks ago, and everything looks fine. Then I pull the account bytes off-chain, line them up against the struct I think is there, and one of three things turns out to be true: the layout shifted, the deployed program isn't the version I think it is, or the account I'm reading isn't owned by the program I assumed owned it.

This is the lesson the chain keeps trying to teach me. Source code shows intention. On-chain data shows reality. When they disagree, intention loses. The bytes win every single time, and they don't care how many hours I spent writing the documentation.

So I've been forcing myself to build a real cross-validation habit — treating the on-chain state as the source of truth and the source code as a hypothesis. Today's notes are about how I do it, what tools I lean on, and the specific moments where this discipline has saved me from shipping a confidently broken bot.

Why Solana Makes This Worse Than You'd Expect

On most general-purpose runtimes, the state lives with the program. You look at one binary, you mostly know what it does. Solana is different. As an infrastructure provider puts it in their security guide, "Programs are stateless — they interact with data stored in other accounts, passed by reference during transactions". The program is a pure function. The state is a separate pile of accounts that the runtime hands it at call time.

Which means: at compile time, there is no guarantee the account you receive contains the data you expect. "Any account can be passed into a program," the same guide warns, "and a malicious actor could pass accounts with arbitrary or crafted data instead of legitimate accounts." The runtime won't catch the mismatch for you. Your program either verifies the bytes itself, or it gets fooled.

This isn't a theoretical problem. It's the daily friction of working on Solana. Every interaction looks like a courier service where the package, the address, and the contents all need to be verified at the door — because anybody can drop off anything. Calling on-chain data "the source of truth" isn't a slogan. It's the only thing that's actually true at runtime.

The upside is that the truth is fully public. Chainlink puts the principle nicely: "On-chain verification is the most transparent method, where every step of the data validation process is recorded on the distributed ledger, offering maximum auditability as anyone can verify the history of the data inputs." If I'm willing to do the work, every byte of state is sitting there waiting for me to inspect it.

The Three Questions I Ask Every Account

When I'm trying to verify an account, I've trained myself to ask three boring questions in order. Skipping any one of them is how I end up debugging a phantom problem for an afternoon.

1. Who owns this account? The owner field on getAccountInfo tells me which program the account belongs to. If I think I'm looking at a Token Program account but the owner is some unrelated program, the rest of my analysis is garbage. This is the single check that catches the most embarrassing mistakes.

2. How big is it? The space field tells me the byte length. SPL Token accounts are exactly 165 bytes per the Solana Cookbook. Anchor accounts start with an 8-byte discriminator, then the serialized fields. If the size is off by even one byte from what my struct claims, I'm decoding the wrong type and I need to stop before I trust anything.

3. Does the data match the layout I think it does? This is where Borsh comes in, and where most of the real work happens. The official Solana docs spell out the response shape for getAccountInfo: data, executable, lamports, owner, rentEpoch, space. Everything I care about flows through those fields.

An infrastructure provider's developer introduction explains the encoding clearly: "Solana employs Borsh (Binary Object Representation Serializer for Hashing) for data encoding," and "one of the key advantages of Borsh is its determinism, ensuring consistent serialized output for the same input." That determinism is what lets me decode bytes by hand if I have to. Given the same struct definition, the bytes are always laid out the same way. There's no padding magic, no compiler-version drift, no Sunday afternoon surprises.

Reading Bytes Like a Mechanic Reads a VIN

I grew up watching my uncle decode car VIN numbers at his shop. Every digit meant something specific — model year, plant, engine code. He didn't memorize the whole thing. He just had a chart taped to the wall and ran the number against it field by field. That's exactly how I read Solana account data now.

The blog post at blog.chalda.cz has the clearest worked example I've seen. For a small Anchor account called SimpleAccount with an admin pubkey and a counter, the layout is:

• Bytes 0–7: discriminator (sha256 hash of the Rust identifier, first 8 bytes)
• Bytes 8–39: admin Pubkey (32 bytes)
• Bytes 40–47: counter as u64, little-endian (8 bytes)

Total: 48 bytes. If the account I fetch is 48 bytes, with the right first 8, and the next 32 bytes decode to the admin pubkey I expect, I'm looking at the real thing. If any of those pieces is off, I have a problem I need to understand before going any further.

The chalda.cz post is explicit about this: "Anchor uses first 8 bytes of the data for the discriminator (sha256 hash of the account's Rust identifier)." That sentence is doing a lot of work. It means I can detect the wrong account type at the byte level — I don't need to deserialize and pray. If the first 8 bytes don't match the discriminator I computed from the struct name, I bail out immediately.

For types: u64 is 8 bytes little-endian. Pubkey is 32 bytes. String is a 4-byte length prefix followed by UTF-8 bytes. SPL Token accounts (which use bincode, not Anchor) put the mint at offset 0, the owner at offset 32, and the amount as a u64 starting at offset 64. Memorize those four facts and 80% of the verification work gets easier.

getProgramAccounts: The Scan-and-Filter Workhorse

Individual account fetches are great when I already know the address. The harder problem is "give me every account this program is tracking." That's what getProgramAccounts is for, and it's a tool that punishes carelessness.

The Solana Cookbook guide is upfront that this method "require[s] RPC nodes to scan large sets of data" that are "both memory and resource intensive." Translation: if I send it without filters, I'm asking the RPC node to dump everything, and the connection will often time out before I get anything useful back.

The filters are the trick. Up to four filter objects, all ANDed together. The two that actually matter are dataSize and memcmp:

• dataSize: narrows results to accounts of an exact byte length. "Narrow[s] the scope of our query to just accounts that are exactly 165 bytes in length," as the Cookbook describes the SPL Token filter.
• memcmp: matches a base58-encoded byte sequence at a specific offset. Up to 128 bytes of comparison value. Exact match only — no greater-than, no fuzzy match.

The Cookbook is blunt about why this matters: "Effective use of memcmp requires understanding the byte layout of the account data you are querying." In other words, the filter is only as good as my mental model of the bytes. If I don't know that token owner sits at offset 32, I can't write the filter that finds "all token accounts owned by this wallet." An infrastructure provider's getProgramAccounts documentation confirms those offsets — offset 0 for mint, offset 32 for owner — and I keep them in a sticky note next to my monitor.

The pattern I default to: dataSize first to shrink the candidate set, then memcmp on the field I actually care about. That order matters because it lets the RPC node short-circuit cheap rejections before doing the more expensive byte comparison.

When the Result Is Too Big

There's no built-in pagination. Quoting the Cookbook again: getProgramAccounts "does not support pagination," and oversized responses get truncated. For programs with tens of thousands of accounts, this is a real problem.

The workaround I use is the two-phase pattern the Cookbook recommends: first, fetch just the pubkeys with dataSlice set to length zero, so the response is cheap. Then, take that list of pubkeys and batch them through getMultipleAccounts to pull the full data in manageable chunks. It's two round trips instead of one, but it's the difference between getting an answer and getting a timeout.

The dataSlice parameter is one of those features that doesn't look important until you've been burned by oversized responses. It "will limit the amount of data for each account" without changing how many results come back. Counting accounts? Use dataSlice: { offset: 0, length: 0 } plus the filter you care about. The RPC node still scans, but it returns almost nothing per match.

Explorers Aren't Optional, But Pick Carefully

I don't trust myself to decode every byte by hand. I lean on explorers for a sanity check — does the human-readable view from a third party agree with what my code is parsing? When the two disagree, one of us is wrong, and that's worth knowing before I trade real money on it.

The Metaplex explorer guide gives a useful tour of the landscape. The canonical Solana Explorer, maintained by Solana Foundation, is "the canonically named Solana Explorer" and the most authoritative. Instruction breakdowns, account data viewer, the feature-gates view with SIMD activation epochs — it's the boring, dependable option.

Solscan, which an infrastructure writeup notes "was acquired by Etherscan in early 2024," became dominant because, in their phrasing, "Solana's on-chain data was notoriously difficult to read in the network's early days." It's still the explorer I open first when I want a portfolio-style view of a wallet's token holdings or DeFi positions.

SolanaFM, also per the Metaplex guide, was "acquired by Jupiter, the Solana DEX aggregator, in September 2024; however, today, the explorer is largely unmaintained." Useful historically, less so as a daily tool.

Orb, launched in late October 2025 per the Metaplex docs, is the newcomer I keep returning to when I need archival speed. The guide claims "Orb's archival RPC calls are 2-10x faster than Google BigTable queries," which matches my experience pulling deep transaction histories. Verified IDLs and inner-instruction inspection are the features that actually move the needle for me when I'm debugging CPI chains.

The meta-lesson: don't pick a favorite. Cross-check between two explorers when the answer matters. If Solscan and the official Explorer both render the same fields the same way, my own decoder is probably right. If they disagree with each other, I have a layout question to resolve before I trust anything.

Simulation: The Free Dress Rehearsal

The single highest-leverage habit I've picked up is using simulateTransaction before sending anything I'm not 100% sure about. The HashBlock post on debugging production failures describes it well: simulation "runs the entire transaction as if it were real — but without committing anything to the chain." No fees, no state changes, full log output.

For a bot that pays priority fees per attempt, simulation is the equivalent of a coach calling a play in chalk before sending the team onto the field. I see exactly which instruction failed, which compute unit budget got blown, and what custom error code the program returned. Then I fix the actual problem instead of firing live shots in the dark.

The logs themselves are worth knowing how to read. The patterns are predictable:

• invoke [1] — a top-level instruction starts executing.
• invoke [2] — a CPI was made from inside another instruction.
• Program log: ... — output from a msg!() macro in the Rust source.
• consumed X of Y compute units — actual versus allocated CU usage.
• success or failed: ... — the verdict for that instruction.

When something blows up at invoke [2], I know the failure happened inside a CPI, not in my top-level handler. That alone narrows the search dramatically. When I see consumed 199,800 of 200,000 compute units followed by a failure, I know I need a higher CU limit, not a code fix.

For Anchor programs, an error like InstructionError(0, Custom(6000)) is decodable straight from the IDL. The HashBlock post shows the pattern: look up code 6000 in program.idl.errors, and the IDL tells me the name and human message. That's the difference between "something broke" and "the program rejected my unauthorized signer."

The Cross-Validation Loop I Actually Run

Here's the workflow I've settled on after enough wasted afternoons:

1. Pull the raw account. getAccountInfo with base64 encoding. Confirm owner is the program I think it is. Confirm space matches my struct. If either is wrong, stop.
2. Decode the bytes against the struct. If it's Anchor, first 8 bytes are the discriminator — verify them. Then walk the fields in order: Pubkeys as 32 bytes, u64s as 8 bytes little-endian, strings as length-prefixed UTF-8.
3. Open the same account in two explorers. If both render the same fields with the same values, my decoder is right. If they don't match what I parsed, I have a layout bug.
4. Run getProgramAccounts to cross-check the population. With dataSize filtering, do I get the number of accounts I expect? If the program should have N pools and the count comes back wildly different, something is misregistered.
5. Simulate before sending. Build the transaction I want to send, run simulateTransaction, read the logs. If simulation fails, the live send will fail too — fix it now.
6. After sending, re-read. Pull the affected accounts again and verify the state changed the way I expected. CPI chains can leave subtle differences between what I think happened and what actually happened.

An infrastructure provider's security guide lists a similar checklist for program-side verification: confirm ownership, deserialize safely, validate values, verify signers, cross-reference keys. The point is the same on both sides of the call: don't trust anything without checking.

The Habits That Make This Cheap

Doing this manually every time would burn out anyone. The habits that make it sustainable:

Keep an offset cheat sheet. Mint at offset 0. Owner at offset 32. Amount at offset 64. Anchor discriminator at offset 0–7. After a while these are muscle memory, but writing them down once means I never have to look them up at 2 a.m.

Wrap getAccountInfo in a thin verifier. Every fetch should also assert the owner and the size. If those don't match expectations, the call returns an error, not a half-parsed struct. I'd rather get an explicit failure than a silent off-by-one.

Use jsonParsed encoding when there's a known parser. An infrastructure provider's developer introduction highlights that for token accounts specifically, "jsonParsed encoding is especially useful for token accounts, as it returns the mint, owner, balance, delegate, and state without manual deserialization." For programs with no parser, it falls back gracefully to base64.

Reload after CPI. Anchor's reload() method is there for a reason. State can become stale after a cross-program call, and using cached values past that point is exactly the kind of bug that survives staging and dies in production.

Treat failing logs as evidence, not failure. A clean log from a failed simulation is more useful than a successful transaction I don't fully understand. The error tells me what the program actually thinks is wrong.

Where This Discipline Has Saved Me

A few moments stand out where the on-chain cross-check caught something my source code wouldn't have.

One case: I was decoding a DEX pool account against a layout I'd copied from a tutorial. The numbers looked right in development. When I cross-checked against Solscan's decoded view, two fields were swapped — a fee numerator I was reading as a reserve. The bot would have under-quoted profitability on every cycle through that pool. The source code wasn't lying. The tutorial layout was just stale.

Another case: a getProgramAccounts call returning fewer accounts than I expected. The dataSize filter was correct, but the memcmp filter targeted the wrong offset because the account layout had an extra padding field I hadn't accounted for. Pulling one account by hand and walking the bytes one at a time revealed the misalignment. The fix was four characters in the memcmp offset.

A third: an Anchor program where the deployed binary was an older version than my local source. The discriminator matched (same account name), but a field that should have been a u64 was still a u32 on-chain. My parser was reading past the end of the field and into the next one, producing nonsense values. The cross-check that caught it was simply that the byte total didn't add up — the deployed account was smaller than my new struct claimed it should be.

In all three, the source code looked correct. The bytes didn't. The bytes were right.

A Note On Performance vs. Paranoia

There's a tension between "verify everything" and "trade fast enough to matter." For an MEV bot, every microsecond of verification is microseconds the competition isn't spending. I've come to think about it the way Chainlink frames the broader architecture: "A common approach is to use the blockchain as the source of truth for key data and to use off-chain systems to perform more complex or resource-intensive operations based on that data."

The hot path doesn't have to re-verify everything from scratch every loop. What it has to do is start from a state that was rigorously cross-validated, and detect when assumptions break. Static facts — account layouts, program IDs, expected sizes — can be verified once at startup and asserted on every read. Dynamic facts — current reserves, current ownership — get pulled fresh, but the parsing path is the same code that passed cross-validation against the explorer view.

The discipline isn't "check everything every time." It's "build a parsing layer that you've already proven matches the chain, then trust it within the bounds you verified." When the bounds change — a new program version deploys, a new account type appears — the cross-validation kicks back in.

What I'm Carrying Forward

The core posture I'm trying to keep: source code is a hypothesis until on-chain data agrees. The chain is the only thing that's actually running, and treating it as the authority isn't paranoia — it's the only stance that survives contact with reality. Anchor's wrappers handle a lot of this automatically, but the moment I'm decoding something Anchor doesn't know about, the discipline has to come from me.

The verification habit pays for itself within a week. Every time the on-chain bytes contradict my expectation, I learn something I didn't know about the program I'm interacting with. Every time they match, I get to trust the next layer of code I write on top. That trust is the actual product. The bytes are how it gets earned.

Key Takeaways

• Source code shows intent; on-chain data shows reality. When they disagree, the bytes are authoritative. Build the habit of cross-checking before trusting.
• Three questions for every account: who owns it (owner field), how big is it (space field), and do the bytes match the struct layout (Borsh decode against expected fields).
• getProgramAccounts is powerful but expensive. Always filter with dataSize first, then memcmp. No pagination — use the two-phase pubkey-then-getMultipleAccounts pattern for large datasets.
• Use multiple explorers as a sanity check. When two independent views agree with each other and with your decoder, you can trust the parsing layer. When they disagree, you have a layout bug to find before trading.
• Simulate before sending. Transaction simulation is a free dress rehearsal — full logs, no fees. If simulation fails, the live send will fail too. Fix the problem there, not in production.