Building ChainShield in Public Taught Us Web3 Security Claims Need Replayable Evidence

Building ChainShield in public taught us that founders and engineers stop listening the moment a security claim cannot be tied to evidence they can inspect, replay, and challenge.

Thirty days of shipping in public stripped away a lot of comfortable language that still dominates Web3 security. "Audited" is too soft. "Monitored" is too vague. "Best practices" is too cheap. Serious buyers want to know what changed, who can still sign, which tests were run against live state, and what proof still survives hostile follow-up.

Establish the problem with technical depth

Most Web3 security marketing still collapses into badge logic. Which firm audited the contracts? How many findings were fixed? Was the multisig used? Those are not useless questions. They are just too weak for the kinds of failures that keep costing nine figures.

Bybit is the clearest recent example. In its own March 3, 2025 timeline, Bybit says a February 21, 2025 incident compromised one Ethereum cold wallet and caused $1.46 billion in losses. The company says attackers exploited the Safe wallet UI during a routine transfer, changed the smart contract logic behind the wallet, and drained the funds. A week later, the Safe Ecosystem Foundation said the forensic review concluded that a compromised Safe developer machine led to a disguised malicious transaction proposal, and that outside researchers did not find a vulnerability in the Safe smart contracts.

That distinction matters. The loss did not come from a cartoonishly obvious Solidity bug in a public repo. It came from a trusted approval path that stopped being trustworthy while still looking normal to human signers.

Radiant Capital exposed the same weakness from another angle. In its March 31, 2025 post-mortem on the October 16, 2024 exploit, Radiant says the breach caused approximately $50 million in losses. The team says attackers compromised core developers' devices and manipulated the Safe interface so it displayed legitimate transaction data while malicious transactions executed in the background. Again, the visible control looked normal right up until capital was gone.

For founders and VCs, that is a pricing problem. If your diligence still centers on whether a protocol "got audited," you are underwriting the wrong surface. For CTOs and Solidity engineers, the lesson is harsher: your security boundary is not the contract in isolation. It is the contract, the signer path, the upgrade path, the roles, the deployment flow, and the tests that tell you whether the live system still behaves as intended.

Building ChainShield in public made that gap impossible to ignore. The questions that held up were not brand questions. They were proof questions. Show the exact diff. Show who can still upgrade. Show how the release candidate was tested on realistic state. Show what monitor fires if authority changes after review. Public scrutiny did not reward confidence. It rewarded evidence that could survive replay.

The mechanism, the mistake, the misunderstanding

The mechanism is simple. Security claims decay faster than most teams think, because live systems change faster than assurance does. A clean review of commit A says less and less the moment commit B lands, a role changes, a signer workflow drifts, or an integration expands what the protocol is willing to trust.

That is why "replayable evidence" matters. By replayable evidence, we mean proof that another skeptical person can independently inspect, reproduce, and pressure-test. A PDF can be evidence. So can a diff review, a role map, a fork test pinned to a block, or an invariant harness that still passes against the current release candidate. A sentence in a pitch deck is not evidence.

Consider a perfectly ordinary privileged function:

function setRiskOracle(address newOracle) external onlyRole(RISK_ADMIN_ROLE) {
    oracle = IOracle(newOracle);
}

There may be nothing wrong with that line of code. The real question is what has to be true around it. Who can grant RISK_ADMIN_ROLE? How is the transaction decoded before signing? What test proves the new oracle does not break solvency assumptions on live state? What alert fires if the role changes outside the expected release window? The line of Solidity is often the cheapest part of the reasoning. The trust path around it is the expensive part.

NIST's Secure Software Development Framework says secure development practices need to be integrated into the software development life cycle, not stapled on afterward. It also says the framework gives buyers and suppliers a common vocabulary. That is exactly the problem public building forced into the open. Too many teams still sell security with artifact language while buyers increasingly need release language: scope, diff, authority, rollback, residual risk, and current proof.

The mistake is assuming public scrutiny rewards certainty. It does not. It rewards falsifiability. Once you build in public, weak claims get challenged immediately. "We have strong controls" turns into "which controls, on which path, and what proves they still hold after the last upgrade?" "We monitor production" turns into "what exactly is monitored, who gets paged, and what action can happen automatically?" The teams that answer cleanly are not the teams with the best slogans. They are the teams that can re-run their own claims.

The misunderstanding underneath it is that Web3 security buyers want more reassurance. Most sophisticated buyers want less reassurance and more evidence. They do not need another confidence artifact. They need a way to tell whether the current system still deserves confidence at all.

What good looks like

Good looks like treating every meaningful release as a security object, not just a product milestone. At minimum, a high-blast-radius change should travel with an exact commit, an explicit reviewed diff, a privilege-change summary, a deployment plan, and a rollback path. If that sounds like release engineering, that is because it is. User funds do not care whether the failure came from code review, operations, or approval flow.

Good also means maintaining a live authority map. OpenZeppelin's access control guidance is blunt about the principle of least privilege: every component should be limited to what it truly needs to do. In practice, that means founders should be able to answer who can upgrade, pause, mint, reconfigure, or reroute value today, not who was supposed to have that power two audits ago. Engineers should make role grants and revocations visible enough that a reviewer can tell when the control plane changed before the exploit does.

Upgrades need to be treated like dangerous state migrations, not harmless housekeeping. OpenZeppelin's upgradeability documentation warns that breaking storage layout rules can mix up stored values and lead to critical errors. That is the kind of warning too many teams read as framework trivia. It is not trivia. It is a reminder that a protocol can ship a catastrophic change while every dashboard still says the deployment succeeded.

Testing has to become more reproducible too. Foundry's invariant testing guide defines invariants as properties that should always hold regardless of the sequence of actions taken, and it checks them after randomized call sequences. Its fork testing guide makes the operational side even clearer: run tests against real chain state and pin block numbers for reproducibility. That combination is the standard more teams should aim for. If a security claim depends on behavior under live balances, live integrations, or live liquidity conditions, the team should be able to replay that condition before shipping.

Finally, the signer path needs independent verification. Bybit and Radiant should have ended the era of trusting one interface, one simulation surface, or one operator explanation. High-impact transactions need out-of-band decoding, role separation between proposers and approvers, and enough ceremony that a malicious payload cannot borrow legitimacy from a familiar UI. Otherwise the multisig threshold is theater with better branding.

ChainShield's angle

Building ChainShield in public changed our view of the product category because it changed the standard we were willing to defend in front of skeptics. We are not trying to help teams sound safer. We are trying to help them produce current proof that survives the next question.

That means compressing review lag between change and scrutiny. It means keeping security evidence attached to the current diff, the current permissions, the current signer path, and the current release candidate. It means making it normal for a founder to answer diligence with a live change packet instead of an old PDF, and normal for an engineer to defend a release with replayable tests instead of general confidence.

Founders should ask for security evidence they can challenge, not just security language they can repeat. CTOs should ship proof packets, not posture. VCs should underwrite change control, not historical branding. Thirty days of building ChainShield in public taught us that the trust premium in Web3 will go to teams whose claims can be re-run after the next change, not teams whose claims merely sounded convincing before it.