The Takeaway:
- The bulk of Ethereum’s DeFi ecosystem went dark earlier today after a latent bug in the Ethereum code split the network’s transaction history in two.
- The split resulted from a code change that was surreptitiously inserted into a previous Geth update; some Ethereum node operators ignored the update, which ironically was meant to prevent the very split that occurred.
- The nodes that did not upgrade were under the impression the update was minor and did not know it included a change to Ethereum’s consensus design.
- A post-mortem released today by Geth indicates that the bug was intentionally triggered. The case is perhaps Ethereum’s greatest challenge since the 2016 DAO fork, and it raises questions about Ethereum’s oft-touted decentralization and the effectiveness of its developer coordination going into Ethereum 2.0.
At first, it was an apparent issue with Infura, the ConsenSys-run servers that keep a majority of decentralized finance (DeFi) applications synced to the Ethereum network.
Infura went down around 8:00 UTC Wednesday, and with it, some of Ethereum’s most popular applications like Metamask, MakerDAO, Uniswap, Compound and MyCrypto, among others. Shortly after, Binance halted Ethereum trading after noticing conflicting transactions on its Ethereum node. As other exchanges suspended trading as well, the real issue became clear: A bug in the Go Ethereum (Geth) client, whose code underpins 80% of Ethereum’s applications, had split the Ethereum blockchain in two.
The two conflicting transaction histories meant Etheruem users were temporarily interacting with different versions of the Ethereum blockchain. More than causing delays, this put user funds at risk by knocking out the majority of Ethereum’s DeFi applications for a few hours.
Infura has fixed the issue, as have other service providers who were affected by the snafu, by updating their nodes. These stakeholders were running an older version of Geth, which contained a bug that Ethereum developers silently patched in recent update – an update which Infura and Blockchair, among others, ignored.
Besides these two service providers, other Ethereum users and wallet providers were also affected because they didn’t update their code, developers have told CoinDesk.
The fiasco has critics challenging Ethereum’s perceived decentralization, while stakeholders are wondering why the change was pushed in secret without coordination between Geth and other development teams.
To some, the split is the most pressing challenge for Ethereum since the infamous DAO hack of 2016.
Ethereum’s chain split: How it happened
In a just-published post-mortem, Péter Szilágyi, a team lead at Ethereum, wrote that a fork “was (deliberately) triggered on the Ethereum network.”
A representative from Optimism, an Ethereum scaling project, recently posted that the project purposefully triggered the bug the Geth update fixed, setting off the fork.
When the fork occurred, it created two versions of the Ethereum transaction ledger: one with transactions from updated Geth clients, and one with transactions from older Geth clients (like Infura).
“The fix was deployed several months ago and only today a transaction that caused that split came in,” Nikitia Zhavoronkov, the lead developer at Blockchair, an Ethereum block explorer who was affected by the fork, told CoinDesk in a direct message.
Thinking the update was “a minor change to the code,” Blockchair didn’t bother with the update because it wouldn’t be worth the downtime for their services. But more than minor, developers apparently made a quiet change to Geth’s consensus mechanism in the update, as well.
“The Geth team indeed changed the consensus implementation in the v1.9.17 release, however the team did not create any new rules that the Ethereum community didn’t know about or agree to,” Szilágyi writes in the post, saying these rules were laid out in an Ethereum Improvement Proposal three years ago.
“If you don’t consider accidentally introducing a bug a ‘consensus upgrade,’ then you should also not consider fixing the said bug a few months later a ‘consensus upgrade’,” he argued.
A call for transparency
Ironically, the fork was caused by the very consensus bug that Geth’s update sought to address.
The Ethereum bounty program recently recognized John Yang, a newcomer to Ethereum’s open-source community, for discovering this and another vulnerability. Geth developer and Ethereum security expert Martin Swende tweeted the changes in the update fix the disclosed issues, intimating that the debacle is a “reminder to keep your node(s) up to date!”
Swende continues to say in the tweet thread that developers did not announce the big change to avoid drawing attention to the flaw. In his own explanation, Szilágyi said that “silently” fixing the bug invited less “disruption.”
Still, other Ethereum stakeholders are wondering why the bug could not have been disclosed in private with the teams that are building on Geth.
“Each major project that the dev team is in close contact with should have a security contact that can help manage and coordinate a smooth upgrade, and we should work together,” Matt Luongo, the founder of Thesis, told CoinDesk.
“When forks are surprises, anyone who has built atop Ethereum like we have could lose money,” he continued.
Thesis builds the Keep Network, which issues tBTC, a form of tokenized bitcoin for the Ethereum blockchain. Luongo said the fork put tBTC users’ funds at risk, but not because of the chain split, which has been resolved after Infura and others updated their Geth clients.
It’s because the downtime meant that users staking Ethereum in Keep Network couldn’t coordinate with the Ethereum mainchain; as a result, they risked having part of their stakes “slashed” for not meeting their fiduciary requirements.
Despite the problems the split caused, prices for ether, the Ethereum blockchain’s native cryptocurrency, rose 4.6% Wednesday after the news emerged, suggesting that traders see little systemic or long-term threat from the snafu.
Picking up the pieces
Zhavoronkov said that the mess-up was not malicious, but that “if [Geth] knew such thing could happen, they should’ve prepared a guide for node operators.” Luongo shared similar frustrations, saying that the Geth team are “good developers” but that they lack “experience running infrastructure” and are “underfunded.”
The comments key in on a frustration some Ethereum stakeholders share pertaining to why Geth kept the consensus change a secret. Going further, why did Infura, the backbone of Ethereum’s decentralized finance ecosystem, among others, not know about a consensus-breaking bug in Ethereum’s code before it was triggered?
“This is a bit of a grey area and requires a case-by-case discussion,” Szilágyi explains in his post. “We all agree that transparency is king and that we should strive as much as possible towards it, but it’s also important to look at all the details before heads start rolling.
“In the case of Ethereum, it takes a lot of time (weeks, months) to get node operators to update even to a scheduled hard fork. Highlighting that a release contains important consensus or DoS fixes always runs the risk of someone trying to beat updaters to the punch line and taking the network down. Security via obscurity is definitely not something to aim for, but delaying a potential attack by enough to get most node operators immune may be worth the temporary ‘hit’ to transparency,” he continued.
Ultimately, Geth’s team believed there was too much risk in disclosing the vulnerability, so they decided that pushing the update surreptitiously invited the least risk.
“We’d argue that it actually did work,” Szilágyi says. Even though the update “took an unexpected turn with yesterday’s network split,” Geth’s team still believes that keeping the issue hush “was the right call”
As Ethereum approaches its largest upgrade ever in Eth 2.0, the case could be a critical study in client coordination for the Ethereum ecosystem.
“The most important thing here IMO is that the folks who made this call are transparent about the reasoning, own any mistakes, and grow,” Luongo said. “Monero has dealt with [consensus bugs] well in the past, as has Bitcoin and Zcash. There are many examples, and while it’s always tricky to coordinate across an industry, eschewing any sort of coordination is extremely dangerous.
“I hope this fork leads to tighter relationships and rethinking how projects on Ethereum interact with client development.”
Update 00:36 UTC – This article has been updated to better convey that the “hard fork” was not a deliberate code change, but a chain split which was triggered by the consensus bug that Ethereum developers quietly patched in the Geth v1.9.17 release.