The 30 September Solana blockchain outage was triggered by a fault in the blockchain’s code and a failing hot-spare node that generated duplicate blocks.
A hot-spare node is created when a validator operates a second online node that is intended to be utilized as a backup if the primary node fails. According to an update from the Solana Foundation, the backup node went online and was operating alongside the primary node. This caused the two nodes to send separate blocks to the network, resulting in parallel blocks.
This was handled properly for the first twenty-four hours, since the blockchain ultimately chose between the two alternative blocks, as it would for any tiny network split. At one time, though, a glitch in the blockchain’s programming prevented it from producing more blocks following one of these options.
Even though the right version of block 221 was validated, a fault in the fork selection algorithm prohibited block producers from building on top of 221 and prevent the cluster from reaching an agreement, according to Austin Federa, head of communications at the Solana Foundation.
As a consequence, the blockchain was down for around seven hours until validators agreed and deployed a code correction.