Solana halts for about 17 hours after a flood of bot transactions

Solana/Solana mainnet & Firedancer

Solana stopped producing blocks for 17 hours on 14 September 2021 after bots flooded the network with transactions during a token sale, causing validators to crash from memory overflow. Engineers worldwide coordinated a hard fork, restarting the network at 80% stake consensus with no funds lost.

• Bots generated hundreds of thousands of transactions per second during a Raydium IDO, overwhelming the forwarder queue and causing unbounded memory growth.
• Validators crashed and couldn't rejoin consensus on restart; a coordinated hard fork was needed to recover.
• Community achieved consensus and full recovery in under 24 hours with engineers working globally; no customer funds were lost.