Solana mainnet halts for about five hours due to a bug in the program cache

Solana/Solana mainnet & Firedancer

On 6 February 2024, a bug in the program cache (LoadedPrograms JIT cache) caused validators on v1.17 software, representing over 95% of stake, to enter an endless recompile loop, halting consensus. Validators restarted the network with a patch that disabled deploys through the legacy loader, removing the bug's trigger.

• Bug: LoadedPrograms JIT cache inserted new output at sentinel slot height, making it invisible and triggering infinite recompilation.
• Duration: approximately 5 hours (09:53 UTC to 14:55 UTC on 6 February 2024).
• Severity: 95%+ of validators running vulnerable v1.17 software, making network-wide halt inevitable.
• Fix: backported changes to disable legacy 'v2' loader, removing preconditions that triggered the bug upon restart.