What the world doesn’t need right now is another CrowdStrike hot take… so here’s mine.
I’m not here to throw any more at CrowdStrike. They’ve had enough. There was a mistake with big impact.1 It happens.
There is a lesson to be learnt though: every security control carries a tradeoff, and in this case it’s operational risk.
We cannot just blindly push security controls; there are tradeoffs which must be understood.
If you deploy an Endpoint Detection and Response (EDR) solution in production and sit on the bleeding edge of content updates, you will see new threats sooner. You are also accepting an internet connected privileged binary operated by a third-party, and with it their bugs, false positives and supply chain compromise.
Remember this was a content update, not a sensor software update issue, therefore the n-1 strategy didn’t apply here. Because EDR content updates are typically pushed near real-time to maintain threat protection, the standard deployment safeguards (staging, canary deployments, where you test updates on 5-10% of systems before full rollout) that would catch a buggy software release simply weren’t in play.
These EDR vendors are also balancing their service between the two; their reputation suffers whether they miss a critical threat or push a buggy update.
The real-world impact of this tradeoff became immediately visible in how organisations responded. A week ago all these impacted companies2 would have boasted about their strong security controls, and flown through customer/vendor security due diligence reviews. In the immediate aftermath, third-party risk teams rapidly scrambled to identify their exposure internally and across materially significant vendors, viewing such dialled up security controls less favourably.
This isn’t about CrowdStrike. This has happened before with other security providers and will happen again. I’m not saying whether you should or shouldn’t be at the forefront of threat management updates. Whatever you decide, ensure you understand the risk. What security tooling do you have deployed, and do you understand its security vs operational risk?
On 19th July 2024, CrowdStrike released a routine content configuration update for their Falcon Windows sensor. A defect in this update caused system crashes (BSOD) for Windows hosts that pulled the update between 04:09 and 05:27 UTC. The issue was identified and reverted at 05:27 UTC, but recovery required manual remediation on affected systems. ↩︎
The outage impacted approximately 8.5 million Windows devices globally. Major organisations affected included airlines (Delta, United, American Airlines with thousands of flight cancellations), healthcare providers (mass appointment cancellations and delayed procedures), banks and financial services (transaction processing disruptions), emergency services (911 systems in multiple US states), and broadcasters (Sky News went off-air). The manual remediation requirement meant recovery took days for many organisations. ↩︎