Lessons from CrowdStrike’s BSOD
July 19th, 2024 was a day many of us won't forget. CrowdStrike, a major cybersecurity player, accidentally unleashed an update to its Falcon Sensor software that sent Windows machines worldwide into a tailspin of blue screens. Banking froze, flights got grounded, and the dreaded BSOD was back in the spotlight. We even got some good memes out of it, like the Southwest Airlines Windows 3.1 throwback (though, spoiler alert, it was a joke).
Fast forward two weeks, and the digital dust has mostly settled. CrowdStrike came clean on July 24th: a faulty Rapid Response Content update slipped through their testing, causing a nasty memory error and the infamous crash. Kudos to them for owning up to it. As a former developer, I know how tough testing can be under the gun. I'd chalk it up to an honest mistake, not malice.
But let's be real, this snafu cost businesses a pretty penny. And let's face it, ditching Microsoft isn't exactly an option for most. So, what can we learn from this mess?
Lesson 1: Purpose-Built Appliances Are Your Friends
This whole debacle highlights why critical services, like DNS, belong on their own dedicated platform. We've been moving in this direction for decades, so this is nothing new. The only thing that has changed is the definition of “critical services”. In 2024, DNS definitely falls into this category and needs to be treated as such. This means, you should not be running DNS on your Microsoft-based Domain Controllers.
"But wait," you say, "Active Directory needs Microsoft DNS!" Not true, my friend. You can absolutely run DNS on a non-Microsoft platform and still have AD humming along. I've done it myself many times, and I hope to write about it in a future post. Don’t believe me? Even the DNS guru Cricket Liu agrees. (Check out his thoughts here: https://blogs.infoblox.com/company/lessons-from-the-global-windows-outage/)
Lesson 2: Automation is Still the Way
Don't let anyone tell you this incident proves automation or automatic update is bad. Nope. Automation is consistent, and it's not any less secure than a human who's had too much coffee. Once you squash a bug in your automated process, it's gone for good. Plus, it frees up your team to focus on more strategic tasks, like, you know, preventing the next global tech meltdown. 😉
Lesson 3: Prepare for the Worst (Because It Will Happen)
No matter how many safeguards we put in place, stuff happens. And guess what? The bad guys wasted no time exploiting this chaos. In just a few days, over 194 malicious domains popped up, trying to trick frazzled IT pros and unsuspecting users. Below are just a few of the domain names detected:
fixcrowdstrike[.]com
crowdstrike[.]fail
crowdstrikebsod[.]co
crowdstrikebsod[.]com
crowdstrike-bsod[.]co
As my colleague Craig Sanderson said so well in his post on this subject: never let a good crisis go to waste.
The Takeaway
So, let's treat DNS like the critical service it is and give it its own secure home. Embrace automation (even if it's a little scary at first), and for goodness sake, get yourself some solid Protective DNS (PDNS) to shield you from those opportunistic bad actors.
Even the most agile falcon can hit turbulence. But with the right tools and a little foresight, we can all navigate those stormy skies, minimize the damage, and get back to SOARing in no time.