After Snowflake and AT&T: Make Identity the Perimeter
July 15, 2024Windows Downdate: When Patched Means Nothing If You Can Be Downgraded
August 8, 2024On July 19, 2024, a faulty content update to CrowdStrike’s Falcon sensor caused millions of Windows machines worldwide to crash into a boot loop. Airlines, hospitals, banks, and government services were disrupted, including many here in Canada. Notably, this was not a cyberattack. It was a defective update to a widely deployed security agent, and the effect was indistinguishable from one.
The uncomfortable lesson
A tool installed to protect endpoints became the cause of a mass outage. Recovery was slow because the fix often required hands-on access to each affected machine, and encrypted disks complicated the process further.
What this teaches incident responders
- Treat security agents as critical software. Anything with kernel-level reach can take a system down as effectively as malware.
- Stage updates where you can. Ring-based rollouts limit blast radius when something goes wrong.
- Keep an offline runbook. Your incident plan is useless if it only exists on the systems that just crashed.
Takeaway: Not every crisis is an attack, but your response plan should handle both. Build recovery procedures for the failure of your own security stack, and keep recovery keys and runbooks reachable offline.