CrowdStrike Update Causing Windows Crashes

This is a huge failure on CrowdStrike's part, akin to SolarWinds failure to protect their update process, albeit right now being attributed to their own error not to an attack. The end result for customers and the customers of customers is the same. Needing to slow-roll updates to security products because of quality concerns is like widening the bullseye for attackers.

John Pescatore

The flawed update is impacting the Windows login service and may result in our old friend BSOD. The fix is to get the updated definition from CrowdStrike, which is problematic if your systems are crashed. As my wife and I sit here waiting for our flight we can't help but notice that 600 flights were delayed or cancelled. Banks are reporting issues processing transactions. Interesting supply chain compromise exercise. Take note of downstream recovery impacts which should be incorporated into your BC/DR plan.

Lee Neely

This is not a quick fix. I can promise that, from talking to a few people working with Azure VMs, the fix will be pretty bad and fairly manual. I know of just one company with half its infrastructure (computers and servers) down; it will take days to recover. A couple of thoughts here: One is that now attackers know who is using what EDR, which will be bad. Two, how many of these people will be giving out the Local Administrator password so that you can get into a Windows Recovery remotely? At this point, you are very exposed and are trusting people not to do anything malicious. It isn't good. Be kind to people.

Moses Frost

The events of today highlight the importance of regulations such as the EU NIS2 Directive and EU DORA in ensuring organisations are taking the appropriate steps to manage cyber risk within their own organisations and just as importantly within their supply chain. While CrowdStrike have issued workarounds and fixes to the issue, in many cases it requires manual intervention to each individually impacted device which could lead to a long recovery time from this problem. Organisations will need to prioritise the systems that are most critical to their business and recover them in order of priority. Questions need to be asked of CrowdStrike as to what went wrong with their testing and quality assurance processes to ensure there was no impact on their customers and what they are going to do to ensure there is no repeat of today's issue.

Brian Honan

Single point of failure? When I was a development manager on an early 5000 user multi-application system, the rule was "If it ran yesterday, it must run today." This was about fall-back procedures that had to be built into every change. This is the second major outage in a year caused by a change.

William Hugh Murray

It might make sense to consider this a friendly ransomware incident and revisit roads to resilience in the face of compromise. "What does continuity and recovery look like here for our organization?"

Gal Shpantzer

That is roughly $100 of cost against revenue from each of UHCÕs roughly 27M US customers, and the actual total will surely be higher. The cause: from UHCÕs CEO during his 'perp walk' in front of Congress: Change Healthcare was a relatively older company with older technologies, which we had been working to upgrade since the acquisition. But for some reason, which we continue to investigate, this particular server did not have MFA on it. Convince your management you need to avoid a similar expensive, avoidable 'but for some reason' incident.

John Pescatore

A lot of lessons learned here that are applicable to other companies. Not only the eye-popping costs of the cyber incident but its root cause. Bottomline, cybersecurity should be a critical component as part of merger/acquisition due diligence.

Curtis Dukes

Management is fundamentally under-estimating both threat and consequences of breaches. These risk acceptance decisions are putting the health and the life of the enterprise at risk. Moreover, as in this case and the CDK case, much of the cost is borne, not just by the enterprise itself, but by its customers and their constituents (e.g. auto dealer employees' compensation that is tied to efficiency.) It is essential to get the basics right. We have a duty; lead or get out of the way.

William Hugh Murray

If you have downtime due to CrowdStrike's disastrous update, sneak in the Oracle CPU updates while systems are down everywhere. That Oracle financial software is installed in a lot of places where it hasn't been used in years.

John Pescatore

Don't let the volume of updates scare you off. There are a lot of products here and you're likely only using a subset of these products. Even so make sure you're doing regression testing, particularly with middleware updates.

Lee Neely

There are definite advantages to using this approach on Windows 11 laptops but given Microsoft's loss of focus on security in recent years, being an early adopter carries risks. I'd like to believe that smaller update packages will then enable Microsoft to move forward with faster than monthly patching.

John Pescatore

Today, the updates, like 23H2, include all the updates since the OS was released. This model is that the update will only contain the delta since the last update which will make them smaller and faster to deploy. This also means you need to apply all the updates in order rather than a single update to catch up. With that process sorted this should lesson the burden of updates which will be a win for users who will have a smaller impact window.

Lee Neely

A CVSS score of 10 basically says no privileges or user assistance is needed to exploit the vulnerability. Users of SSM should download and patch immediately, as evildoers are likely reverse engineering the patch and developing exploit code.

Curtis Dukes

Cisco SSM is used to license software. If someone actually has this deployed internally, you can bet their infrastructure is massive.

Moses Frost

Do you have an AI security policy for your organization that addresses the need to keep customer and employee information safe from internal use of AI software? Don't forget the 2023 Microsoft AI exposure of 38TB of sensitive data, including over 30,000 internal teams messages.

John Pescatore

GenAI requires large, diverse datasets in which to train its models. The collection and use come at a cost in user privacy. More needs to be done to explain what safeguards are in place to protect data used by these systems.

Curtis Dukes

The good news is the VA paused use of a flawed tool, the bad news is it is having a big impact on business processes. They now face the tough question of which is worse: the flaws or business impact. If they're lucky they can mitigate the flaws until a replacement can be deployed. If not, alternate procedures will need to be implemented to resume processing.

Lee Neely