CrowdStrike Says Global IT Outage Impacting Windows PCs, But Mac and Linux Hosts Not Affected

[Godspeed8230]

Shows how little our infrastructure is built to withstand single points of failure. That's not just Crowdstrike, but all of the web too with their endless libraries maintained by a few people.

This has to do with the fact that it was a security-update. For that CrowdStrike needs "god-like" access to all the inner workings of the IT system. A botched update can then lead to what we in Austria/Germany call a Super-GAU or in English a "disaster beyond all expectations".

 

[@Brett]

As a software engineer who has been through many releases, some better than others, I can imagine the chaos going on behind the scenes. Of course I feel for all those impacted too.

I was in multiple war rooms and Bridges at my corporation. It was chaos. I have never seen anything like this.

 

[cyb3rdud3]

This only affected machines that were powered up when they pushed stuff out. They fixed it quickly but this was enough to destroy a ton of servers across the planet.

Sure, possibly. I haven't had the need to dig into the details of this one. That would still be countless machines. And also with the autowake features of endpoint devices, unless they were truly powered off, they likely wake up to get the updates. Combine that with a recommended minimum of every four hours for security updates, this could have spread quickly despite a quick fix. I don't know many people who truly power down nowadays.

And then there is the corporate network caching for distribution as well.

But yes I agree, it wouldn't affect everything, but by the looks of the news reports spread pretty quick.

Have been up since 5AM….

Feel for you, not what you want on a Thursday night going into Friday morning. Especially if you are in the UK as we finally have the hottest day of the year.

 

[Heindijs]

This has to do with the fact that it was a security-update. For that CrowdStrike needs "god-like" access to all the inner workings of the IT system. A botched update can then lead to what we call in Austria/Germany a Super-GAU or in English a "disaster beyond all expectations".

These systems should be immutable. Why would a POS terminal need Windows 10 with automatic updates enabled? Why would they do this with airports? It's just ridiculous

 

[cyb3rdud3]

This has to do with the fact that it was a security-update. For that CrowdStrike needs "god-like" access to all the inner workings of the IT system. A botched update can then lead to what we call in Austria/Germany a Super-GAU or in English a "disaster beyond all expectations".

Exactly, and typically, you accept the risks to allow regular updates as they normally outweigh the risks of not updating. This is very bad for crowdstrike.

 

[Ctrlos]

"This is what will happen if the EU gets their way with Apple"

- somebody around here, probably.

 

[Godspeed8230]

Exactly, and typically you accept the risks to let accept regular updates as they normally outway the risks of not updating. This is very bad for crowdstrike.

Definitely. Imagine being a sales rep trying to sell their product now. "Yes, we made headlines by causing one of the worst IT-outages since the WannaCry incident. Please buy our product".

 

[BanjoDudeAhoy]

I imagine there will be some really juicy lawsuits over this.

 

[ericinboston]

Troubleshooting steps

1. Shut down your PC
2. Get a mac.

This has nothing to do with Windows/PC. It's a 3rd party app, running on Windows, whose job is to act as a security tool...the tool updated silently (as it does often) with a bad update by its own developers, rebooted, and crashed Windows. It's clear the developers at Crowdstrike did absolutely zero testing before pushing the update.

A similar tool could have been installed on non-Windows machines and had the same effect.

I've been running both Mac and Windows at home for 30+ years and neither systems have any 3rd party security tools and hence neither OSes have ever crashed (knock on wood) because of of a 3rd party security update. This tool appears to be aimed at corporate environments where the company really needs extra paranoid protection against all kinds of attacks that us home users normally will never see.

I'm sure all the "just get a Mac and you'll NEVER have to worry about ANY security problems FOREVER" comments have never considered there's a reason why Macs have never prevailed in the corporate world a)in employee user base and b)as the core infrastructure of businesses. I'm not knocking on Macs, but there are plenty of reasons why.

 

[tuckerjj]

We offer our users a choice between Macs and PCs, and I’m very happy for all those that choose Macs for a variety of infrastructure management and support reasons. Having said that I am certainly not crowing or saying “I told you so” to anyone pushing the PC side today. We run Crowdstrike on both and this could easily have gone down the other way and may still do so in the future.

 

[STOCK411]

Pour one out for us IT folks. We are NOT okay this morning and for some of us, very sleep deprived.

NO we are not....
Crowdstrike (which pretty much every damn company uses from government to transit to media to banks use it.
Im in IT. and this is going to be a major ******** including for my company

https://twitter.com/x/status/1814198854671368525

View: https://twitter.com/sinnet3000/status/1814198854671368525?s=46

https://twitter.com/x/status/1814174010202345761

View: https://twitter.com/troyhunt/status/1814174010202345761?s=46

both of the above Twitter threads show the extent of the ********

This is also literally what everyone was afraid of happening to way back during the Y2K scare.

also. Crowdstrike is now going to be referred to as Crowdstroke given the amount of hell they are going to put companies through with this.

Any emergency patch that Crowdstrike pushes out will be useless for those machines stuck in a BSOD.

which means… someone needs to either guide a user over the damn phone or physically get their hands on the machine so that it can get the patch.

 

[3xBoom]

Crowdstrike - name checks out... 👍

 

[goobot]

I find it ironic that Xbox(a Microsoft product) was down all day yesterday as a result of it

 

[Godspeed8230]

Any emergency patch that Crowdstrike pushes out will be useless for those machines stuck in a BSOD.

which means… someone needs to either guide a user over the damn phone or physically get their hands on the machine so that it can get the patch.

They do have 24.000 customers and each customer has multiple machines. Let's say on average a customer has 100 machines then we have roughly 2,5 million computers in need of hands-on-keyboard treatment. Sysadmins will have some work to do.

 

[Saturnine]

For context, I’ve worked in IT for nearly 25 years so I do understand the technologies involved in this incident.

While Crowdstrike is clearly the cause of their distruption, I wonder if there isn’t also a case for Microsoft to answer.

Crowdstrike is crashing the operating system. Presumably it’s able to do this because of the low-level system access is required to function as an EDR solution. I wonder if Windows should be better able to recover from issues caused by third party software. A perfect system would be uncrashable. Of course, no system is perfect.

While I’m sure they’ll accept no responsibility, I’m almost certain Microsoft will be looking into exactly how a software product running on its platform was even able to crash its operating system.

 

[addamas]

“X is limited to A, but B and C are not affected”.

Dude…

 

[edubfromktown]

Doesn't happen often (every ~5 years or so in my experience) and sure does hurt. It is imperative to test this sort of stuff before deploying across an enterprise.

Same is true for all security appliances and software subscriptions that push dynamic updates automatically. Got burned by Palo Alto Networks, CrowdStrike and a few others years back from similar sorts of occurrences.

 

[Pezimak]

The is what happens when many fingers are in the pie of the core OS and release updates, one of them is bound to be buggy due to lack of concentration or meeting a deadline. And bring it all crashing down, literally. Mac for the win!

 

[Larsvonhier]

“X is limited to A, but B and C are not affected”.

Dude…


Also deploy of fixes without checking them on test environment is a pure joke from Microsoft.

And in additon, wrong: If you run Windows on your Mac, you have been affected.
This is what you get when you confuse a hardware platform (Mac) with an OS (Windows, Linux).

 

[@Brett]

If you have a company deployed laptop, and you use something like crowdstrike, then surely they configured it correctly, and thus you wouldn't be able to boot it in safemode and do this yourself

Oopsie

Especially for machines with Bitlocker encryption which requires the key to alter startup.

 

[therunningman]

Does anyone know if this was an actual PRODUCT update, or was it a anti-malware content updates only?

 

[Pezimak]

I was in multiple war rooms and Bridges at my corporation. It was chaos. I have never seen anything like this.

But why? It’s pointless, you have to take the loss as most companies rely entirely on IT infrastructure, and when it all breaks you have NO backup, no paperwork to replace it all, and you can’t fix it when it relies entirely on an external source and the cloud. Just sit back, stop worrying, and wait for that external source to fix it. Bosses can ask as many questions as they like and jump up and down all they like. Still won’t fix anything.

 

[falkon-engine]

It sucks when a bad update has the same effect as malware.

 

[unchecked]

Affects Windows10? That’s about as big a pitch for updating to Windows11 as it gets.

 

[ifxf]

Missed Microsoft Azure having a failure in the Midwest. Many airlines are delaying their flights.