Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Ok, I haven't read all 17 pages of this, so it may have been mentioned before, but...

This was caused by a global, simultaneous rollout of an update that wasn't tested properly (but that's a different problem).
Both Apple and Ubuntu (and I don't know about other organisations) have staged rollouts. If anything bad happens, only part of the digital ecosystem is affected.

It will be up to others to determine --
  1. What happened with the pre-rollout testing.
  2. Why it wasn't pushed out in a staged manner (New Zealand first, then Australia, then Canada, Britain, Europe, Africa, Asia, and finally the US).
Yes, it was a dinky, little change, but dinky little faults can have major consequences. Even a 1 sq. inch bald spot on your tire can cause you to spin out and hit a tree.

I've heard a couple of explanations...

1) That the file was corrupted.

2) That it was a combination of the CrowdStrike update being applied after a Microsoft update that occurred shortly beforehand.

Sounds like bad luck more than anything. I trust that they had tested with what was in place at the time.

Definitely should be a more gradual staged rollout to reduce the impact.
 
If you have a company deployed laptop, and you use something like crowdstrike, then surely they configured it correctly, and thus you wouldn't be able to boot it in safemode and do this yourself ;)

Oopsie
Correct. The fairly simple instructions cannot just be disseminated and executed by the end user. Properly secured systems have several security products that also must be disabled (temporarily) with the assistance of the appropriate corporate security support group(s). As such even a single machine issue takes a lot of time to fix.
 
  • Like
Reactions: cyb3rdud3
Give them some credit… no malware today on machines with boot issues.

More annoyed with Azure VMs being a real pain to boot in to safe mode to delete the affected file.
biggest SO FAR... why assume we've peaked so early in the year? ;)

all those AI phone releases coming...
 
  • Like
Reactions: The Apple Bitch
Not sure why people are crapping on Microsoft here. The issue is crowdstrike. They have Linux and Mac versions of Falcon sensor too. They just happened to screw up the windows version today and push it to production past everyone’s security controls.

This could be Apple or Linux tomorrow.
Not likely if you practice “no deploy fridays”.
 
Not sure why people are crapping on Microsoft here. The issue is crowdstrike. They have Linux and Mac versions of Falcon sensor too. They just happened to screw up the windows version today and push it to production past everyone’s security controls.

This could be Apple or Linux tomorrow.
Its MacRumors, if there is something that affects something related to Microsoft, people will pile on it here without thinking.
 
Windows world is not at a good place right now. Windows devotees do not like the new Qualcomm chips at all, while the latest high-end i9 Intel chip seems to be a massive disaster with up to a 100% failure rate acording to some sources and a blame game as to why.
 
  • Like
Reactions: com.B
Not sure why people are crapping on Microsoft here. The issue is crowdstrike. They have Linux and Mac versions of Falcon sensor too. They just happened to screw up the windows version today and push it to production past everyone’s security controls.

This could be Apple or Linux tomorrow.
Yes but MS. Needs to take some responsibility with why a 3rd party application could cripple so many machines. However you would have thought Crowdstrike would have tested this in a Dev environment first.
 
Sure, possibly. I haven't had the need to dig into the details of this one. That would still be countless machines. And also with the autowake features of endpoint devices, unless they were truly powered off, they likely wake up to get the updates. Combine that with a recommended minimum of every four hours for security updates, this could have spread quickly despite a quick fix. I don't know many people who truly power down nowadays.

And then there is the corporate network caching for distribution as well.

But yes I agree, it wouldn't affect everything, but by the looks of the news reports spread pretty quick.

Feel for you, not what you want on a Thursday night going into Friday morning. Especially if you are in the UK as we finally have the hottest day of the year.
It started in Australia so we were well up and running business wise. Around 2:45 I got the first reports. Worked late back again tonight for BCP and fixing the remainder of the machines.
 
  • Like
Reactions: cyb3rdud3
About the global computer outage (July 19, '24): excerpt from an article in the « Journal de Montréal » (Canada) :
...Friday's situation raises questions about the Windows architecture. "Microsoft allows manufacturers, third parties to drop files into the core of Windows, which is very sensitive. And here, we see the effect, "says the cybersecurity consultant, who also specifies that Apple has banned this way of doing things for several years. " Maybe it's time for Microsoft to think about its architecture, "suggests Mr. Sauvé.

JdM article link here (in french).
 
  • Like
Reactions: The Apple Bitch
Linux wasn't affected this time. Last year we discovered a kernel memory leak by them that was killing our high-volume backend services with out-of-memory errors every 3-4 days. Memory usage for these processes was very mysteriously increasing even though address sanitizer didn't show any leaks on our end.
 
When you deploy, distribute, a bad update on a Tuesday it will still be a bad update ;) The old fashioned do not deploy on Fridays practice would not mitigate against that risk at all.
True, but whoever dealing with the issue will - in general - have more human resources available to deal with it. On the other hand, depending upon the industries affected, if no one or fewer needs their bricked stuff until monday, the losses would be smaller if they got the staff to fix it throughout the weekend.

The better option would be to TEST the updates on a small representative array of computers BEFORE dropping it on the globe though.
 
Crazy how something I'd never even heard of until today (seriously, what is CrowdStrike?) just caused Y2K.
Did you ever hear about Takaka before they had to start recalling airbags?

The are thousands of companies that supply parts or tools to other companies. Most of them will never be a household name to the general public. That does not mean they are not a good company, just they don't focus on consumers/
 
Crowdstrike has a posted a preliminary explanation about what happen.

https://www.crowdstrike.com/blog/technical-details-on-todays-outage/

As I suspected earlier, this was a NOT a sensor update, it was a bad patch definition. These definition are issued multiple times a day to all devices to address new security threats. So this would be similar to how Apple issues patches to XProtect, although way more frequently. That is why there was there no phased rollout.

Crowdstrike and Microsoft both will need to look deeply at what happened. Why did a definition update bring down the whole system? What can be done to protect against that in the future?

And hats off to my Windows counterparts that had to deal with it. I am sure many of you will be working the weekend to bring systems back up.

At our org, we just moved off Crowdstrike earlier this year, so the damages was limited to a small percentage of our computers and servers that had not yet been migrated to Defender. And, as the Mac Admin, I didn't have anything to worry about.
 
Windows world is not at a good place right now. Windows devotees do not like the new Qualcomm chips at all, while the latest high-end i9 Intel chip seems to be a massive disaster with up to a 100% failure rate acording to some sources and a blame game as to why.
Both Intel and Microsoft have "always" spent A LOT of resources preventing competiton such as the Intel compilator scandal where the code limited AMD performance, the document format war to prevent "office competition", killing of competitor contracts, Mono, buying GitHub and so on. They did the same stuff with gaming back in the day. Did I mention BIOS?

What it did was slowing down the competition which gave them time to do their own stuff without changing much. For instance making it very difficult to use Linux clients with Microsoft servers, which owns e.g. public sector. The only reason I have had to use Windows the last 25 years is document formats and public sector Microsoft servers.

One consequence is the vast numbers of professionals working with Microsoft software, and they would of course not want to reeducate themselves on open source, Linux, Apple stuff and so on, which pretty much serves as barrier against competition.

As for Intel, the number of lies they have used to sell their ovens (performance, heat, power consumption and what not) are nothing less than fantastic.

Both Microsoft and Intel have been too full of themselves, confident they could keep on playing their game to stifle competition. Intel and Microsoft remains irrelevant for pads/phones, the rumors about Apple developing their own hardware was going on for years, but still Intel was to full of themselves to realise what was going on.

Now, they are still in panic, everyone knows that it is early days for M, and it seems like Intel still are looking for the shortcuts to performance, whilst the Arm competition have "nicked" an Apple team to catch up.

And that`s what they all do, playing catch up. Anything on the client side is driven by either Apples present hardware or the fear of what it will progress towards. They still aren`t there with gaming/workstations, but they will be.

As for Microsoft, they have tried to clean up their client platforms for a loooong time, but until they realise it`s rotten core up they still won`t become technically and structurally superior to *nix iterations. They have messed up so many Windows versions that I have lost count. One can start anywhere, but Windows 3.1 was heavily critisised by their users (including yours truly). Then came Vista, which was so bad that the critics of Windows 3.1 started glorifying it. Windows still isn`t good.
 
  • Like
Reactions: dimittar and rp2011
Did you ever hear about Takaka before they had to start recalling airbags?

The are thousands of companies that supply parts or tools to other companies. Most of them will never be a household name to the general public. That does not mean they are not a good company, just they don't focus on consumers/
Find it rather intersting that there still are around 17 million cars not recalled. BMW just recalled short of 400.000 cars due to Takaka airbags. It`s been a while since the scandal broke (2013).....
 
MacRumors, your headline makes no sense. The second clause does not negate the first.
Surprisingly, most people don't understand how to use 'and' or 'yet' properly, so they use 'but' instead. This makes no sense and is sloppy and unprofessional. Another misused word is 'only', or rather its placement. The sentence 'They only washed their car with water' means that washing their car was the only thing they did. However, people often mean to say that they washed their car using only water.
 
Crowdstrike has a posted a preliminary explanation about what happen.

https://www.crowdstrike.com/blog/technical-details-on-todays-outage/

As I suspected earlier, this was a NOT a sensor update, it was a bad patch definition. These definition are issued multiple times a day to all devices to address new security threats. So this would be similar to how Apple issues patches to XProtect, although way more frequently. That is why there was there no phased rollout.

Crowdstrike and Microsoft both will need to look deeply at what happened. Why did a definition update bring down the whole system? What can be done to protect against that in the future?

And hats off to my Windows counterparts that had to deal with it. I am sure many of you will be working the weekend to bring systems back up.

At our org, we just moved off Crowdstrike earlier this year, so the damages was limited to a small percentage of our computers and servers that had not yet been migrated to Defender. And, as the Mac Admin, I didn't have anything to worry about.
I've heard a rumour that it did pass all testing in their CI pipeline, something like 40,000 different configurations included. But that there was a repackaging step that corrupted the file before deployment. They won't be the only ones doing that, but it is a bit silly. Always test what you actually deploy.
 
Surprisingly, most people don't understand how to use 'and' or 'yet' properly, so they use 'but' instead. This makes no sense and is sloppy and unprofessional. Another misused word is 'only', or rather its placement. The sentence 'They only washed their car with water' means that washing their car was the only thing they did. However, people often mean to say that they washed their car using only water.

I think the best solution here would have been not to use a conjunction at all. "CrowdStrike Says Global IT Outage Impacting Windows PCs. Mac and Linux Hosts Not Affected". (A semicolon could also work). Headlines are weird though, since they are generally not written as complete sentences, with many words omitted.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.