CrowdStrike Says Global IT Outage Impacting Windows PCs, But Mac and Linux Hosts Not Affected

[cyb3rdud3]

@davide_eu Yeah, truth, If every unix & linux operator shouted at the top of their DACs, it wouldn't get covered in popular press. Popular means Windows. No mass anxiety means no engagement means no ad revenue means no coverage.

Meanwhile, Apple doesn't get to sit at the grownup enterprise table because their current ideology, and resultant product lines, including IOS, are inimical to enterprise admin (like they're proud of that). Plus Apple users' culty b.s. makes IT admins tired, even the few Apple enterprise admins, honestly.

Yes, but definitely not iOS. iOS is very mature, and even more importantly, very secure. It is an excellent choice in an enterprise setting.

This time isn't even Microsoft's fault, directly, but blaming their weaknesses on the EU, and on "We're not Apple", is petty, ignorant, superficial Gen-Z hogwash. Even if Microsoft isn't allowed to interfere with third-party wtfery, CUSTOMER ITSM's certainly can. If not out-of-box, then in AD GPOs, or in local policy objects, or in Crowdstrike policies, or even firewalls. Even then, the ITSM should have a netboot or PXE backplane ready to perform recovery actions.

This particular Crowdstrike hornswaggle happened one ridiculous way; Crowdstrike owns the root cause and at least they're not passing blame. Nonetheless, ITSMs own the business outcomes, and there were half a dozen ways ITSMs could have prevented or mitigated the damage.

Unguarded, autonomous Auto-Update is a choice - quite simply, an abdication of ITSM responsibilities. Prod should simply never be subject to auto-updates from external vendor or internal developers - even if you organization mandates rapid response to zero-day threats. Rapid response doesn't mean allowing your infrastructure to get bitraped by a vendor.

Four decades into the modern distributed computing, hub and spoke client server IT Industrial shell game, ITSM's should be benefiting from lessons learned, no longer BEING THE BAD EXAMPLE. One of the lessons might be "Don't skimp on an ITSM Security Team." Bureaucrats forget that real IT Security is far more involved than churning out b.s. compliance reports and b.s. "Five 9's" uptime statistics.

How do you propose your organisation overcomes not doing auto-updates for your cybersecurity stance, and achieve or maintain an adequate security posture, let alone certification? Yes, it is a choice, typically made based on a risk appetite. The likelihood of an event that has just happened is very low, was generally accepted as negligle. Whilst the likelihood of zero-day threats having an impact, was and is higher. I fully expect that in a mature organisation the risk score will temporarily go up in these area, and that accepted mitigations will change. But I wouldn't switch off auto-updates and regular roll out of these configurations and signatures any time soon.

 

[BotchQue]

Maybe not entirely on-topic, but it made me giggle.

 

[ipaqrat]

Yes, but definitely not iOS. iOS is very mature, and even more importantly, very secure. It is an excellent choice in an enterprise setting.

How do you propose your organisation overcomes not doing auto-updates for your cybersecurity stance, and achieve or maintain an adequate security posture, let alone certification? Yes, it is a choice, typically made based on a risk appetite. The likelihood of an event that has just happened is very low, was generally accepted as negligle. Whilst the likelihood of zero-day threats having an impact, was and is higher. I fully expect that in a mature organisation the risk score will temporarily go up in these area, and that accepted mitigations will change. But I wouldn't switch off auto-updates and regular roll out of these configurations and signatures any time soon.

Yeah, it's tough, not cheap, and not a proposal. We do auto-ish-update, with a delay loop, of an hour or so, for sanity checking. System owners do have an unenviable balancing act, between one regulation that demands instant response to zero-days, and more mature regulations that demand Supply Chain Security validation, plus a rigorous Configuration Management Plan, plus a Continuity of Operations plan.

We run a lab with the closet possible mirror of Prod OSs, apps and workflows (though not data and work loading, obviously). We live on call (well, not me, of course, because I'm old and only architected it). There's health monitoring with 24-7 alerts from SOC when they hit something truly urgent, like a new zero-day in the wild. We're usually up, caffeinated and waiting around for vendors to get their **** together. We zip through the tests, and then enable auto-update in Prod.

Sometimes our SOC alert starts with CISA vetting exploit and detection updates, which means we Protect-Serve and have a joy-joy day. In any event, I can't see a rush to rip and replace Crowdstrike. We've ALL seen lots of vendors screw up, and if we cancel every time, there's no benefit from the vendors' lessons learned.

As for IOS, yes, the core is reasonably secure. Compliance scans are grudgingly accepted even though most configuration and vulnerability scanners only query an MDM, as opposed to the device itself (mixed blessing). Crowdstrike is an odd duck in IOS support, by providing an on-device software agent (also a mixed blessing)

 

[cyb3rdud3]

Yeah, it's tough, not cheap, and not a proposal. We do auto-ish-update, with a delay loop, of an hour or so, for sanity checking. System owners do have an unenviable balancing act, between one regulation that demands instant response to zero-days, and more mature regulations that demand Supply Chain Security validation, plus a rigorous Configuration Management Plan, plus a Continuity of Operations plan.

We run a lab with the closet possible mirror of Prod OSs, apps and workflows (though not data and work loading, obviously). We live on call (well, not me, of course, because I'm old and only architected it). There's health monitoring with 24-7 alerts from SOC when they hit something truly urgent, like a new zero-day in the wild. We're usually up, caffeinated and waiting around for vendors to get their **** together. We zip through the tests, and then enable auto-update in Prod.

Sometimes our SOC alert starts with CISA vetting exploit and detection updates, which means we Protect-Serve and have a joy-joy day. In any event, I can't see a rush to rip and replace Crowdstrike. We've ALL seen lots of vendors screw up, and if we cancel every time, there's no benefit from the vendors' lessons learned.

As for IOS, yes, the core is reasonably secure. Compliance scans are grudgingly accepted even though most configuration and vulnerability scanners only query an MDM, as opposed to the device itself (mixed blessing). Crowdstrike is an odd duck in IOS support, by providing an on-device software agent (also a mixed blessing)

All makes sense. Nice job.

 

[kk200]

eBPF does exist on windows. Crowdstrike don’t do it because they hook NT API. Also DTrace exists on windows as well.

GitHub - microsoft/ebpf-for-windows: eBPF implementation that runs on top of Windows

eBPF implementation that runs on top of Windows. Contribute to microsoft/ebpf-for-windows development by creating an account on GitHub.

github.com

As far as I read, it only supports a limited kernel functions on windows. Not fit for anti virus yet.

 

[cyb3rdud3]

Update from CrowdStrike https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/

 

[ipaqrat]

Yeah... they sound real good with the testing and then dogfooding, which they claim to do with the Rapid Response content, same as Sensor Content. They described layers of testing, quite appropriate...

Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production.
>>> but right here, no dogfooding <<<
When received by the sensor and loaded into the Content Interpreter, problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception. This unexpected exception could not be gracefully handled, resulting in a Windows operating system crash (BSOD).

Dodfooding was the final safety net, last chance to catch the error with a manned sanity check, like "Hey, BSOD!" Then everybody gets to drink a shot -- we made BSOD into a drinking game so the admins have something to look forward to.

Note that Crowdstrike is characterized by its distributed, i.e., CROWD-SOURCED, intel gathering. THAT'S what makes it Crowdstrike. By installing their platform, you essentially consent to BECOME THEIR CROWD. Rapid Response content isn't necessarily preventative in nature; they publish it when they get a hit on some new badness. The Rapid Response template enumerate the environment - and, of course, block it if they already know how - before a little bad thing grows into a big bad thing. The bigger their crowd the more sensitive their system becomes.

That's probably why Crowdstrike hasn't provided throttle controls on that content on customer ITSM's infrastructure (like they do with Sensor Content N-1, N-2). V'Ger requires the goddam information, and no pesky ITSM's allowed to interfere. It's pivotal to their zero-day-suppression marketing. And, hey, mostly it works.

Anyway, when we realized that Rapid Response content was funneled directly into prod through the agent, we created some rather elaborate network and content filtering files and directory permissions to enable us to interdict, dogfood and release. We don't capture-and-redeploy; rather, when we're satisfied, we just lift the gate and let it run normally. We learned to do this at a different agency that was heavily invested in McAfee at the time (although McAfee did have super-granular policy settings - very complex, but very granular).

Frankly, we'd be relieved if Crowstrike would fast-track those throttling policy tools for the Rapid Response content. Be a lot less expensive if we could reliably govern them with their own tools, and it could only improve Crowdstrike's legal liability exposure, as well. Admittedly, we didn't get a giant Windows problem, we're mainly linux, so this particular event is largely academic.

 

[Luposian]

Yesterday my PC at work kept freezing and got the blue screen of death. I wonder if it was related. Probably a coincidence though.

My wife opened her laptop (here at home) and, lo and behold, it was going bonkers. Something about HTML login or something, and then when she went to reboot, it gave her a blue screen saying something was wrong and to skip the current drive or something. When I had her reboot and press F8, the system just booted normally, Weirdest thing! That's Windows 11 for ya! She uses a bunch of Adobe and Microsoft products on her laptop, rarely shuts down/reboots.

I've heard that the way macOS/Linux address drivers or whatever is why they weren't affected, while Windows was. So, it seems more to do with how Windows 10/11 handles programs/drivers and less to do with CrowdStrike... at least as far as the software overall. If the OS is more hardened against such poor code, it can't be crashed, like Windows 10/11 can be.

 

[cjsuk]

As far as I read, it only supports a limited kernel functions on windows. Not fit for anti virus yet.

Yeah it needs to hook some of the Zw APIs for file and network initiation. That’s all. Not rocket science. Actually windows has APIs for that already that don’t require in-kernel hooks.