CrowdStrike issues go beyond Windows: company's security software has reportedly been causing Linux kernel panics since at least April

Red Hat Linux Raleigh, NC HQ
(Image credit: Shutterstock)

Last Friday, the world experienced the biggest global outage of key Windows PC infrastructure in history — this issue, caused by a botched CrowdStrike update to its kernel-level Falcon Sensor software, made modern Windows systems so fundamentally non-functional that flights around the world were delayed. Southwest Airlines managed to avoid the issue, because the company was using Windows 3.1 instead of any remotely modern version of the OS.

But as it turns out, the problem isn't just isolated to modern Windows operating systems. Linux users have been reporting kernel panics and crashes related to the same software since as early as April of this year, per a report from The Register.

So, how is this issue cross-platform? Chances are the specific issue that caused chaos over the last few days is not— after all, we would've seen it cripple Windows machines much sooner if that were the case. However, what this does demonstrate is that CrowdStrike has apparently been lax with its Falcon Sensor Security software for quite a while now.

For those unfamiliar, the "kernel" of an operating system refers to the layer outside of user interaction (typically called the "shell"), and most directly connected to the hardware beneath. The thing is, very little computer software actually needs kernel access to get its work done. And while security software can certainly be an exception because threats often may attempt to infiltrate the kernel, it's still very important to ensure that the software isn't also causing kernel instability and crashes for any target platform.

An interesting sidenote pointed out by The Register is that CrowdStrike's current CEO, George Kurtz, was also CEO of McAFee during an infamous 2010 update that caused several PCs to be stuck in an endless boot loop. This likely makes George Kurtz the first CEO in history to preside over two major global PC outages caused by bad security software updates.

Linux users who have been impacted reportedly include those using Red Hat Enterprise Linux, Debian Linux (and Debian is the basis for the more-widespread Ubuntu), and Rocky Linux. All of the issues in question are impacting the underlying Linux kernel (universal across Linux distributions), though, seemingly crashing any Linux distributions using kernel versions 5.14.0-42713.1 and newer.

Linux users do seem to have more recourse for issues like this— including switching to an eBPF "User Mode"— but it speaks to the severity of CrowdStrike's kernel software development issues if the company is managing to cripple Linux and Windows operating systems. 

It also shows that there were warning signs for this past global outage, and that systems should have been in place at CrowdStrike some time ago to test these enterprise and government-targeted updates vigorously enough to prevent these kernel-level crashes. After all, most impacted users in these strictly-controlled environments likely don't have the administrative access or knowledge required to fix these problems once they occur. In other words, much-improved QA testing would seem to be mandatory for CrowdStrike's continued long-term success.

Christopher Harper
Contributing Writer

Christopher Harper has been a successful freelance tech writer specializing in PC hardware and gaming since 2015, and ghostwrote for various B2B clients in High School before that. Outside of work, Christopher is best known to friends and rivals as an active competitive player in various eSports (particularly fighting games and arena shooters) and a purveyor of music ranging from Jimi Hendrix to Killer Mike to the Sonic Adventure 2 soundtrack.

  • bit_user
    The article said:
    the "kernel" of an operating system refers to the layer outside of user interaction (typically called the "shell")
    The dividing line is usually between kernel and userspace. Userspace includes things besides the interactive shells, such as services (aka "daemons"). Libraries and executables are comprised of 100% userspace code. Drivers, filesystems, networking protocols, and other kinds of modules, which either need privileged access to the hardware or other parts of the kernel, are what go in that domain.

    Some operating systems define multiple levels of privilege. Since the 80386, x86 CPUs have supported 4 "rings", with the kernel running in "ring 0". Transitions between these rings adds a lot of overhead, which is one reason it's not popular to make much use of more than 2 (usually rings 0 and 3).

    The article said:
    very little computer software actually needs kernel access to get its work done. And while security software can certainly be an exception because threats often may attempt to infiltrate the kernel
    Uh, the reason security software wants (at least some portion) to reside in the kernel is so it can see and potentially intercept what all the userspace code is doing.

    The article said:
    Linux users do seem to have more recourse for issues like this— including switching to an eBPF "User Mode"— but it speaks to the severity of CrowdStrike's kernel software development issues if the company is managing to cripple Linux and Windows operating systems.
    It's not up to the user, whether & to what extent CrowdStrike uses eBPF. That's a matter of how they choose to design & implement their software. eBPF provides a sort of virtual machine-like sandbox, where kernel functionality can be extended in a way that can't cause collateral damage and has limited security liabilities.
    Reply
  • DS426
    I'd also have to suggest that they are prioritizing feature development pace over stability. It's been part of their competitive strategy in the security world and has paid dividends for them... up until now when pretty much the worst-case scenario happens.

    Definitely not ideal that customers have almost not control over channel file updates.

    They also recently had faulty sensor versions for Windows that would rocket one core on Intel systems to 100% load. While that doesn't sound like it'd be a big deal on quad-core plus systems that are common today, Windows gets rather strange when even a single core gets pegged at 100% utilization.
    Reply
  • vern72
    The null pointer reference heard 'round the world!
    Reply
  • Alvar "Miles" Udell
    Wonder how many meetings went like this:

    Executive: Why are we using CrowdStrike?
    Senior IT: It's cheaper than Microsoft.
    Executive: In that case, your services are no longer required/
    Senior IT: Why?
    Executive: It's cheaper to replace you.
    Reply
  • wingfinger
    Maybe a change of staff is the cause ?

    and

    Executive: Everyone is using cloudstrike why are we using microsoft?
    Reply
  • ekio
    This article was sponsored by Microsoft :)
    Our garbage OS is that only one to get bad publicity. Instead of focusing on improving it, let’s try to drag the other’s reputation too.

    Seriously, this company causes more issues than fixes. Why pay for that sh!t?
    Reply
  • Vanderlindemedia
    The new boeing under the IT.
    Reply
  • CmdrShepard
    bit_user said:
    Since the 80386, x86 CPUs have supported 4 "rings", with the kernel running in "ring 0". Transitions between these rings adds a lot of overhead, which is one reason it's not popular to make much use of more than 2 (usually rings 0 and 3).
    You forgot Hypervisor which is running at ring -1.
    bit_user said:
    Uh, the reason security software wants (at least some portion) to reside in the kernel is so it can see and potentially intercept what all the userspace code is doing.
    All the people clamoring here about how Microsoft should limit kernel access should be careful what they wish for.

    Also, all the people blaming Microsoft are blaming them for the wrong thing. The level of access in kernel mode is not the problem -- the problem is the lax WHQL certification which doesn't apparently mean anything as the driver code isn't reviewed by humans but just passes automated testing. Second problem is the existence of double standards.

    Many of you don't know what you need in order to release a kernel mode driver today.

    You need EV certification which is not something individual developers can get (I heard those rogue engineers can be very dangerous if let near the kernel), and also some Azure integration for your driver build and release process.

    I have been informed couple of months ago by a friend that Microsoft is supposedly tightening kernel driver security by not allowing arbitrary memory / bus access anymore -- drivers supposedly should declare what they will access and stick to that or they risk getting blacklisted.

    Now take into account that this CrowdStrike "definition" file which caused the whole ruckus was actually a file containing custom scripting which was parsed and executed in kernel mode without as much as a CRC32 to detect file corruption, let alone authenticated using PKI and signature validation to make sure it came unaltered from a trusted source and you can't but ask yourself -- how in the hell has Microsoft allowed an unsecured script interpreter in a kernel driver code?

    So once again, problem isn't the level of access drivers have, they need that to do their job -- the problem is Microsoft doing absolutely nothing to prevent drivers from containing bytecode or script interpreters. The other problem is as I said double standards -- individual developers aren't even allowed to write kernel mode drivers while corporations like CrowdStrike are allowed to do anything without any scrutiny.

    You'd get absolutely the same problem if you took Java, Javascript or Python executable and somehow ran them in ring 0.
    Reply