Sebastian Döll’s Post

☢ Crowdstrike caused Bluescreen from 19.07.2024. ▶ Youre affected when files in C:\Windows\System32\drivers\crowdstrike\ and name like: C-00000291-****.sys have a timestamp before 19.07.2024 UTC 5:27AM. (starting ~03AM UTC) ❇ Fix: Boot to SafeMode and delete C-00000291-****.sys File. Reboot. Fixed. Crowdstrike will then refresh the C-00000291-****.sys file with the next update

A 12 pages RCA from crowdstrike finally. So what happened. Lets look at it in a simple way: Expected Input Fields: 20 Provided Input Fields: 21 Memory Allocation: Memory was allocated for 20 input fields.Out-of-Bounds Read: The 21st field caused the system to access memory outside the allocated range, leading to a crash. #define MAX_FIELDS 20

If you're affected by the defective CrowdStrike driver update, you can run the following as a temporary workaround to apply the fix to your endpoints as soon as possible. This workaround requires bitlocker to be *disabled*: 1. Click See Advanced Repair Options 2. Click Troubleshoot 3. Click Command prompt and enter the following: "pushd C:\Windows\System32\drivers\CrowdStrike del 'C-00000291*.sys' exit" 4. Click continue. After a reboot, your endpoints should boot up. Apply the latest CrowdStrike fix as soon a possible.

Good technical breakdown of the meltdown resulting from the latest Crowdstrike update (link in comments). Tl;dr #kerneldrivers open up all kinds of risks – including system degradation or even outages. When we built Senser, we were aware of these risks – from our own experience putting out fires as a result of errors in driver code. That's why we used #eBPF tech for lightweight, non-intrusive data collection. eBPF programs are executed in an isolated environment (so they can’t access or modify sensitive kernel data structures) and go through a verification process before they are loaded into the kernel. Of course comprehensive data collection is just the first step in smart observability. But today's meltdown shows the devastating cost of outages and the risks of kernel drivers – a good reminder of the benefits of safe, secure, and lightweight system monitoring.

As systems become online after the CrowdStrike nullptr issue of yesterday, I can’t stop wondering if this can change Microsoft’s exception handling in kernel space. So a driver access the first page because it has a logical error - ok. Page isn’t there, no real harm can be done. OS know knows this driver has an issue - why dowsn’t the OS try to remedy the situation instead of just putting both hands in the air and giving up? Why not attempt to unload the driver, unhook it or whatever means possible instead of BSOD? Report it back via telemetry and make sure the driver doesn’t load again? Maybe this can be a policy setting companies can configure - in critical systems giving up this easily could be avoided? Just my 2 cents….

Workaround options for Crowdstrike wrong update: 1. Roll back to a virtual snapshot before 0409 UTC. 2. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory. Locate the file matching “C-00000291*.sys” and delete it. Note 1: file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version. Note 2: file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.

Curious what the CrowdStrike issue is and what it means? I do my best to break it down into non-tech talk for easy consumption here https://lnkd.in/gEfhmC_Y . Are you a tech leader who is pulling your hair out manually fixing each computer one at a time? A reddit user has a brilliant fix using pxe boot you can read about here https://lnkd.in/gyKYDBMc