Crowdstrike’s bad firmware update grounds the world

One bug can take down most of the world it seems as we all face a world where even a small error in the world’s “digital fabric” can have tremendous wide spread ramifications. This week banks, airports, TV stations, hospitals, hotels, and countless other businesses were all affected by widespread IT outages, leaving flights grounded and causing widespread disruption, after Windows machines displayed errors worldwide.

In the early hours of Friday, companies in Australia running Microsoft’s Windows operating system started reporting devices showing Blue Screens of Death (BSODs). Shortly after, reports of disruptions started flooding in from around the world, including from the UK, India, Germany, the Netherlands, and the US: TV station Sky News went offline, and US airlines including American Airlines, Delta, and United Airlines, for the first time ever issued a full “global ground stop” on all flights.

It could be a lot worse in the future ...

The widespread Windows were linked to a software update from cybersecurity giant CrowdStrike and it’s estimated impact so far is at least $5.4 Billion. It is not believed the issues are linked to a malicious cyberattack, cybersecurity officials say, but stem from a misconfigured  software update that CrowdStrike pushed out to its customers.

Engineers from CrowdStrike posted to the company’s Reddit forum that it has seen “widespread reports of BSODs on Windows hosts” occurring across its software, is working on the problem, and has advised a workaround for impacted systems. It also issued instructions to its customers in an advisory.

The incident, so far, has only been impacting devices running Windows and not other operating systems, but it’s unclear exactly how widespread the issues are and how long they will take to resolve.

However, the outages could result in billions of dollars worth of claims by organisations who’ve been impacted and have had to halt their operations or stop business, says Lukasz Olejnik, an independent cybersecurity consultant, who says the CrowdStrike update appears to be linked to its Falcon Sensor product. The Falcon system is part of CrowdStrike’s security tools and can block attacks on systems, according to the company.

“It reminds us about our dependence on IT and software,” Olejnik says. “When a system has several software systems maintained by various vendors, this is equivalent to placing trust on them. They may be a single point of failure - like here, when various firms feel the impact.”

The outage stemming from the CrowdStrike update has had a huge knock-on impact on public services and businesses around the world. Scores of airports are facing delays and long queues, with one passenger in India sharing a hand-written boarding pass that they have been issued.

In the UK, NHS England has confirmed that GP appointment and patient record systems have been impacted by the outages. Also in the country, train operators have said there are delays across the network, with multiple companies being impacted.

Among other services, CrowdStrike provides endpoint detection and response (EDR) to companies around the world. This EDR technology runs on thousands of “endpoints” - such as computers, ATMs, and internet of things devices - and scans them to identify real-time threats, such as malicious activity from cybercriminals. The company has more than 24,000 customers around the world.

Cybersecurity researcher Kevin Beaumont posted on X that he has seen a copy of the CrowdStrike update that was issued and says the file isn’t properly formatted and “causes Windows to crash every time.” Beaumont says, in further posts, that it appears there isn’t an automated way to fix the issues, at least currently.

Brody Nisbet, the director of overwatch at CrowdStrike, also posted on X indicating the workaround fix the company had issued involves booting up Windows machines into safe mode, finding a file called “C-00000291*.sys,” deleting it, and then rebooting the machine normally.

“There is a fix of sorts so some devices in between BSODs should pick up the new channel file and remain stable,” Nisbet posted, however, with many sysadmins having to actually physically visit the affected machines to de-bug and restore them it’s likely that a full recovery from the calamity will take quite a while.