CrowdStrike shares details on cause of global tech outage
Last week’s global tech outage has been traced back to a bug in U.S. cybersecurity firm CrowdStrike’s quality control system.
The outage’s impacts have been far-reaching, affecting roughly 8.5 million Windows devices and disrupting banks, emergency call centers and airlines. Fortune 500 companies – not including Microsoft – face an estimated $5.4 billion in losses from the outage, according to insurer Parametrix. Meanwhile, hackers have used the outage as an opportunity to target CrowdStrike customers.
“The fact that a proper analysis wasn't done ended up having this huge cascading problem that companies are still dealing with today,” said Scott White, an associate professor and director of the cybersecurity program and cyber academy at George Washington University in Washington, D.C.
What was the cause of the IT outage?
Early in the day Friday, CrowdStrike pushed out what was supposed to be a routine software update to help monitor for possible emerging threats. But the update was “problematic," triggering a memory problem that set off Window's "Blue Screen of Death," according to the firm's preliminary post incident review. Mac and Linux hosts were not affected.
The software "attempted to do something Windows couldn’t process, and the system crashed as a result,” according to Dominic Sellitto, clinical assistant professor of management science and systems at the University at Buffalo School of Management in New York.
CrowdStrike said it has a "content validator" review software updates before launch, but the program missed the update's problematic content due to a bug.
“On Friday we failed you, and for that I'm deeply sorry,” wrote CrowdStrike Chief Security Officer Shawn Henry in a Monday LinkedIn post, adding that "thousands of our team members have been working 24/7 to get our customer systems fully restored."
The firm told USA TODAY it sent Uber Eats gift cards to teammates and partners who have been helping customers. TechCrunch reported that some recipients have had trouble accessing the gift, and CrowdStrike confirmed that Uber flagged the gift cards as fraud "because of high usage rates."
What happens next for CrowdStrike?
CrowdStrike said it plans to improve its testing, give customers more control over when updates are installed and stagger future software updates to its “Rapid Response” content.
Gregory Falco, assistant professor of engineering at Cornell University in New York, described the steps as "good software deployment and engineering practices." Some cybersecurity experts are questioning why certain safeguards weren’t in place before the tech outage.
“It’s easy to be an armchair expert, but there are best practices at play here that probably should have been in place sooner,” Sellitto said, adding that he gives CrowdStrike credit for their quick response to the outage.
Nikolas Behar, an adjunct professor of cybersecurity at the University of San Diego, said it was a surprise to see the outage tied to CrowdStrike – “one of the best, if not the best” cybersecurity firms in the country.
“They talked about how they're putting more checks into place in order to prevent this from happening again. But they were already supposed to have checks in the first place,” Behar said.
The U.S. House of Representatives Homeland Security Committee has sent a letter asking CrowdStrike CEO George Kurtz to testify on the outage.
“We cannot ignore the magnitude of this incident, which some have claimed is the largest IT outage in history,” the letter reads, adding that Americans will “undoubtedly feel the lasting, real-world consequences of this incident” and “deserve to know in detail how this incident happened and the mitigation steps CrowdStrike is taking.”
'Painful' wake-up call:What's next for CrowdStrike, Microsoft after update causes outage?
CrowdStrike said it plans to release a full analysis on the cause of Friday’s disruption once its investigation is complete. Experts who spoke to USA TODAY said they hope future reports shed more light on the decision-making process that allowed the bug to impact millions of devices.
“You hope that the producers are doing their due diligence. And I have to wait to see what their explanation is,” White of George Washington University said. “I don't care that you found the glitch. My problem is, why did the glitch hit the marketplace at all? And that's what seems to be missing here.”
Reuters contributed to this report.