InfoSecBulletin Cybersecurity for mankind
Yesterday, Cloudflare outage disrupted major platforms like ChatGPT, X, Uber, Canva, Downdetector and some other services, leaving users with 500 Internal Server Errors. Cloudflare traced the issue to an oversized configuration file that affected its network. A fix was implemented by around 9:42 a.m. ET, leading to a gradual recovery, although some users were facing ongoing issues. Cloudflare confirmed the outage was due to an internal configuration error, not a cyberattack.
The outage analysis:
Researchers show new technique to bypass AI safety guardrails in Grok and Gemini
About thousands of leaked AWS keys Held Full Admin Rights
US Bank investigates LockBit’s Data Breach Claims
Five new malware families actively targeting Asian Gov.t infra
T-Mobile Cuts Cables to Remove Chiness Salt Typhoon Hackers from Network
Splunk, Zyxel Patch Multiple Flaws Enabling RCE and Root Command Execution
“Zombie Card” attack revels expired Visa card may be used for contactless payments
Critical Zimbra RCE Flaw Actively Exploited in the Wild
Operation CameraSwarm
A single hacker compromise 1400+ Dahua camera worldwide
Cl0p Ransomware Listed 40+ Victims of PTC Windchill Campaign
Cloudflare spokesperson Jackie Dutton tells The Verge that, “The disruption, which started at around 6:20AM ET, was linked to a “configuration file that is automatically generated to manage threat traffic,”. “The file grew beyond an expected size of entries and triggered a crash in the software system that handles traffic for a number of Cloudflare’s services.” Dutton added that “there is no evidence” of an attack or other malicious activity.
Cybersecuritynews reported that “Cloudflare’s troubles stemmed from a routine update to permissions in its ClickHouse database cluster, intended to enhance security for distributed queries.
At 11:05 UTC, the change made underlying table metadata in the ‘r0’ database visible to users, but a Bot Management query failed to account for this, pulling duplicate column data and bloating a critical feature file to double its expected size.
This file, refreshed every five minutes to combat evolving bot threats via machine learning, overwhelmed the software’s hardcoded limit of 200 features, triggering panics in the core proxy system known as FL.
Initially mistaken for a massive DDoS attack coinciding with the downtime of Cloudflare’s external status page, the fluctuating failures puzzled investigators as good and bad files alternated during the cluster’s gradual rollout.
The Bot Management module, essential for scoring automated traffic, halted request processing, cascading errors through the network. In the newer FL2 proxy, this caused outright 5xx HTTP errors; older FL versions defaulted bot scores to zero, potentially blocking legitimate traffic for customers using bot-blocking rules.”
Cloudflare spokesperson said in a statement that, “The cause of the outage was a configuration file that is automatically generated to manage threat traffic. There was no evidence of a cyberattack or malicious activity. “The file grew beyond an expected size of entries and triggered a crash in the software system that handles traffic for a number of Cloudflare’s services,”.
“In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack,” Knecht wrote, referring to a bug that goes undetected in testing and has not caused a failure,” Cloudflare’s chief technology officer Dane Knecht wrote on Twitter.
How much is the Cloudflare outage costing?
Outages like this can lead to huge financial losses for businesses of all sizes. Website maintenance service provider, SupportMy.Website, told Cybernews the outage could be costing companies anywhere from $5 to $15 billion for every hour of downtime.
“Right now, our customers, from major banks to small mom-and-pop businesses, are struggling to do business and fulfill customer requests. From reputation to the bottom line, Cloudflare is one of those systems that businesses don’t realize they need or even use sometimes. But when it’s down, they feel it,” said Jason Long, founder of SupportMy.Website.
Parametrix Insurance estimates that Fortune 500 companies (excluding Microsoft) suffered $5.4 billion in direct losses due to the CrowdStrike outage in July 2024, which caused 8.5 million Windows devices to malfunction from a failed software patch.
