A single computer file unintentionally disrupted 20% of the internet yesterday.

Yesterday’s disruption highlighted the extent to which the contemporary web relies on a limited number of essential infrastructure providers.

In fact, the reliance is so significant that a solitary configuration mistake rendered vast sections of the internet completely inaccessible for several hours.

Many individuals engage in the crypto sector due to their awareness of the risks associated with centralization in finance, but the incidents of yesterday served as a stark reminder that centralization at the internet’s foundation is an equally pressing issue that needs addressing.

The prominent players such as Amazon, Google, and Microsoft manage substantial portions of cloud infrastructure.

Equally important are companies like Cloudflare, Fastly, Akamai, DigitalOcean, and CDN (servers that enhance website delivery speed globally) or DNS (the internet’s “address book”) providers like UltraDNS and Dyn.

Most individuals are hardly familiar with their names, yet their outages can be just as debilitating, as demonstrated yesterday.

To begin, here’s a compilation of companies you might not recognize that are vital for maintaining the internet’s expected functionality.

Category	Company	What They Control	Impact If They Go Down
Core Infra (DNS/CDN/DDoS)	Cloudflare	CDN, DNS, DDoS protection, Zero Trust, Workers	Significant portions of global web traffic fail; numerous sites become inaccessible.
Core Infra (CDN)	Akamai	Enterprise CDN for banks, logins, commerce	Critical enterprise services, banks, and login systems fail.
Core Infra (CDN)	Fastly	CDN, edge compute	Potential global outage (as observed in 2021: Reddit, Shopify, gov.uk, NYT).
Cloud Provider	AWS	Compute, hosting, storage, APIs	SaaS applications, streaming services, fintech, and IoT networks fail.
Cloud Provider	Google Cloud	YouTube, Gmail, enterprise backends	Widespread disruption across Google services and dependent applications.
Cloud Provider	Microsoft Azure	Enterprise & government clouds	Outages for Office365, Teams, Outlook, and Xbox Live.
DNS Infrastructure	Verisign	.com & .net TLDs, root DNS	Severe global routing failures for significant portions of the web.
DNS Providers	GoDaddy / Cloudflare / Squarespace	DNS management for millions of domains	Entire companies disappear from the internet.
Certificate Authority	Let’s Encrypt	TLS certificates for most of the web	Global HTTPS failures; users encounter security errors everywhere.
Certificate Authority	DigiCert / GlobalSign	Enterprise SSL	Large corporate websites lose HTTPS trust.
Security / CDN	Imperva	DDoS, WAF, CDN	Protected sites become inaccessible or exposed.
Load Balancers	F5 Networks	Enterprise load balancing	Banking, hospitals, and government services may fail nationwide.
Tier-1 Backbone	Lumen (Level 3)	Global internet backbone	Routing issues lead to global latency spikes and regional outages.
Tier-1 Backbone	Cogent / Zayo / Telia	Transit and peering	Regional or national internet disruptions.
App Distribution	Apple App Store	iOS app updates & installations	iOS app ecosystem effectively halts.
App Distribution	Google Play Store	Android app distribution	Android apps cannot install or update globally.
Payments	Stripe	Web payments infrastructure	Thousands of applications lose the ability to process payments.
Identity / Login	Auth0 / Okta	Authentication & SSO	Logins fail for thousands of applications.
Communications	Twilio	2FA SMS, OTP, messaging	A significant portion of global 2FA and OTP codes fail.

What happened yesterday

The issue yesterday was caused by Cloudflare, a company that manages nearly 20% of all web traffic.

It has since stated that the outage originated from a minor database configuration adjustment that inadvertently caused a bot-detection file to contain duplicate entries.

This file unexpectedly exceeded a strict size limit. When Cloudflare’s servers attempted to load it, they failed, resulting in many websites utilizing Cloudflare returning HTTP 5xx errors (error codes displayed to users when a server malfunctions).

Here’s the straightforward sequence:

Chain of events

A Small Database Tweak Sets Off a Big Chain Reaction.

The complications began at 11:05 UTC when a permissions update caused the system to pull additional, duplicate data while constructing the file used to evaluate bots.

This file typically contains around sixty items. The duplicates pushed it beyond a strict limit of 200. When machines across the network attempted to load the oversized file, the bot component failed to initialize, leading to server errors.

According to Cloudflare, both the current and previous server paths were impacted. One returned 5xx errors, while the other assigned a bot score of zero, which could have incorrectly flagged traffic for clients who block based on bot scores (Cloudflare’s bot versus human detection).

Diagnosing the issue was challenging because the faulty file was rebuilt every five minutes from a database cluster being updated incrementally.

If the system pulled from an updated segment, the file was faulty. If not, it was functional. The network would recover, then fail again as versions changed.

Cloudflare noted that this on-off pattern initially resembled a potential DDoS, particularly since a third-party status page also failed around the same time. Attention shifted once teams connected the errors to the bot-detection configuration.

By 13:05 UTC, Cloudflare implemented a bypass for Workers KV (login checks) and Cloudflare Access (authentication system), rerouting around the failing behavior to minimize impact.

The primary resolution occurred when teams ceased generating and distributing new bot files, deployed a known good file, and restarted core servers.

Cloudflare reported that core traffic began flowing by 14:30, and all downstream services were restored by 17:06.

The failure highlights some design tradeoffs.

Cloudflare’s systems impose strict limits to maintain predictable performance. This approach helps prevent excessive resource usage, but it also means that a malformed internal file can trigger a hard stop instead of a graceful fallback.

Since bot detection operates on the main path for many services, the failure of one module cascaded into the CDN, security features, Turnstile (CAPTCHA alternative), Workers KV, Access, and dashboard logins. Cloudflare also observed increased latency as debugging tools consumed CPU while providing context for errors.

On the database side, a minor permissions adjustment had extensive repercussions.

The change allowed the system to “see” more tables than previously. The task that constructs the bot-detection file did not filter tightly enough, resulting in the inclusion of duplicate column names and expanding the file beyond the 200-item limit.

The loading error then triggered server failures and 5xx responses on the affected paths.

Impact varied by product. Core CDN and security services generated server errors.

Workers KV experienced elevated 5xx rates because requests to its gateway traversed the failing path. Cloudflare Access encountered authentication failures until the 13:05 bypass, and dashboard logins failed when Turnstile could not load.

Cloudflare Email Security temporarily lost an IP reputation source, diminishing spam detection accuracy for a period, although the company stated there was no critical customer impact. After the good file was restored, a backlog of login attempts briefly strained internal APIs before normalizing.

The timeline is straightforward.

The database change was implemented at 11:05 UTC. The first customer-facing errors emerged around 11:20–11:28.

Teams initiated an incident at 11:35, applied the Workers KV and Access bypass at 13:05, halted the propagation of bad files around 14:24, deployed a known good file, and observed global recovery by 14:30, marking full restoration at 17:06.

Cloudflare reported that automated tests identified anomalies at 11:31, and manual investigation commenced at 11:32, which accounts for the shift from a suspected attack to configuration rollback within two hours.

Time (UTC)	Status	Action or Impact
11:05	Change deployed	Database permissions update led to duplicate entries
11:20–11:28	Impact starts	HTTP 5xx surge as the bot file exceeds the 200-item limit
13:05	Mitigation	Bypass for Workers KV and Access reduces error surface
13:37–14:24	Rollback prep	Stop bad file propagation, validate known good file
14:30	Core recovery	Good file deployed, core traffic routes normally
17:06	Resolved	Downstream services fully restored

The numbers explain both cause and containment.

A five-minute rebuild cycle repeatedly reintroduced faulty files as different database segments updated.

A 200-item cap safeguards memory usage, and a typical count near sixty provided ample headroom until the duplicate entries emerged.

The cap functioned as intended, but the absence of a tolerant “safe load” for internal files transformed a bad configuration into a crash rather than a soft failure with a fallback model. Cloudflare indicated that this is a critical area for improvement.

Cloudflare stated it will enhance how internal configurations are validated, introduce more global kill switches for feature pipelines, prevent error reporting from consuming excessive CPU during incidents, review error handling across modules, and refine how configurations are distributed.

The company described this as its most significant incident since 2019 and expressed regret for the impact. According to Cloudflare, there was no attack; recovery was achieved by halting the faulty file, restoring a known good file, and restarting server processes.

The post How a single computer file accidentally took down 20% of the internet yesterday – in plain English appeared first on CryptoSlate.