Facebook's six hours offline: when a company withdraws its own routes
A maintenance command cut Facebook's backbone, its DNS servers withdrew their own BGP routes as designed, and the company vanished from the internet for hours.

Most large outages make a service slow, or broken, or partly available. On 4 October 2021, Facebook, Instagram and WhatsApp did something stranger. They stopped existing.
Not unreachable in the sense of a server returning an error. The names did not resolve. The address ranges had no route. To a computer anywhere else on the internet, Facebook was indistinguishable from a company that had never been connected at all. Even internal systems were affected: reporting at the time described staff unable to use their own tools, and in some cases unable to badge into buildings.
The company published two accounts, a short one the same day and a technical one the next. They are worth reading because the sequence they describe is both simple and very hard to defend against.
What Facebook says happened
During routine maintenance, an engineer issued a command intended to assess the capacity of the global backbone, the private network connecting Facebook's data centers to each other. The command unintentionally disconnected all backbone connections, isolating every data center.
Facebook says auditing tools exist specifically to block commands with that effect, and that a bug in the audit tool prevented it from stopping this one.
So far this is a bad but survivable internal network failure. The part that turned it into a global disappearance is what happened next, and it was entirely by design.
Facebook's DNS servers withdraw their BGP route advertisements when they cannot reach the data centers, on the reasonable basis that a DNS server unable to reach the systems it answers for should take itself out of service. Every DNS server independently evaluated the same condition, reached the same correct conclusion, and withdrew. The servers themselves were running fine. They had simply, and unanimously, told the internet to stop sending them traffic.
With no route to the DNS servers, there was no way to resolve any Facebook name, and therefore no way to reach anything.
The observed timeline
Facebook did not publish times. Cloudflare, watching from outside, did:
- Around 15:40 UTC routing changes from Facebook's network peak.
- Around 15:50 UTC facebook.com stops resolving on Cloudflare's 1.1.1.1 service.
- 15:58 UTC Cloudflare observes that Facebook has stopped announcing the routes to its DNS prefixes.
- Around 21:00 UTC renewed BGP activity begins, peaking at 21:17.
- 21:20 UTC facebook.com resolves again.
That is about five and a half hours of DNS unavailability. Figures of six hours and seven hours both circulate depending on what is being measured and from where, which is a good illustration of why external observation and vendor measurement should not be mixed casually.
Why it took so long to fix
This is the part most worth learning from, because the recovery problems were self-inflicted in a way that is extremely common.
Facebook's normal network access was down, because the backbone was down. Its out-of-band access was also down. Out-of-band management exists for precisely this scenario, and here it shared enough fate with the primary network that it went with it. Engineers had to physically travel to data centers.
Once there, they ran into their own security controls. Facebook's write-up is candid about this: physical and system security at the facilities is designed to make access and hardware modification difficult, and the company says it accepts that trade-off for everyday security. On the day you need to reconfigure a router by hand, hardening is friction.
Meanwhile, the loss of DNS had broken many of the internal tools engineers would normally use to diagnose an outage. A company whose monitoring, ticketing, documentation and chat all resolve through its own DNS loses them all when its own DNS is unreachable.
Restoring service was itself delicate. Facebook says power usage in its data centers had dropped by tens of megawatts, and bringing everything back at once risked straining electrical systems and caches. Services were ramped up gradually, and the company credits earlier drills, which it calls storm exercises, with making that manageable.
Error, not attack
Facebook's account calls this an error of its own making and describes a faulty configuration change. Its initial post also said there was no evidence that user data was compromised as a result of the downtime.
That statement was necessary because of unfortunate timing. A dataset was being advertised online around the same period, and a separate whistleblower story was running, and speculation about an attack was abundant. The technical evidence available publicly, chiefly the pattern of BGP withdrawals observed by external monitors, is consistent with Facebook's account and inconsistent with an external attack on its routing, which would be unlikely to produce an orderly withdrawal of exactly the right prefixes by the organization's own autonomous systems.
The security consequence that did arise was collateral. Cloudflare reported resolvers worldwide handling around thirty times the usual query volume, as billions of clients and apps retried relentlessly. Every DNS operator on the internet absorbed load generated by one company's absence. Scams and phishing referencing the outage also followed, as they reliably do.
What is confirmed and what is not
Confirmed from Facebook: the maintenance command and its unintended effect, the audit tool bug, the DNS servers' withdrawal behavior, the loss of both normal and out-of-band access, the dependence of internal tools on the affected DNS, the power-ramp concern, and that no malicious activity was involved.
Not published: the text of the command, the identity or role of the person who ran it, the specific nature of the audit tool bug, and precise start and end times. Communications of the ACM reported afterwards that the command, the audit tool and the bug all remained uncertain, that Facebook declined to add anything beyond the blog posts, and that some experts questioned how central the audit bug really was to the outcome. That disagreement has never been resolved publicly, and it is worth knowing about rather than smoothing over: the company's own account is the only detailed account, and specialists have reasonable doubts about parts of it.
Also unpublished: any figure for revenue lost or users affected. Estimates exist and are modeled, not measured.
What to take from it
Test whether your out-of-band access is actually out of band. The question is not whether you have a management network. It is whether it shares power, transit, DNS, authentication or physical path with the thing it is supposed to rescue. The honest test is to disconnect the primary and see what you can still reach. Most organizations discover a shared dependency they had forgotten.
Put a floor under automated self-removal. Health-check-driven withdrawal is good engineering until every instance withdraws at once. The pattern worth adopting is a minimum-capacity rule: a system may remove some of itself but never all of itself, and a unanimous failure signal should be treated as suspicious rather than obeyed. The same lesson appears in load balancer failover designs after similar incidents.
Keep the tools you need during an incident off the infrastructure that might be the incident. Runbooks, contact lists, conference bridges, ticketing and authentication. If your incident process requires resolving a name in a zone you serve yourself, it has a circular dependency. The incident response guide covers how to structure that, and the practical test takes an afternoon.
Accept that physical security and emergency access are in tension, and decide the trade-off deliberately. Facebook made its choice explicitly and said so. The wrong outcome is making the choice by accident and discovering it at 4am.
Treat routing and DNS as assets with owners. BGP configuration and DNS zones are usually managed by a small number of people, documented informally, and absent from risk registers, while being the dependency that makes everything else reachable. Published operational guidance such as RFC 7454 and the MANRS initiative is a reasonable starting point for what good routing hygiene looks like, alongside general network security practice.
Run the drill. The one thing in Facebook's account that clearly went well was the power ramp, and the company attributes that to having rehearsed it. Rehearsal is the only part of resilience that reliably pays off, and the only part most organizations skip. A post-incident review of someone else's outage is a cheap way to find out which drill you are missing.
Message claiming to restore access after an outage? Account-recovery scams follow every high-profile platform failure. Paste the message or link into CyberWatch AI Scan for a free check.
The uncomfortable symmetry of this incident is that nothing malfunctioned except the audit tool. The command executed correctly. The backbone disconnected as instructed. The DNS servers assessed their health accurately and took the textbook action. Every component behaved as designed, and the design, taken as a whole, was capable of removing one of the largest networks on earth from the internet in about ten minutes with no way to undo it remotely.
Frequently asked questions
What caused the Facebook outage on 4 October 2021?
Facebook says a command issued during routine maintenance, intended to assess global backbone capacity, unintentionally disconnected all the connections in its backbone network and cut its data centers off from each other. Auditing tools exist to block commands like that, and a bug in the audit tool prevented it from stopping this one.
Was it a cyber attack?
No. Facebook's own write-up calls it an error of its own making and says the cause was a faulty configuration change rather than malicious activity. Its initial post also stated there was no evidence that user data was compromised as a result of the downtime.
Why did Facebook disappear from the internet entirely?
Facebook's DNS servers are designed to withdraw their BGP route advertisements if they cannot reach the data centers, because that normally indicates an unhealthy network connection. With the backbone gone, every DNS server concluded it should stop advertising, so the rest of the internet had no route to them and no way to look up any Facebook name.
What is BGP?
The Border Gateway Protocol is how networks on the internet tell each other which address ranges they can reach. If an organization stops advertising a range, the rest of the internet stops having a path to it. Withdrawing a route makes you unreachable just as effectively as unplugging a cable.
How long did it last?
Facebook has not published start and end times. Cloudflare, observing from outside, reported that Facebook stopped announcing its DNS prefix routes at 15:58 UTC, that facebook.com stopped resolving on its 1.1.1.1 service at around 15:50 UTC, that renewed routing activity began around 21:00 UTC, and that the name resolved again at 21:20 UTC. That is roughly five and a half to six hours.
Why did recovery take so long?
Facebook's normal network access and its out-of-band access were both down, so engineers had to travel to data centers in person. Physical and system security at those facilities deliberately makes rapid hardware access difficult. The loss of DNS also broke many of the internal tools engineers would normally use to investigate.
Why was bringing everything back a problem?
Facebook says its data centers had seen power usage fall by tens of megawatts, and reversing that suddenly risked straining electrical systems and caches. Services were restored gradually, helped by drills the company had run previously for exactly this kind of scenario.
Did the outage affect other companies?
Yes, indirectly. Cloudflare reported that resolvers worldwide handled around thirty times the usual volume of queries during the outage, as billions of clients and apps retried continuously. DNS infrastructure with no connection to Facebook absorbed that load.
Is withdrawing routes on failure a bad design?
Not in itself. Removing yourself from the internet when you cannot serve traffic correctly is usually the right behavior, and it is how automated failover works. The flaw was that the health signal every DNS server used was the same signal, so a single backbone failure produced a unanimous, simultaneous withdrawal with no way to override it remotely.
What is the main lesson?
That out-of-band management must genuinely be out of band, and that automated self-removal needs a floor. A system allowed to withdraw all of itself, based on a signal shared by every instance, has a single point of failure no matter how many instances there are.
Sources
- More details about the October 4 outage, Meta Engineering
- Update about the October 4th outage, Meta Engineering
- Understanding How Facebook Disappeared from the Internet, Cloudflare Blog
- What Happened to Facebook, Instagram, and WhatsApp?, KrebsOnSecurity
- What Caused the Facebook Outage?, Communications of the ACM
- RFC 7454, BGP Operations and Security, RFC Editor, Internet Engineering Task Force
- Mutually Agreed Norms for Routing Security (MANRS), Internet Society
- Managing Incidents, Google Site Reliability Engineering
- SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems, US National Institute of Standards and Technology


