At 07:48 UTC on 20 October 2025, customers began seeing errors from DynamoDB in Amazon's us-east-1 region in Northern Virginia. Over the next fourteen hours a large share of the consumer internet went with it.
AWS published a post-incident report, and the cause is worth understanding precisely, because it is not the kind of failure most people imagine when a cloud region goes down.
What actually broke
Nothing was overloaded. No hardware failed. No one was attacked.
DynamoDB's regional endpoint — dynamodb.us-east-1.amazonaws.com — is published through an automated DNS management system. Two DNS Enactor processes ran at the same time. A stale-plan check failed to catch that one of them was working from an outdated plan, so the old plan overwrote the newer one. Cleanup automation then saw the superseded plan and deleted it.
The result was an empty DNS record for the endpoint. Not a wrong address — no address at all. And because the automation had already done what it believed was correct, it did not repair the gap.
AWS called it a latent race condition: a bug that had existed all along and required two processes to overlap in a specific order to appear.
Why a DNS record took down so much
DynamoDB is not just a customer-facing database. A great deal of AWS's own machinery depends on it, which is why the failure spread sideways rather than staying inside one service.
New EC2 instance launches, Lambda invocations and Fargate task launches were all affected. Anything that needed to start something new — which includes most systems' automatic response to trouble — could not.
And us-east-1 is not an ordinary region. It is the oldest, the largest, and the default in countless tutorials, templates and configuration files. A disproportionate share of the internet is there because it was easiest to leave it there.
The part that explains the fourteen hours
This is the detail most coverage missed, and it is the most instructive.
The DNS problem itself was fixed in about three hours. Full DNS information was restored by 09:25 UTC, and clients recovered between 09:25 and 09:40 as cached records expired.
But the outage ran roughly fourteen hours, because a second system failed during the recovery. EC2's DropletWorkflow Manager entered congestive collapse — the state where a system receiving a backlog of retries spends all its capacity on retrying rather than completing work, so load stays at maximum and throughput approaches zero. Recovering network state took more than five further hours.
That is the characteristic shape of a modern large-scale outage. The original fault is often small and quickly fixed. The duration comes from what the restart does to everything downstream, as thousands of clients that have been failing and retrying for hours all reconnect at once.
What follows from it
Three things, and none of them is "avoid the cloud".
- Single-region is a decision, even when nobody made it. Most of the services that went down could have run in more than one region. They did not because us-east-1 was the default and multi-region costs money and effort that is invisible until a day like this.
- Retry behaviour is part of your reliability, not your competitor's. Congestive collapse is caused by clients. Exponential backoff with jitter, and circuit breakers that stop calling a service that is clearly down, are the difference between a system that recovers in minutes and one that extends somebody else's outage.
- Automation that repairs is also automation that can delete. The cleanup process here did exactly what it was built to do. Systems that modify production state need the same guards as a person doing it by hand — and race conditions are precisely the class of bug that testing finds last.
Sources: Amazon's post-incident report; The Register; ThousandEyes' outage analysis; InfoQ; Computerworld; and Prothom Alo's original report.
