We’ve got more on the Amazon issues this week

Earlier this week, we learned through KNX about an Amazon issue that ended up affecting affecting many services including their own.

Now, we’re learning that this DNS issue was caused by a race condition, and those types of conditions are not necessarily a good thing.

If you want to read their reporting that was earlier this week, please read the article Amazon outage will cost company “reputational damage,” tech expert says which was breaking at the time.

Amazon did a post mortem report, and that’s what this this article will be talking about.

The article says:

According to a post-mortem published on Thursday, a race condition caused a major DNS failure in Amazon DynamoDB’s infrastructure, specifically within its DNS management system that controls how user requests are routed to healthy servers, which led to the accidental deletion of all IP addresses for the database service’s regional endpoint.

The article continues:

“The root cause of this issue was a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service’s regional endpoint (dynamodb.us-east-1.amazonaws.com) that the automation failed to repair,” Amazon said.

“When this issue occurred at 11:48 PM PDT, all systems needing to connect to the DynamoDB service in the N. Virginia (us-east-1) Region via the public endpoint immediately began experiencing DNS failures and failed to connect to DynamoDB. This included customer traffic as well as traffic from internal AWS services that rely on DynamoDB.”

The DynamoDB failure triggered cascading problems across AWS infrastructure, leaving DynamoDB’s DNS system in an inconsistent state that automated recovery couldn’t fix, requiring manual operator intervention.

While the article isn’t long, we’ll let you hear and read the entire article now.

The article is titled Amazon: This week’s AWS outage caused by major DNS failure if that’s what you want to read.