Google’s Sunday Service Outage Caused by an “Incorrectly Applied” Server Configuration

Google Cloud Outage

Yesterday’s Google Fi outage wasn’t the only major outage Google services suffered this week. On Sunday, a somewhat major outage took out Gmail, YouTube, Google Drive, and other services for a number of hours for millions of users. Google blamed “network congestion” for the down time at first, but has now released official findings.

According to the Google Cloud blog, the disruption was caused by an “incorrectly applied” server configuration. The configuration was supposed to be applied to a small number of servers in a specific region, but was applied to a larger number across several regions instead. That configuration limited network capacity, hence the congestion explanation. Google says they identified the issue within seconds, but it took much longer to fix the problem.

How many people were impacted? While not a single DL reader reached out to tell us that they were impacted, Google did see a 2.5% drop in views from YouTube for one hour and a 30% reduction in traffic across Google Cloud Storage. They estimate that 1% of Gmail users were affected, which sounds small, but still means millions of people.

Google says that it is taking steps to be “guarded against the entire class of issues illustrated by this event.”

// Google Cloud Blog

This post was last modified on June 4, 2019 10:00 am