On June 12, 2025, Google Cloud experienced a significant global outage that disrupted over 70 services, including Gmail, Google Meet, Google Drive, and third-party platforms such as Spotify, OpenAI, and Shopify. The disruption began around 1:50 p.m. ET and was identified within 10 minutes by Google’s Site Reliability Engineering team. However, full recovery took several hours due to the complexity of the issue.

Root Cause and Resolution

The outage was caused by a flawed update in Google’s Service Control system, which manages API requests and quota policies. A policy change on June 12 introduced unintended blank fields, leading to null pointer errors and subsequent crashes in Service Control binaries. Despite rapid identification of the issue, recovery was delayed due to infrastructure overloads and lack of appropriate error handling. Service was restored across most regions within two hours, but some areas experienced prolonged disruptions.

Apology and Future Improvements

Google issued a public apology for the impact of the outage and committed to several corrective actions:

Modularizing Service Control’s architecture to ensure isolated failures do not affect the entire system.
Implementing feature flags for all critical updates to allow gradual rollouts and early detection of issues.
Enhancing error handling and testing practices to prevent similar failures in the future.
Improving communication protocols to provide timely and accurate information to customers during incidents.
Auditing systems that rely on globally replicated data to ensure consistency and reliability.

The company also acknowledged the importance of maintaining operational monitoring systems even during outages to ensure business continuity.

wolfcanine

Google Apologizes for Global Cloud Outage Affecting Over 70 Services