Google Compute Engine Incident #19012
Issues with PD-SSD in us-east1-b zone
Incident began at 2019-12-07 10:46 (all times are US/Pacific).
DATE TIME DESCRIPTION
Dec 07, 2019 13:03
Description: Mitigation work is still underway by our engineering team.
Current data indicates that approximately 0.01% of PD-SSD volumes in us-east1-b are affected by this issue, from a peak of approximately 8.45%.
At this time, the issue is contained, and we can confirm that this is only impacting us-east1-b.
We will provide more information by Saturday, 2019-12-07 14:02 US/Pacific.
Diagnosis: High IO wait times, failed IO operations, or IO Operations taking over 2 seconds to complete
Workaround: None at this time
Dec 07, 2019 11:55
Description: Mitigation work is currently underway by our engineering team.
We do not have an ETA for mitigation at this point.
We will provide more information by Saturday, 2019-12-07 12:42 US/Pacific.
Diagnosis: Seeing High IO wait times, failed IO operations, or IO Operations taking over 2 seconds to complete
Workaround: None at this time
Dec 07, 2019 11:30
Description: We are experiencing an issue with SSD Persistent Disk and SSD Regional Persistent Disks,beginning at Saturday, 2019-12-07 06:45 US/Pacific.
Symptoms: High IO wait times, IO operations failing or taking over 2 seconds to complete.
Self-diagnosis: Seeing High IO wait times, failed IO operations, or IO Operations taking over 2 seconds to complete
Workaround: N/A
Our engineering team continues to investigate the issue.
We will provide an update by Saturday, 2019-12-07 12:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Seeing High IO wait times, failed IO operations, or IO Operations taking over 2 seconds to complete
Workaround: None at this time