Host provisioning in AWS zones are failing

Resolved

Degraded performance

Started 3 months ago Lasted about 8 hours

Affected

Zones

dev.aws-us-east-1c

perf.aws-us-east-1c

test.aws-us-east-1c

test.gcp-us-central1-f

staging.aws-us-east-1c

Updates

Resolved
March 02, 2024 at 01:45
Resolved
March 02, 2024 at 01:45
This incident has been resolved.
Monitoring
March 02, 2024 at 01:28
Monitoring
March 02, 2024 at 01:28
The fix has been pushed out to production, and the system stabilizes. We will continue to monitor for some time to ensure everything is working as expected.
Identified
March 02, 2024 at 00:47
Update
March 02, 2024 at 00:47
Fully rolling out the fix to production is ongoing, but it is taking longer than anticipated. We will report back within the hour with another update.
Identified
March 01, 2024 at 23:36
Update
March 01, 2024 at 23:36
We are currently rolling out a fix that is expected to resolve this issue. The rollout is estimated to take 1 hour.
Identified
March 01, 2024 at 22:40
Update
March 01, 2024 at 22:40
We are continuing to implement a fix. We will provide another update within 1 hour.
Identified
March 01, 2024 at 17:55
Identified
March 01, 2024 at 17:55
We have identified the underlying cause of the issue and are currently implementing a fix.