New nodes can fail to provision -
Affected
- Resolved · UTCThis incident has been resolved.
- Monitoring · UTC
The rollout is compete and new nodes are able to provision.
We are monitoring the situation and will post a final update later today.
- Update · UTC
We have found an edge-case which has been slowing down the rollout, we're in the process of mitigating this issue.
For now new hosts are still impacted.
New update at 12:15 or earlier.
There is no impact for existing nodes.
- Update · UTC
Rollout is still in progress, still impacting new nodes on AWS.
Existing nodes are not impacted.
- Update · UTC
We have implemented a fix and it is currently rolling out fleet-wide.
Provision of new nodes is still impacted, but we should resolve within 30 minutes, we will post an update as soon as possible, or by 11:00 UTC in any case.
Existing nodes are not affected.
- Update · UTC
We are continuing to work on a fix for this incident, we have noticed failures in more zones for new hosts.
Ticket updated with the new Zones.
Existing nodes are unaffected, only provisioning of new nodes is impacted.
- Identified · UTC
We have identified the root cause and started working on a workaround.
- Investigating · UTC
We have observed that new nodes are failing to provision.
Existing nodes are unaffected.