Vespa Cloud - New nodes can fail to provision - – Incident details

New nodes can fail to provision -

ResolvedDegraded performance6 hours 33 minutes

Affected

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    The rollout is compete and new nodes are able to provision.

    We are monitoring the situation and will post a final update later today.

  • UpdateUTC

    We have found an edge-case which has been slowing down the rollout, we're in the process of mitigating this issue.

    For now new hosts are still impacted.

    New update at 12:15 or earlier.

    There is no impact for existing nodes.

  • UpdateUTC

    Rollout is still in progress, still impacting new nodes on AWS.

    Existing nodes are not impacted.

  • UpdateUTC

    We have implemented a fix and it is currently rolling out fleet-wide.

    Provision of new nodes is still impacted, but we should resolve within 30 minutes, we will post an update as soon as possible, or by 11:00 UTC in any case.

    Existing nodes are not affected.

  • UpdateUTC

    We are continuing to work on a fix for this incident, we have noticed failures in more zones for new hosts.

    Ticket updated with the new Zones.

    Existing nodes are unaffected, only provisioning of new nodes is impacted.

  • IdentifiedUTC

    We have identified the root cause and started working on a workaround.

  • InvestigatingUTC

    We have observed that new nodes are failing to provision.

    Existing nodes are unaffected.