Vespa Cloud - Application deployment may fail due to certificate provisioning – Incident details

Application deployment may fail due to certificate provisioning

ResolvedDegraded performance77 hours 21 minutes

Affected

Infrastructure

Degraded performance from 8:20 AM to 5:15 PM, Operational from 5:15 PM to 7:50 AM, Degraded performance from 7:50 AM to 1:41 PM

  • ResolvedUTC

    We have not seen any certificate issues the last day, resolving.

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result.

    Please reach out to support if you are experiencing issues with deployments

  • UpdateUTC
    We are continuing to work on a fix for this incident.
  • IdentifiedUTC
    We are continuing to work on a fix for this incident.
  • ResolvedUTC

    We have now resolved this issue, thanks for hanging in there

  • MonitoringUTC

    Some certificate provisioning requests may fail at this time. This will appear as various errors deploying applications in Vespa Cloud.

    We have deployed a fix and we are working to stabilise the system and work through its backlog of certificate requests.

  • UpdateUTC

    We are continuing to work on a fix for this incident. A side effect has appeared now.

    Deployments are failing with 'UNEXPECTED ERROR', due to 'Endpoints failed to show up within 15 minutes!'

  • IdentifiedUTC

    We have identified the problem and will deploy a patch as soon as possible

  • InvestigatingUTC

    We are currently investigating this incident.

    In the deployment view this will be shown as
    "Unexpected exception running run 474 of <zone> for <tenant>.<application>: Timed out after waiting PT5M to acquire lock '/controller/v1/locks/certificatePool'"