Vespa Cloud - Application deployment may fail due to certificate provisioning – Incident details

All systems operational

Application deployment may fail due to certificate provisioning

Resolved
Degraded performance
Started about 2 years agoLasted 3 days

Affected

Infrastructure

Degraded performance from 8:20 AM to 5:15 PM, Operational from 5:15 PM to 7:50 AM, Degraded performance from 7:50 AM to 1:41 PM

Updates
  • Resolved
    UTC
    Resolved

    We have not seen any certificate issues the last day, resolving.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

    Please reach out to support if you are experiencing issues with deployments

  • Update
    UTC
    Update
    We are continuing to work on a fix for this incident.
  • Identified
    UTC
    Identified
    We are continuing to work on a fix for this incident.
  • Resolved
    UTC
    Resolved

    We have now resolved this issue, thanks for hanging in there

  • Monitoring
    UTC
    Monitoring

    Some certificate provisioning requests may fail at this time. This will appear as various errors deploying applications in Vespa Cloud.

    We have deployed a fix and we are working to stabilise the system and work through its backlog of certificate requests.

  • Update
    UTC
    Update

    We are continuing to work on a fix for this incident. A side effect has appeared now.

    Deployments are failing with 'UNEXPECTED ERROR', due to 'Endpoints failed to show up within 15 minutes!'

  • Identified
    UTC
    Identified

    We have identified the problem and will deploy a patch as soon as possible

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

    In the deployment view this will be shown as
    "Unexpected exception running run 474 of <zone> for <tenant>.<application>: Timed out after waiting PT5M to acquire lock '/controller/v1/locks/certificatePool'"