Vespa Cloud - Notice history

Console - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

Infrastructure - Operational

99% - uptime
May 2024 · 97.61%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024
100% - uptime

dev.aws-us-east-1c - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

dev.gcp-us-central1-f - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

perf.aws-us-east-1c - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

test.aws-us-east-1c - Operational

99% - uptime
May 2024 · 97.61%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

test.gcp-us-central1-f - Operational

99% - uptime
May 2024 · 97.61%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

staging.aws-us-east-1c - Operational

99% - uptime
May 2024 · 97.61%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

staging.gcp-us-central1-f - Operational

99% - uptime
May 2024 · 97.61%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

prod.aws-us-east-1c - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

prod.aws-us-west-2a - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

prod.aws-eu-west-1a - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

prod.ap-northeast-1a - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

prod.gcp-europe-west3-b - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

prod.gcp-us-central1-f - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024
100% - uptime

AWS - Operational

99% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 97.58%
May 2024
Jun 2024
Jul 2024

GCP - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%
May 2024
Jun 2024
Jul 2024

Notice history

Jul 2024

Application deployment may fail due to certificate provisioning
  • Resolved
    Resolved

    We have not seen any certificate issues the last day, resolving.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

    Please reach out to support if you are experiencing issues with deployments

  • Update
    Update
    We are continuing to work on a fix for this incident.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Resolved
    Resolved

    We have now resolved this issue, thanks for hanging in there

  • Monitoring
    Monitoring

    Some certificate provisioning requests may fail at this time. This will appear as various errors deploying applications in Vespa Cloud.

    We have deployed a fix and we are working to stabilise the system and work through its backlog of certificate requests.

  • Update
    Update

    We are continuing to work on a fix for this incident. A side effect has appeared now.

    Deployments are failing with 'UNEXPECTED ERROR', due to 'Endpoints failed to show up within 15 minutes!'

  • Identified
    Identified

    We have identified the problem and will deploy a patch as soon as possible

  • Investigating
    Investigating

    We are currently investigating this incident.

    In the deployment view this will be shown as
    "Unexpected exception running run 474 of <zone> for <tenant>.<application>: Timed out after waiting PT5M to acquire lock '/controller/v1/locks/certificatePool'"


May 2024

System Test and Staging Test runs is not finishing
  • Resolved
    Resolved

    The patch applied to address the issue from Thursday, May 23 was not active on one of our controllers, this would cause System Test and Staging Test runs to fail some times. We have reapplied the patch now.

  • Monitoring
    Monitoring

    We have applied a workaround for this issue. All current application have been pinned to vespa-8.345.20

  • Identified
    Identified

    The fix for the previous incident, https://status.vespa.ai/clwjjxvt07700bvoom119zon5, was not fully successful, as it failed the System Test and Staging Test runs for applications deployed before the incident

May 2024 to Jul 2024

Next