Vespa Cloud - Notice history

All systems operational

Console - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Infrastructure - Operational

99% - uptime
May 2024 · 97.61%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
100% - uptime

AWS - Operational

99% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 97.58%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

GCP - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
100% - uptime

Notice history

Jul 2024

New deployments, and node increase/replacement might fail for enclave
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Update
    Update

    We are still investigating. Existing deployments are currently stable

  • Investigating
    Investigating
    We are currently investigating this incident.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We are currently investigating this incident.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The immediate issue has been resolved, we are monitoring the system

Application deployment may fail due to certificate provisioning
  • Resolved
    Resolved

    We have not seen any certificate issues the last day, resolving.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

    Please reach out to support if you are experiencing issues with deployments

  • Update
    Update
    We are continuing to work on a fix for this incident.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Resolved
    Resolved

    We have now resolved this issue, thanks for hanging in there

  • Monitoring
    Monitoring

    Some certificate provisioning requests may fail at this time. This will appear as various errors deploying applications in Vespa Cloud.

    We have deployed a fix and we are working to stabilise the system and work through its backlog of certificate requests.

  • Update
    Update

    We are continuing to work on a fix for this incident. A side effect has appeared now.

    Deployments are failing with 'UNEXPECTED ERROR', due to 'Endpoints failed to show up within 15 minutes!'

  • Identified
    Identified

    We have identified the problem and will deploy a patch as soon as possible

  • Investigating
    Investigating

    We are currently investigating this incident.

    In the deployment view this will be shown as
    "Unexpected exception running run 474 of <zone> for <tenant>.<application>: Timed out after waiting PT5M to acquire lock '/controller/v1/locks/certificatePool'"


Outdated cost per hour values in console
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    The cost per hour values seen in the application lists will display old values and are currently not being updated. The underlying issue has been identified and a fix is under development.

Jun 2024

Node provisioning in AWS is not working
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    During the release of vespa-8.362.16 today, we identified failing deployments to test and staging. This failed due to a change that broke provisioning of nodes from AWS.

    We are deploying a fix for this now

Console might not load
  • Resolved
    Resolved

    We have deployed a fix that should prevent this from happening again

  • Update
    Update

    This is caused by a changed to an un-versioned embedded file. We will address this as soon as possible

  • Identified
    Identified

    Some might get "You encountered a bug", with "TypeError: Cannot read properties of undefined (reading 'apiUrl') in the error trace. We will address this as soon as possible.

    To work around this issue do a forced refresh.

May 2024

System Test and Staging Test runs is not finishing
  • Resolved
    Resolved

    The patch applied to address the issue from Thursday, May 23 was not active on one of our controllers, this would cause System Test and Staging Test runs to fail some times. We have reapplied the patch now.

  • Monitoring
    Monitoring

    We have applied a workaround for this issue. All current application have been pinned to vespa-8.345.20

  • Identified
    Identified

    The fix for the previous incident, https://status.vespa.ai/clwjjxvt07700bvoom119zon5, was not fully successful, as it failed the System Test and Staging Test runs for applications deployed before the incident

System Test and Staging Test runs in new production deployments is not working
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    Due to a failure during the upgrade of our infrastructure today, the upgrade was incomplete and was causing issues for the System Test and Staging Test runs for applications. We have pinned all applications to their current vespa version to work around this.

    If you deploy a new application to production, you will be blocked by this. Please reach out to us if you see that your new application is failing deployment in the System Test and Staging Test runs.

May 2024 to Jul 2024

Next