Vespa Cloud - Notice history

All systems operational

Console - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2026
Oct 2026
Nov 2026

Infrastructure - Operational

99.78% uptime
Sep 2026 · 100.0%Oct · 99.35%Nov · 100.0%

Previous page

Next page

Sep 2026
Oct 2026
Nov 2026
100.0% uptime

AWS - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2026
Oct 2026
Nov 2026

GCP - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2026
Oct 2026
Nov 2026
100.0% uptime

Notice history

View current status

Nov 2026

Vespa zone impaired: dev.aws-euw1-az1
Resolved1 hour 19 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    We have restored the infrastructure in the affected Vespa Zone, we are monitoring the situation.

    If you encounter any issue deploying in this zone please contact support through the Vespa Cloud Support oportal.

  • InvestigatingUTC

    At the current time we are experiencing issues with deployments in the Vespa Zone: dev.aws-euw1-az1

    We are currently investigating this incident.

    Running applications are unaffected.

    All other zones are unaffected.

Oct 2026

Query timeouts for some applications
ResolvedPartial outage27 hours 23 minutes
  • ResolvedUTC
    This incident has been resolved.
  • UpdateUTC

    Service fully restored; a small share of metrics for some applications returns by about 16:00 UTC

  • UpdateUTC

    We still observe ~1.4% loss of metrics across some applications.

    Besides that, infrastructure has been recovered and no traffic/data loss was observed.

    We will publish next update at 12:00 UTC.

  • UpdateUTC

    Some metrics might be still missing in Grafana for some applications.

    We will keep monitoring the system until further update.

  • MonitoringUTC

    The change has been rolled back and affected nodes have received new certificates.

    We are monitoring while everything returns to normal.

  • InvestigatingUTC

    We are investigating elevated query latency and timeouts (HTTP 504) affecting some applications in Vespa Cloud.

    A change to internal certificate handling caused some nodes to lose connectivity within their application.

    We have rolled back the change and are restoring the affected nodes.

    Feeding and stored data are not affected.

Intermittent “Node Allocation Failure” in GCP test and staging environments
Resolved650 hours 1 minute
  • ResolvedUTC

    What have we done?

    We made several provisioning changes that reduce how often GCP test and staging deployments fail with “Node Allocation Failure”. We also added new Vespa Zones in Europe for GCP test and staging, which have much better capacity than the US zone we used before. The new zones are enabled by default for all Vespa Cloud customers.

    We are also working with Google to resolve the capacity limits in the original zone long term.

    What should you do?

    For most customers, nothing.

    Vespa Cloud: Enclave customers who haven’t yet updated the Terraform module and contacted us to start using the new zone should do so.

    If you still see this error, retry the run or contact support at support@vespa.ai. Production applications that were already running were not affected at any point.

    This issue applied to GCP only. We’re sorry for the disruption.

  • MonitoringUTC

    A quick update from the team on this issue.

    What have we done?

    We have added new Vespa Zones in Europe for GCP test and staging environments.

    The new zones appear to have much better capacity than the US Zone we used before, so “Node Allocation Failure” errors should be much rarer.

    What should you do?

    Good news, for most customers: nothing!

    The new zone is enabled by default for all Vespa Cloud customers, so you don’t need to do anything.

    Vespa Cloud: Enclave customers should update the Terraform module and contact us through the support portal to start using the zone for test and staging.

    This issue applies to GCP only. We’ll keep monitoring and will mark this resolved once we’re confident the failures have stopped. If you still see this error, retry the run or contact support.

  • IdentifiedUTC

    Some deployments to our GCP (Google Compute Cloud) test and staging environments have been failing now and then with “Node Allocation Failure”. Google Cloud has limited compute capacity in the affected availability zone, so new machines sometimes can’t be provisioned in time. Your application isn’t causing these failures, and production applications that are already running are not affected.

    We have made several provisioning improvements that reduce how often this happens. We are also setting up new test and staging zones with more capacity, which we expect to be ready soon. We will update this page when they are.

    Until then, retrying a failed run usually works. Using the default resources for test and staging, rather than pinning a specific architecture or machine size, also lowers the chance of hitting the error.

    If a change is urgent, you can use “Skip tests” in the Console to deploy to production.

    We’re sorry for the disruption. If you are still blocked, contact support@vespa.ai.

Sep 2026

No notices reported this month

Previous

Sep 2026 to Nov 2026

Next