- Monitoring · UTC
A quick update from the team on this issue.
What have we done?
We have added new Vespa Zones in Europe for GCP test and staging environments.
The new zones appear to have much better capacity than the US Zone we used before, so “Node Allocation Failure” errors should be much rarer.
What should you do?
Good news, for most customers: nothing!
The new zone is enabled by default for all Vespa Cloud customers, so you don’t need to do anything.
Vespa Cloud: Enclave customers should update the Terraform module and contact us through the support portal to start using the zone for test and staging.
This issue applies to GCP only. We’ll keep monitoring and will mark this resolved once we’re confident the failures have stopped. If you still see this error, retry the run or contact support.
- Identified · UTC
Some deployments to our GCP (Google Compute Cloud) test and staging environments have been failing now and then with “Node Allocation Failure”. Google Cloud has limited compute capacity in the affected availability zone, so new machines sometimes can’t be provisioned in time. Your application isn’t causing these failures, and production applications that are already running are not affected.
We have made several provisioning improvements that reduce how often this happens. We are also setting up new test and staging zones with more capacity, which we expect to be ready soon. We will update this page when they are.
Until then, retrying a failed run usually works. Using the default resources for test and staging, rather than pinning a specific architecture or machine size, also lowers the chance of hitting the error.
If a change is urgent, you can use “Skip tests” in the Console to deploy to production.
We’re sorry for the disruption. If you are still blocked, contact support@vespa.ai.
Console - Operational
Infrastructure - Operational
AWS - Operational
AWS
GCP - Operational
GCP