Vespa Cloud - System Test and Staging Test runs in new production deployments is not working – Incident details

All systems operational

System Test and Staging Test runs in new production deployments is not working

Resolved
Partial outage
Started over 2 years agoLasted about 9 hours

Affected

Infrastructure

Partial outage from 10:00 AM to 7:02 PM, Operational from 7:02 PM to 6:00 AM

Zones

Partial outage from 10:00 AM to 7:02 PM, Operational from 7:02 PM to 7:26 AM

test

Partial outage from 10:00 AM to 7:02 PM, Operational from 7:02 PM to 7:26 AM

staging.aws-us-east-1c

Partial outage from 10:00 AM to 7:02 PM, Operational from 7:02 PM to 7:26 AM

staging.gcp-us-central1-f

Partial outage from 10:00 AM to 7:02 PM, Operational from 7:02 PM to 12:00 AM

test.aws-us-east-1c

Partial outage from 10:00 AM to 7:02 PM, Operational from 7:02 PM to 1:27 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    Due to a failure during the upgrade of our infrastructure today, the upgrade was incomplete and was causing issues for the System Test and Staging Test runs for applications. We have pinned all applications to their current vespa version to work around this.

    If you deploy a new application to production, you will be blocked by this. Please reach out to us if you see that your new application is failing deployment in the System Test and Staging Test runs.