Vespa Cloud - System Test and Staging Test runs is not finishing – Incident details

All systems operational

System Test and Staging Test runs is not finishing

Resolved
Partial outage
Started over 2 years agoLasted 2 days

Affected

Console

Operational from 2:25 PM to 4:30 PM

Infrastructure

Partial outage from 2:25 PM to 4:30 PM

Zones

Partial outage from 2:25 PM to 4:30 PM

test

Partial outage from 2:25 PM to 4:30 PM

staging.aws-us-east-1c

Partial outage from 2:25 PM to 4:30 PM

staging.gcp-us-central1-f

Partial outage from 2:25 PM to 4:30 PM

Updates
  • Resolved
    UTC
    Resolved

    The patch applied to address the issue from Thursday, May 23 was not active on one of our controllers, this would cause System Test and Staging Test runs to fail some times. We have reapplied the patch now.

  • Monitoring
    UTC
    Monitoring

    We have applied a workaround for this issue. All current application have been pinned to vespa-8.345.20

  • Identified
    UTC
    Identified

    The fix for the previous incident, https://status.vespa.ai/clwjjxvt07700bvoom119zon5, was not fully successful, as it failed the System Test and Staging Test runs for applications deployed before the incident