Vespa Cloud - Host provisioning in AWS zones are failing – Incident details

All systems operational

Host provisioning in AWS zones are failing

Resolved
Degraded performance
Started 11 months agoLasted about 8 hours

Affected

Zones

Degraded performance from 5:55 PM to 1:45 AM

dev.aws-us-east-1c

Degraded performance from 5:55 PM to 1:45 AM

perf.aws-us-east-1c

Degraded performance from 5:55 PM to 1:45 AM

test.aws-us-east-1c

Degraded performance from 5:55 PM to 1:45 AM

test.gcp-us-central1-f

Degraded performance from 5:55 PM to 1:45 AM

staging.aws-us-east-1c

Degraded performance from 5:55 PM to 1:45 AM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The fix has been pushed out to production, and the system stabilizes. We will continue to monitor for some time to ensure everything is working as expected.

  • Update
    Update

    Fully rolling out the fix to production is ongoing, but it is taking longer than anticipated. We will report back within the hour with another update.

  • Update
    Update

    We are currently rolling out a fix that is expected to resolve this issue. The rollout is estimated to take 1 hour.

  • Update
    Update

    We are continuing to implement a fix. We will provide another update within 1 hour.

  • Identified
    Identified

    We have identified the underlying cause of the issue and are currently implementing a fix.