Vespa Cloud - Query timeouts for some applications – Incident details

All systems operational

Query timeouts for some applications

Resolved
Partial outage 30 %
Started 10 days agoLasted 27 hours 23 minutes

Affected

Infrastructure

Partial outage from 3:46 PM to 7:57 AM, Under maintenance from 7:57 AM to 7:10 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    Service fully restored; a small share of metrics for some applications returns by about 16:00 UTC

  • Update
    UTC
    Update

    We still observe ~1.4% loss of metrics across some applications.

    Besides that, infrastructure has been recovered and no traffic/data loss was observed.

    We will publish next update at 12:00 UTC.

  • Update
    UTC
    Update

    Some metrics might be still missing in Grafana for some applications.

    We will keep monitoring the system until further update.

  • Monitoring
    UTC
    Monitoring

    The change has been rolled back and affected nodes have received new certificates.

    We are monitoring while everything returns to normal.

  • Investigating
    UTC
    Investigating

    We are investigating elevated query latency and timeouts (HTTP 504) affecting some applications in Vespa Cloud.

    A change to internal certificate handling caused some nodes to lose connectivity within their application.

    We have rolled back the change and are restoring the affected nodes.

    Feeding and stored data are not affected.