Vespa Cloud - Query timeouts for some applications – Incident details

All systems operational

Show current status

Query timeouts for some applications

ResolvedPartial outage 30%27 hours 23 minutes

Affected

Infrastructure

Partial outage from 3:46 PM to 7:57 AM, Under maintenance from 7:57 AM to 7:10 PM

  • ResolvedUTC
    This incident has been resolved.
  • UpdateUTC

    Service fully restored; a small share of metrics for some applications returns by about 16:00 UTC

  • UpdateUTC

    We still observe ~1.4% loss of metrics across some applications.

    Besides that, infrastructure has been recovered and no traffic/data loss was observed.

    We will publish next update at 12:00 UTC.

  • UpdateUTC

    Some metrics might be still missing in Grafana for some applications.

    We will keep monitoring the system until further update.

  • MonitoringUTC

    The change has been rolled back and affected nodes have received new certificates.

    We are monitoring while everything returns to normal.

  • InvestigatingUTC

    We are investigating elevated query latency and timeouts (HTTP 504) affecting some applications in Vespa Cloud.

    A change to internal certificate handling caused some nodes to lose connectivity within their application.

    We have rolled back the change and are restoring the affected nodes.

    Feeding and stored data are not affected.