Vespa Cloud - Missing metrics from on GCP and Enclave – Incident details

All systems operational

Missing metrics from on GCP and Enclave

Resolved
Partial outage
Started over 1 year agoLasted about 4 hours

Affected

Enclave

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

AWS

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

GCP

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

Zones

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

dev

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

dev.gcp-us-central1-f

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

Updates
  • Resolved
    UTC
    Resolved

    We can confirm that all the systems have been successfully working since our workaround is applied and the final fix has been applied.

  • Monitoring
    UTC
    Monitoring

    We have applied a workaround and we see the data metrics flowing back in our systems.

    We are now monitoring the situation.

  • Identified
    UTC
    Identified

    We have identified one problem with our telemetry gateway and are working on a fix.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.