Vespa Cloud - Missing metrics from on GCP and Enclave – Incident details

All systems operational

Missing metrics from on GCP and Enclave

Resolved
Partial outage
Started about 1 month agoLasted about 4 hours

Affected

Zones

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

dev.gcp-us-central1-f

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

test.gcp-us-central1-f

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

staging.gcp-us-central1-f

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

prod.gcp-europe-west3-b

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

prod.gcp-us-central1-f

Partial outage from 7:36 AM to 9:19 AM, Operational from 9:19 AM to 11:56 AM

Updates
  • Resolved
    Resolved

    We can confirm that all the systems have been successfully working since our workaround is applied and the final fix has been applied.

  • Monitoring
    Monitoring

    We have applied a workaround and we see the data metrics flowing back in our systems.

    We are now monitoring the situation.

  • Identified
    Identified

    We have identified one problem with our telemetry gateway and are working on a fix.

  • Investigating
    Investigating
    We are currently investigating this incident.