Duration: 1 day, 3 hours and 45 minutes

Affected Components: Console

Jul 3, 10:00:00 GMT+0
Identified - Please confirm with the metrics view.

Jul 4, 13:44:30 GMT+0
Resolved - This incident has been resolved..

Jul 3, 21:04:45 GMT+0
Identified - We are continuing to work on a fix for this incident. ETA is unknown at the moment.

Jul 4, 09:44:32 GMT+0
Identified - We have startet the process to release the build with the fix to our systems.

Duration: 5 hours and 1 minute

Affected Components: Infrastructure

Jul 2, 23:05:46 GMT+0
Resolved - This incident has been resolved..

Jul 2, 18:05:00 GMT+0
Identified - We are waiting for an update from LetsEncrypt at https://letsencrypt.status.io/.

Duration: 42 minutes

Affected Components: Console

Jun 26, 07:00:34 GMT+0
Investigating - We have are received reports of elevated authentication issues and we are onvestigating..

Jun 26, 07:42:07 GMT+0
Resolved - This incident has been resolved..

Jun 26, 07:11:38 GMT+0
Identified - Follow the status of the issue on our service provider: <https://status.auth0.com/incidents/z8sc9m1gzqb7>.

Jun 26, 07:22:42 GMT+0
Monitoring - We have not observed any login issues in the past few minutes, but we are still monitoring the status..

Duration: 57 minutes

Affected Components: Console

Apr 17, 09:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Apr 17, 09:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 17, 09:57:01 GMT+0
Completed - Maintenance has completed successfully..

Apr 17, 09:57:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 3 hours and 38 minutes

Affected Components: Console

Apr 9, 12:44:00 GMT+0
Investigating - We are currently investigating this incident..

Apr 9, 14:50:18 GMT+0
Investigating - We are currently investigating this incident..

Apr 9, 16:22:23 GMT+0
Resolved - This incident has been resolved..

Affected Components: Console

Apr 8, 09:13:00 GMT+0
Investigating - We are currently investigating issues with authentication in Vespa Console.

Apr 8, 09:28:00 GMT+0
Resolved - This incident has been resolved..

Apr 8, 09:25:00 GMT+0
Identified - We are have identified the issue and proceeding with a fix.

Duration: 2 hours and 6 minutes

Affected Components: , AWS, staging.gcp-us-central1-f, prod.aws-eu-west-1a, prod.aws-us-west-2a, perf.aws-us-east-1c, Infrastructure, prod.aws-us-east-1c, prod.gcp-us-central1-f, test.aws-us-east-1c, dev.aws-euw1-az1, prod.aws-apne1-az1, prod.aws-cac1-az1, prod.aws-euw1-az1, prod.gcp-us-central1-c, prod.gcp-us-central1-a, GCP, staging.aws-us-east-1c, prod.aws-euc1-az1, prod.gcp-europe-west3-b, prod.gcp-us-central1-b, prod.aws-ap-northeast-1a, prod.aws-aps1-az1, prod.aws-euc1-az3, prod.aws-use1-az4, test.gcp-us-central1-f, , , , prod.aws-cac1-az2, , prod.aws-use2-az1, prod.aws-use2-az3, prod.aws-usw2-az3, , dev.aws-us-east-1c, dev.gcp-us-central1-f, Enclave → dev → prod → test → perf → Zones →

Apr 1, 00:50:14 GMT+0
Investigating - We are currently investigating this incident..

Apr 1, 02:22:24 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 1, 02:56:04 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 53 minutes

Affected Components: prod.gcp-europe-west3-b

Mar 27, 03:47:23 GMT+0
Investigating - prod - prod.gcp-europe-west3-b cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 27, 05:40:43 GMT+0
Resolved - This incident has been resolved..

Duration: 6 hours and 33 minutes

Affected Components: AWS, prod.aws-eu-west-1a, prod.aws-us-west-2a, perf.aws-us-east-1c, prod.aws-us-east-1c, test.aws-us-east-1c, dev.aws-euw1-az1, prod.aws-apne1-az1, prod.aws-cac1-az1, prod.aws-euw1-az1, staging.aws-us-east-1c, prod.aws-euc1-az1, prod.aws-ap-northeast-1a, prod.aws-aps1-az1, prod.aws-euc1-az3, prod.aws-use1-az4, prod.aws-cac1-az2, prod.aws-use2-az1, prod.aws-use2-az3, prod.aws-usw2-az3, dev.aws-us-east-1c

Mar 6, 07:30:00 GMT+0
Investigating - We have observed that new nodes are failing to provision. **Existing nodes are unaffected.**.

Mar 6, 09:07:35 GMT+0
Identified - We have identified the root cause and started working on a workaround..

Mar 6, 10:06:58 GMT+0
Identified - We are continuing to work on a fix for this incident, we have noticed failures in more zones for new hosts. _Ticket updated with the new Zones._ **Existing nodes are unaffected, only provisioning of new nodes is impacted.**.

Mar 6, 10:35:34 GMT+0
Identified - We have implemented a fix and it is currently rolling out fleet-wide. Provision of **new nodes** is still impacted, but we should resolve within 30 minutes, we will post an update as soon as possible, or by 11:00 UTC in any case. **Existing nodes are not affected.**.

Mar 6, 11:09:03 GMT+0
Identified - Rollout is still in progress, still impacting **new nodes on AWS**. **Existing nodes are not impacted.**.

Mar 6, 11:43:09 GMT+0
Identified - We have found an edge-case which has been slowing down the rollout, we're in the process of mitigating this issue. For now **new hosts** are still impacted. New update at 12:15 or earlier. There is **no impact for existing nodes.**.

Mar 6, 12:47:19 GMT+0
Monitoring - The rollout is compete and new nodes are able to provision. We are monitoring the situation and will post a final update later today..

Mar 6, 14:03:22 GMT+0
Resolved - This incident has been resolved..

Duration: 22 days, 8 hours and 28 minutes

Affected Components: staging.gcp-us-central1-f, test.aws-us-east-1c, staging.aws-us-east-1c, test.gcp-us-central1-f

Mar 4, 23:15:00 GMT+0
Identified - The root cause has been identified.

Mar 5, 05:45:00 GMT+0
Monitoring - We deployed a fix and are currently monitoring the result..

Mar 27, 07:43:29 GMT+0
Resolved - This incident has been resolved..

Duration: 5 hours and 41 minutes

Affected Components: Infrastructure

Mar 4, 12:01:01 GMT+0
Resolved - This incident has been resolved..

Mar 4, 06:20:00 GMT+0
Identified - We are investigating.

Mar 4, 10:41:18 GMT+0
Identified - We are continuing to work on a fix for this incident. We are currently testing an attempted mitigation.

Mar 4, 10:43:31 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 4, 10:53:07 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Duration: 7 hours and 40 minutes

Affected Components: test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, test.gcp-us-central1-f

Mar 3, 23:15:00 GMT+0
Investigating - There was a period where some staging and system test deployments failed..

Mar 4, 06:55:00 GMT+0
Resolved - The root cause has been identified and mitigated..

Duration: 1 day and 1 hour

Affected Components: test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, test.gcp-us-central1-f

Feb 28, 14:00:00 GMT+0
Investigating - There was a period where some staging and system test deployments failed. The root cause has been identified and fixed..

Mar 1, 15:00:00 GMT+0
Resolved - This incident has been resolved..

Duration: 12 minutes

Affected Components: Console

Feb 20, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 20, 11:11:58 GMT+0
Completed - Maintenance has completed successfully..

Feb 20, 11:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. No downtime is expected..

Duration: 3 hours and 48 minutes

Affected Components: AWS, prod.gcp-europe-west3-b, test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, prod.aws-eu-west-1a, prod.aws-us-west-2a, perf.aws-us-east-1c, GCP, Infrastructure, prod.aws-us-east-1c, prod.gcp-us-central1-f, test.gcp-us-central1-f, dev.aws-us-east-1c, dev.gcp-us-central1-f, Console

Feb 10, 11:28:07 GMT+0
Investigating - We are experiencing a technical issue that is impacting provisioning of new Vespa hosts. Provisioning requests may fail or be delayed while we work to resolve the problem. **Existing deployments remain unaffected.** We are currently investigating this incident..

Feb 10, 12:39:50 GMT+0
Identified - We have identified the root cause and are taking the necessary steps to resolve the issue..

Feb 10, 12:59:35 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Feb 10, 15:15:59 GMT+0
Resolved - This incident has been resolved..

Duration: 43 minutes

Affected Components: Console

Feb 5, 20:45:00 GMT+0
Investigating - We are aware of slowdowns and downtime in the Vespa Console, we are currently investigating this incident..

Feb 5, 21:07:46 GMT+0
Identified - We have identified the issue and are working on a workaround..

Feb 5, 21:28:21 GMT+0
Resolved - This incident has been resolved, we have had a misconfiguration in autoscaling which prevented an immediate automatic resolution. We have implemented a workaround and will implement a permanent fix in the next days..

Duration: 39 minutes

Affected Components: Console

Jan 17, 09:52:58 GMT+0
Investigating - The metrics shown in console are incomplete for applications running in GCP and enclave. We are currently investigating this incident..

Jan 17, 10:16:09 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 17, 10:31:32 GMT+0
Resolved - This incident has been resolved..

Duration: 8 hours and 39 minutes

Affected Components: prod.gcp-europe-west3-b, staging.gcp-us-central1-f, GCP, prod.gcp-us-central1-f, test.gcp-us-central1-f

Jan 4, 12:20:00 GMT+0
Investigating - We are currently investigating this incident..

Jan 4, 19:57:49 GMT+0
Monitoring - We have applied a fix and are currently monitoring the result..

Jan 4, 20:58:36 GMT+0
Resolved - This incident has been resolved..

Duration: 5 days and 30 minutes

Affected Components: AWS, prod.gcp-europe-west3-b, staging.gcp-us-central1-f, GCP, prod.gcp-us-central1-f, dev.gcp-us-central1-f

Dec 12, 11:09:00 GMT+0
Investigating - We identified an issue which is preventing metrics to be published to our systems.. We are investigating the issue..

Dec 17, 11:39:08 GMT+0
Resolved - This incident has been resolved..

Dec 16, 10:24:22 GMT+0
Identified - The first attempts to resolve this failed We are continuing to work on a fix for this incident..

Dec 12, 15:00:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the progress..

Dec 16, 16:47:27 GMT+0
Monitoring - We have applied a fix and are currently monitoring the result..

Duration: 30 minutes

Affected Components: Console

Dec 9, 13:00:00 GMT+0
Identified - We are preparing to perform maintenance on the sign-in functionality on the Vespa Cloud Console. Sign in functionality may be sporadically unavailable for up to 30 minutes. This will affect only the control plane login, data plane authentication will remain functional during this maintenance window..

Dec 9, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 9, 13:30:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 23 hours and 36 minutes

Affected Components: AWS, dev.aws-us-east-1c, test.aws-us-east-1c, staging.aws-us-east-1c, perf.aws-us-east-1c, Infrastructure, prod.aws-us-east-1c

Oct 20, 07:11:00 GMT+0
Identified - Track status on <https://health.aws.amazon.com/health/status> Existing applications are not affected by outage. Only maintenance operations are degraded..

Oct 20, 10:26:36 GMT+0
Monitoring - Infrastructure services restored, we will continue to monitor the situation for any residual issues..

Oct 20, 16:50:39 GMT+0
Monitoring - We are still experiencing the failures with provisioning new hosts on AWS us-east region. Therefore, we recommend to avoid applying changes to the applications on this region until the issue is resolved on AWS side. .

Oct 20, 20:58:27 GMT+0
Monitoring - Infrastructure services are now opertional. We continue to monitor our system performance to detect residual issues..

Oct 21, 06:47:23 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 13 minutes

Affected Components: Console

Aug 22, 15:08:28 GMT+0
Resolved - This incident on Grafana is still open, but we don't observe any effect on our side..

Aug 22, 11:55:00 GMT+0
Monitoring - We have identified an issue in our vespa Console, when clicking on "Metrics" for an application we you might experience "Failed to load dashboard - Internal Server Error" This is caused by an outage in Grafana, for more details please refer to \[the Grafana Status Page\](https://status.grafana.com/incidents/z42xrjqydny8) updates..

Duration: 1 hour and 12 minutes

Affected Components: perf.aws-us-east-1c, test.gcp-us-central1-f, prod.aws-us-east-1c, prod.gcp-europe-west3-b, prod.gcp-us-central1-f, staging.gcp-us-central1-f, test.aws-us-east-1c, staging.aws-us-east-1c, dev.aws-us-east-1c, , prod.aws-eu-west-1a, prod.aws-us-west-2a, dev.gcp-us-central1-f, Zones →

Jul 2, 06:31:58 GMT+0
Investigating - We are currently investigating this incident..

Jul 2, 06:47:51 GMT+0
Identified - We have identified the root cause and we are applying a mitigation to allow nodes to be provisioned..

Jul 2, 07:00:57 GMT+0
Monitoring - Workaround is in place, so it should be possible to provision new nodes. If you encounter issues don't hesitate to contact support..

Jul 2, 07:43:51 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 1 minute

Affected Components: Console

Jun 26, 12:17:20 GMT+0
Investigating - New data will not be populated in the dashboard, while historical data is still visible. We are currently investigating this incident. .

Jun 26, 13:18:09 GMT+0
Resolved - This incident has been resolved..

Duration: 14 hours and 10 minutes

Affected Components: Infrastructure

Jun 16, 08:30:00 GMT+0
Identified - Impact: \* Any application changes will fail to be successfully deployed \* Autoscaling: New nodes added to such container clusters will fail to start We have implemented a fix that we currently are testing \[Embedding Reference\](https://docs.vespa.ai/en/reference/embedding-reference.html).

Jun 16, 14:54:51 GMT+0
Identified - We are continuing to work on a fix for this incident..

Jun 16, 21:29:16 GMT+0
Monitoring - We have implemented a fix and created a new release, which is now being rolled out across our infrastructure. Once the deployment is complete, the issue reported in this incident will be considered resolved..

Jun 16, 22:40:06 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 23 minutes

Affected Components: perf.aws-us-east-1c, test.gcp-us-central1-f, prod.aws-us-east-1c, prod.gcp-europe-west3-b, prod.gcp-us-central1-f, staging.gcp-us-central1-f, test.aws-us-east-1c, staging.aws-us-east-1c, dev.aws-us-east-1c, , prod.aws-eu-west-1a, prod.aws-us-west-2a, dev.gcp-us-central1-f, Zones →

Jun 3, 11:36:20 GMT+0
Investigating - We are have identified a regression on the vespa release 8.528.12..

Jun 3, 11:47:59 GMT+0
Identified - All the upgrades in Vespa Cloud are being stopped and any application already upgraded will be automatically downgraded..

Jun 3, 12:17:43 GMT+0
Monitoring - All production applications have been rolled back the the previous version, there should not be any impact at the current moment. We will be monitoring the situation for a while..

Jun 3, 14:59:40 GMT+0
Resolved - We don't see any further impact. Please don't hesitate to contact our support through the Vespa Cloud panel if you still experience any disruption..

Duration: 12 hours and 47 minutes

Affected Components: dev.aws-us-east-1c, dev.gcp-us-central1-f

May 22, 22:17:00 GMT+0
Resolved - This incident has been resolved..

May 22, 20:30:00 GMT+0
Investigating - We are currently investigating an incident affecting new deployments in Dev..

May 22, 20:58:04 GMT+0
Identified - We identified the root of the problem and it is affecting only a subset of hosts. We are working on a fix..

May 22, 22:10:22 GMT+0
Monitoring - We have resolved the issue that caused some deployments to fail.

Duration: 23 hours and 29 minutes

Affected Components: dev.aws-us-east-1c

Apr 21, 11:57:00 GMT+0
Investigating - Due to increased demand, we are seeing failures in launching instances for new tenants. We are currently investigating this incident..

Apr 22, 08:52:23 GMT+0
Monitoring - We have identified the issue and increased the limits so new tenants should start working. We are now monitoring to confirm that this resolves the issue..

Apr 22, 11:25:49 GMT+0
Resolved - This incident has been resolved..

Duration: 5 minutes

Affected Components: Console

Sep 4, 07:15:01 GMT+0
Identified - Maintenance is now in progress.

Sep 4, 07:20:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 4, 07:15:00 GMT+0
Identified - We are planning a Vespa Cloud console maintenance. The console will be unavailable for a few minutes.