tag:status.vespa.ai,2005:/history Vespa Cloud Status - Incident history 2026-03-06T07:30:00.000+00:00 Vespa Cloud tag:status.vespa.ai,2005:Incident/cmmepi8z4000d3xjzgbtmlg49 2026-03-06T07:30:00.000+00:00 2026-03-06T07:30:00.000+00:00 New nodes can fail to provision - Type: Incident

Duration: 6 hours and 33 minutes

Affected Components: AWS, prod.aws-eu-west-1a, prod.aws-us-west-2a, perf.aws-us-east-1c, prod.aws-us-east-1c, test.aws-us-east-1c, dev.aws-euw1-az1, prod.aws-apne1-az1, prod.aws-cac1-az1, prod.aws-euw1-az1, staging.aws-us-east-1c, prod.aws-euc1-az1, prod.aws-ap-northeast-1a, prod.aws-aps1-az1, prod.aws-euc1-az3, prod.aws-use1-az4, prod.aws-cac1-az2, prod.aws-use2-az1, prod.aws-use2-az3, prod.aws-usw2-az3, dev.aws-us-east-1c

Mar 6, 07:30:00 GMT+0
Investigating - We have observed that new nodes are failing to provision. **Existing nodes are unaffected.**.

Mar 6, 09:07:35 GMT+0
Identified - We have identified the root cause and started working on a workaround..

Mar 6, 10:06:58 GMT+0
Identified - We are continuing to work on a fix for this incident, we have noticed failures in more zones for new hosts. _Ticket updated with the new Zones._ **Existing nodes are unaffected, only provisioning of new nodes is impacted.**.

Mar 6, 10:35:34 GMT+0
Identified - We have implemented a fix and it is currently rolling out fleet-wide. Provision of **new nodes** is still impacted, but we should resolve within 30 minutes, we will post an update as soon as possible, or by 11:00 UTC in any case. **Existing nodes are not affected.**.

Mar 6, 11:09:03 GMT+0
Identified - Rollout is still in progress, still impacting **new nodes on AWS**. **Existing nodes are not impacted.**.

Mar 6, 11:43:09 GMT+0
Identified - We have found an edge-case which has been slowing down the rollout, we're in the process of mitigating this issue. For now **new hosts** are still impacted. New update at 12:15 or earlier. There is **no impact for existing nodes.**.

Mar 6, 12:47:19 GMT+0
Monitoring - The rollout is compete and new nodes are able to provision. We are monitoring the situation and will post a final update later today..

Mar 6, 14:03:22 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cmmdbpczm02cpx7bgt52w6ujt 2026-03-04T23:15:00.000+00:00 2026-03-04T23:15:00.000+00:00 Issues with staging/system test deployments Type: Incident

Affected Components: test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, test.gcp-us-central1-f

Mar 4, 23:15:00 GMT+0
Identified - The root cause has been identified.

Mar 5, 05:45:00 GMT+0
Monitoring - We deployed a fix and are currently monitoring the result..

]]> tag:status.vespa.ai,2005:Incident/cmmbs9mim00nl10ssrdpw94vi 2026-03-04T06:20:00.000+00:00 2026-03-04T12:01:01.920+00:00 Node provisioning is failing, impacting all test and staging test runs, and node changes in deployments Type: Incident

Duration: 5 hours and 41 minutes

Affected Components: Infrastructure

Mar 4, 12:01:01 GMT+0
Resolved - This incident has been resolved..

Mar 4, 06:20:00 GMT+0
Identified - We are investigating.

Mar 4, 10:41:18 GMT+0
Identified - We are continuing to work on a fix for this incident. We are currently testing an attempted mitigation.

Mar 4, 10:43:31 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 4, 10:53:07 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

]]> tag:status.vespa.ai,2005:Incident/cmmbwbt670n5p50p1wokttahl 2026-03-03T23:15:00.000+00:00 2026-03-03T23:15:00.000+00:00 Issues with staging/system test deployments Type: Incident

Duration: 7 hours and 40 minutes

Affected Components: test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, test.gcp-us-central1-f

Mar 3, 23:15:00 GMT+0
Investigating - There was a period where some staging and system test deployments failed..

Mar 4, 06:55:00 GMT+0
Resolved - The root cause has been identified and mitigated..

]]> tag:status.vespa.ai,2005:Incident/cmm9dyv1w0af46l72zsgjqgfs 2026-02-28T14:00:00.000+00:00 2026-02-28T14:00:00.000+00:00 Issues with staging/system test deployments Type: Incident

Duration: 1 day and 1 hour

Affected Components: test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, test.gcp-us-central1-f

Feb 28, 14:00:00 GMT+0
Investigating - There was a period where some staging and system test deployments failed. The root cause has been identified and fixed..

Mar 1, 15:00:00 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Maintenance/cmlthtoio04qj24hsd6x76jaq 2026-02-20T11:00:00.000+00:00 2026-02-20T11:00:01.000+00:00 Vespa Cloud Console Maintenance Type: Maintenance

Duration: 12 minutes

Affected Components: Console

Feb 20, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 20, 11:11:58 GMT+0
Completed - Maintenance has completed successfully..

Feb 20, 11:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. No downtime is expected..

]]> tag:status.vespa.ai,2005:Incident/cmlgiost217l05p7vd6jowrgy 2026-02-10T11:28:07.334+00:00 2026-02-10T11:28:07.334+00:00 Issues with new host provisioning Type: Incident

Duration: 3 hours and 48 minutes

Affected Components: AWS, prod.gcp-europe-west3-b, test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, prod.aws-eu-west-1a, prod.aws-us-west-2a, perf.aws-us-east-1c, GCP, Infrastructure, prod.aws-us-east-1c, prod.gcp-us-central1-f, test.gcp-us-central1-f, dev.aws-us-east-1c, dev.gcp-us-central1-f, Console

Feb 10, 11:28:07 GMT+0
Investigating - We are experiencing a technical issue that is impacting provisioning of new Vespa hosts. Provisioning requests may fail or be delayed while we work to resolve the problem. **Existing deployments remain unaffected.** We are currently investigating this incident..

Feb 10, 12:39:50 GMT+0
Identified - We have identified the root cause and are taking the necessary steps to resolve the issue..

Feb 10, 12:59:35 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Feb 10, 15:15:59 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cml9y5r7v00033m8w7z1ezl9u 2026-02-05T20:45:00.000+00:00 2026-02-05T20:45:00.000+00:00 Vespa Cloud Console Slow or Unavailable Type: Incident

Duration: 43 minutes

Affected Components: Console

Feb 5, 20:45:00 GMT+0
Investigating - We are aware of slowdowns and downtime in the Vespa Console, we are currently investigating this incident..

Feb 5, 21:07:46 GMT+0
Identified - We have identified the issue and are working on a workaround..

Feb 5, 21:28:21 GMT+0
Resolved - This incident has been resolved, we have had a misconfiguration in autoscaling which prevented an immediate automatic resolution. We have implemented a workaround and will implement a permanent fix in the next days..

]]> tag:status.vespa.ai,2005:Incident/cmki4pzur073e8nn6utsrf1x3 2026-01-17T09:52:58.876+00:00 2026-01-17T09:52:58.876+00:00 Missing metrics from GCP and enclave Type: Incident

Duration: 39 minutes

Affected Components: Console

Jan 17, 09:52:58 GMT+0
Investigating - The metrics shown in console are incomplete for applications running in GCP and enclave. We are currently investigating this incident..

Jan 17, 10:16:09 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 17, 10:31:32 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cmk03c8690123et5dbqlkmarl 2026-01-04T12:20:00.000+00:00 2026-01-04T12:20:00.000+00:00 Missing Metrics for GCP and Enclave Accounts Type: Incident

Duration: 8 hours and 39 minutes

Affected Components: prod.gcp-europe-west3-b, staging.gcp-us-central1-f, GCP, prod.gcp-us-central1-f, test.gcp-us-central1-f

Jan 4, 12:20:00 GMT+0
Investigating - We are currently investigating this incident..

Jan 4, 19:57:49 GMT+0
Monitoring - We have applied a fix and are currently monitoring the result..

Jan 4, 20:58:36 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cmj32ve7r070j12dnedon30sv 2025-12-12T11:09:00.000+00:00 2025-12-12T11:09:00.000+00:00 Missing Metrics for GCP and Enclave (All) Type: Incident

Duration: 5 days and 30 minutes

Affected Components: AWS, prod.gcp-europe-west3-b, staging.gcp-us-central1-f, GCP, prod.gcp-us-central1-f, dev.gcp-us-central1-f

Dec 12, 11:09:00 GMT+0
Investigating - We identified an issue which is preventing metrics to be published to our systems.. We are investigating the issue..

Dec 17, 11:39:08 GMT+0
Resolved - This incident has been resolved..

Dec 16, 10:24:22 GMT+0
Identified - The first attempts to resolve this failed We are continuing to work on a fix for this incident..

Dec 12, 15:00:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the progress..

Dec 16, 16:47:27 GMT+0
Monitoring - We have applied a fix and are currently monitoring the result..

]]> tag:status.vespa.ai,2005:Maintenance/cmiyl2l2d07k9c8n9oaf29fu3 2025-12-09T13:00:00.000+00:00 2025-12-09T13:00:00.000+00:00 Temporary downtime of sign in on Vespa Cloud Console Type: Maintenance

Duration: 30 minutes

Affected Components: Console

Dec 9, 13:00:00 GMT+0
Identified - We are preparing to perform maintenance on the sign-in functionality on the Vespa Cloud Console. Sign in functionality may be sporadically unavailable for up to 30 minutes. This will affect only the control plane login, data plane authentication will remain functional during this maintenance window..

Dec 9, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 9, 13:30:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.vespa.ai,2005:Incident/cmgytgo3p00mlmx24ihc87kdx 2025-10-20T07:11:00.000+00:00 2025-10-20T07:11:00.000+00:00 AWS experiences issues in us-east-1 Type: Incident

Duration: 23 hours and 36 minutes

Affected Components: AWS, dev.aws-us-east-1c, test.aws-us-east-1c, staging.aws-us-east-1c, perf.aws-us-east-1c, Infrastructure, prod.aws-us-east-1c

Oct 20, 07:11:00 GMT+0
Identified - Track status on <https://health.aws.amazon.com/health/status> Existing applications are not affected by outage. Only maintenance operations are degraded..

Oct 20, 10:26:36 GMT+0
Monitoring - Infrastructure services restored, we will continue to monitor the situation for any residual issues..

Oct 20, 16:50:39 GMT+0
Monitoring - We are still experiencing the failures with provisioning new hosts on AWS us-east region. Therefore, we recommend to avoid applying changes to the applications on this region until the issue is resolved on AWS side. .

Oct 20, 20:58:27 GMT+0
Monitoring - Infrastructure services are now opertional. We continue to monitor our system performance to detect residual issues..

Oct 21, 06:47:23 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cmems9mnk0001107un22ahtkn 2025-08-22T11:55:00.000+00:00 2025-08-22T15:08:28.240+00:00 Issues with metric dashboards in Vespa Console Type: Incident

Duration: 3 hours and 13 minutes

Affected Components: Console

Aug 22, 15:08:28 GMT+0
Resolved - This incident on Grafana is still open, but we don't observe any effect on our side..

Aug 22, 11:55:00 GMT+0
Monitoring - We have identified an issue in our vespa Console, when clicking on "Metrics" for an application we you might experience "Failed to load dashboard - Internal Server Error" This is caused by an outage in Grafana, for more details please refer to \[the Grafana Status Page\](https://status.grafana.com/incidents/z42xrjqydny8) updates..

]]> tag:status.vespa.ai,2005:Incident/cmclkwzow01fxt1chtevzvasm 2025-07-02T06:31:58.999+00:00 2025-07-02T06:31:58.999+00:00 Provisioning new nodes might fail Type: Incident

Duration: 1 hour and 12 minutes

Affected Components: perf.aws-us-east-1c, test.gcp-us-central1-f, prod.aws-us-east-1c, prod.gcp-europe-west3-b, prod.gcp-us-central1-f, staging.gcp-us-central1-f, test.aws-us-east-1c, staging.aws-us-east-1c, dev.aws-us-east-1c, , prod.aws-eu-west-1a, prod.aws-us-west-2a, dev.gcp-us-central1-f, Zones →

Jul 2, 06:31:58 GMT+0
Investigating - We are currently investigating this incident..

Jul 2, 06:47:51 GMT+0
Identified - We have identified the root cause and we are applying a mitigation to allow nodes to be provisioned..

Jul 2, 07:00:57 GMT+0
Monitoring - Workaround is in place, so it should be possible to provision new nodes. If you encounter issues don't hesitate to contact support..

Jul 2, 07:43:51 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cmcdcm03p002f7gde7sg7nl6g 2025-06-26T12:17:20.032+00:00 2025-06-26T12:17:20.032+00:00 Metrics dashboard not working for GCP and enclave applications Type: Incident

Duration: 1 hour and 1 minute

Affected Components: Console

Jun 26, 12:17:20 GMT+0
Investigating - New data will not be populated in the dashboard, while historical data is still visible. We are currently investigating this incident. .

Jun 26, 13:18:09 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cmbz7ph1i000gq87zjzk79lst 2025-06-16T08:30:00.000+00:00 2025-06-16T08:30:00.000+00:00 The vespa release rolled out today, vespa-8.534.59, introduced an issue affecting applications with containers running embedders Type: Incident

Duration: 14 hours and 10 minutes

Affected Components: Infrastructure

Jun 16, 08:30:00 GMT+0
Identified - Impact: \* Any application changes will fail to be successfully deployed \* Autoscaling: New nodes added to such container clusters will fail to start We have implemented a fix that we currently are testing \[Embedding Reference\](https://docs.vespa.ai/en/reference/embedding-reference.html).

Jun 16, 14:54:51 GMT+0
Identified - We are continuing to work on a fix for this incident..

Jun 16, 21:29:16 GMT+0
Monitoring - We have implemented a fix and created a new release, which is now being rolled out across our infrastructure. Once the deployment is complete, the issue reported in this incident will be considered resolved..

Jun 16, 22:40:06 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cmbgg0p79000equffv437p932 2025-06-03T11:36:20.545+00:00 2025-06-03T11:36:20.545+00:00 Performance regression on release 8.528.12 Type: Incident

Duration: 3 hours and 23 minutes

Affected Components: perf.aws-us-east-1c, test.gcp-us-central1-f, prod.aws-us-east-1c, prod.gcp-europe-west3-b, prod.gcp-us-central1-f, staging.gcp-us-central1-f, test.aws-us-east-1c, staging.aws-us-east-1c, dev.aws-us-east-1c, , prod.aws-eu-west-1a, prod.aws-us-west-2a, dev.gcp-us-central1-f, Zones →

Jun 3, 11:36:20 GMT+0
Investigating - We are have identified a regression on the vespa release 8.528.12..

Jun 3, 11:47:59 GMT+0
Identified - All the upgrades in Vespa Cloud are being stopped and any application already upgraded will be automatically downgraded..

Jun 3, 12:17:43 GMT+0
Monitoring - All production applications have been rolled back the the previous version, there should not be any impact at the current moment. We will be monitoring the situation for a while..

Jun 3, 14:59:40 GMT+0
Resolved - We don't see any further impact. Please don't hesitate to contact our support through the Vespa Cloud panel if you still experience any disruption..

]]> tag:status.vespa.ai,2005:Incident/cmazula79001311nk2jb9tsn7 2025-05-22T20:30:00.000+00:00 2025-05-22T22:17:00.000+00:00 Issues new deployments in Dev zones Type: Incident

Duration: 12 hours and 47 minutes

Affected Components: dev.aws-us-east-1c, dev.gcp-us-central1-f

May 22, 22:17:00 GMT+0
Resolved - This incident has been resolved..

May 22, 20:30:00 GMT+0
Investigating - We are currently investigating an incident affecting new deployments in Dev..

May 22, 20:58:04 GMT+0
Identified - We identified the root of the problem and it is affecting only a subset of hosts. We are working on a fix..

May 22, 22:10:22 GMT+0
Monitoring - We have resolved the issue that caused some deployments to fail.

]]> tag:status.vespa.ai,2005:Incident/cm9s8b72e004lh2rii8i6frg5 2025-04-21T11:57:00.000+00:00 2025-04-21T11:57:00.000+00:00 Provisioning failures in dev.aws-us-east-1c Type: Incident

Duration: 23 hours and 29 minutes

Affected Components: dev.aws-us-east-1c

Apr 21, 11:57:00 GMT+0
Investigating - Due to increased demand, we are seeing failures in launching instances for new tenants. We are currently investigating this incident..

Apr 22, 08:52:23 GMT+0
Monitoring - We have identified the issue and increased the limits so new tenants should start working. We are now monitoring to confirm that this resolves the issue..

Apr 22, 11:25:49 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/cm986shm0008qsp4flqbc6ms6 2025-04-08T07:36:26.242+00:00 2025-04-08T08:16:44.429+00:00 Missing metrics from on GCP and Enclave Type: Incident

Duration: 4 hours and 20 minutes

Affected Components: staging.gcp-us-central1-f, dev.gcp-us-central1-f, test.gcp-us-central1-f, prod.gcp-europe-west3-b, prod.gcp-us-central1-f, AWS, GCP

Apr 8, 08:16:44 GMT+0
Identified - We have identified one problem with our telemetry gateway and are working on a fix..

Apr 8, 07:36:26 GMT+0
Investigating - We are currently investigating this incident..

Apr 8, 09:19:15 GMT+0
Monitoring - We have applied a workaround and we see the data metrics flowing back in our systems. We are now monitoring the situation..

Apr 8, 11:56:01 GMT+0
Resolved - We can confirm that all the systems have been successfully working since our workaround is applied and the final fix has been applied..

]]> tag:status.vespa.ai,2005:Incident/cm7act6cx002om2eggsyn4fwt 2025-02-18T08:43:00.000+00:00 2025-02-18T08:43:00.000+00:00 Metrics view in application console unavailable due to Grafana incident with AWS Type: Incident

Duration: 3 hours and 39 minutes

Affected Components: Console

Feb 18, 08:43:00 GMT+0
Identified - We are monitoring the progress via <https://status.grafana.com/>.

Feb 18, 12:21:43 GMT+0
Resolved - The part of the Grafana incident that had an impact on the rendring of the metrics in the Vespa Cloud console has been resolved.

]]> tag:status.vespa.ai,2005:Incident/cm1yxjree003n11ysqtggmk5c 2024-10-07T08:00:00.000+00:00 2024-10-07T08:00:00.000+00:00 New deployments, node increase, and replacements might fail in prod.gcp-europe-west3-b Type: Incident

Duration: 5 hours and 48 minutes

Affected Components: prod.gcp-europe-west3-b

Oct 7, 08:00:00 GMT+0
Investigating - We have identified that some deployments are fail in the `prod.gcp-europe-west3-b` zone, we are investigating the issue..

Oct 7, 09:00:00 GMT+0
Identified - We have identified the source of the issue being one of the Quota limits in GCP. The team has been working with Google Cloud Platform (GCP) to increase quotas..

Oct 7, 12:40:44 GMT+0
Identified - We are still waiting from Google Cloud Platform to update our quota for the affected Zone. **Note:** This incident has **no impact on application performances**. Only _new applications and deployments_ with large local storage _might_ be affected..

Oct 7, 13:47:34 GMT+0
Resolved - We have received confirmation from the Google Compute Platform (GCP) that quotas on our account have been updated, this issue has now been be resolved..

]]> tag:status.vespa.ai,2005:Maintenance/cm0niqq02004vl3c8tq3x35xq 2024-09-04T07:15:00.000+00:00 2024-09-04T07:15:01.000+00:00 Vespa Cloud console maintenance Type: Maintenance

Duration: 5 minutes

Affected Components: Console

Sep 4, 07:15:01 GMT+0
Identified - Maintenance is now in progress.

Sep 4, 07:20:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 4, 07:15:00 GMT+0
Identified - We are planning a Vespa Cloud console maintenance. The console will be unavailable for a few minutes.

]]> tag:status.vespa.ai,2005:Incident/cm0ldcmuc00039c9x7pirorg5 2024-09-02T19:03:44.028+00:00 2024-09-02T20:52:21.611+00:00 New deployments, and node increase/replacement might fail in dev.gcp-us-central1-f Type: Incident

Duration: 1 hour and 49 minutes

Affected Components: dev.gcp-us-central1-f

Sep 2, 20:52:21 GMT+0
Resolved - This incident has been resolved..

Sep 2, 19:03:44 GMT+0
Investigating - New hosts are not being provisioned in dev.gcp-us-central1-f. This prevents new dev deployments, and node replacements. We are currently investigating this incident..

]]> tag:status.vespa.ai,2005:Incident/cm03wbqd90001vsdxvubkss0d 2024-08-21T13:35:03.062+00:00 2024-08-21T15:07:16.455+00:00 Node failures in dev environment Type: Incident

Duration: 1 hour and 32 minutes

Affected Components: dev.aws-us-east-1c

Aug 21, 15:07:16 GMT+0
Resolved - This incident has been resolved..

Aug 21, 13:35:03 GMT+0
Investigating - Several nodes have been failed out in the dev environment. We are currently investigating this incident..

Aug 21, 13:51:29 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 21, 14:49:23 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. Due to the fact that there is no redundant storage in the dev environment, some documents may have been lost. We have restored as much as possible..

]]> tag:status.vespa.ai,2005:Incident/clyhowd9843891htn0ec73hkyd 2024-07-11T19:56:30.829+00:00 2024-07-15T13:43:18.964+00:00 New deployments, and node increase/replacement might fail for enclave Type: Incident

Duration: 14 hours and 21 minutes

Affected Components: AWS

Jul 15, 13:43:18 GMT+0
Resolved - This incident has been resolved..

Jul 11, 19:56:30 GMT+0
Monitoring - The immediate issue has been resolved, we are monitoring the system.

Jul 11, 22:41:13 GMT+0
Resolved - This incident has been resolved..

Jul 12, 08:57:16 GMT+0
Investigating - We are currently investigating this incident..

Jul 14, 18:14:17 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 16, 17:40:44 GMT+0
Investigating - We are currently investigating this incident..

Jul 19, 08:15:28 GMT+0
Investigating - We are still investigating. Existing deployments are currently stable.

Jul 24, 09:28:20 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 12, 10:17:48 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.vespa.ai,2005:Incident/clye77mk622191hrn9p8bcuuaz 2024-07-09T08:20:00.000+00:00 2024-07-09T17:15:27.293+00:00 Application deployment may fail due to certificate provisioning Type: Incident

Duration: 3 days, 5 hours and 21 minutes

Affected Components: Infrastructure

Jul 9, 17:15:27 GMT+0
Resolved - We have now resolved this issue, thanks for hanging in there.

Jul 9, 09:28:15 GMT+0
Identified - We have identified the problem and will deploy a patch as soon as possible.

Jul 9, 08:20:00 GMT+0
Investigating - We are currently investigating this incident. In the deployment view this will be shown as "Unexpected exception running run 474 of <zone> for <tenant>.<application>: Timed out after waiting PT5M to acquire lock '/controller/v1/locks/certificatePool'" .

Jul 9, 13:05:29 GMT+0
Identified - We are continuing to work on a fix for this incident. A side effect has appeared now. Deployments are failing with 'UNEXPECTED ERROR', due to 'Endpoints failed to show up within 15 minutes!' .

Jul 9, 14:01:20 GMT+0
Monitoring - Some certificate provisioning requests may fail at this time. This will appear as various errors deploying applications in Vespa Cloud. We have deployed a fix and we are working to stabilise the system and work through its backlog of certificate requests..

Jul 11, 07:44:57 GMT+0
Identified - We are continuing to work on a fix for this incident..

Jul 11, 07:50:31 GMT+0
Identified - We are continuing to work on a fix for this incident..

Jul 11, 13:28:16 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. Please reach out to support if you are experiencing issues with deployments .

Jul 12, 13:41:21 GMT+0
Resolved - We have not seen any certificate issues the last day, resolving..

]]>