/
[post-mortem] 2026-05-18 9:30 AM PST (4:30 PM UTC) - Resolved - Cloud Intermittent Issues

[post-mortem] 2026-05-18 9:30 AM PST (4:30 PM UTC) - Resolved - Cloud Intermittent Issues

Incident Overview

Date: May 18, 2026 – May 22, 2026
Initial Detection Time: 9:30 AM PDT
Primary Area Impacted: Cloud services and upstream service connectivity
Initially Suspected Area: Cloud DB
Confirmed Root Cause Area: Networking / service traffic routing layer
Severity: Medium
Status: Resolved
Resolution Confirmed: May 22, 2026, 9:00 AM PDT

Summary

Beginning on May 18, 2026, some customers experienced intermittent failures across multiple cloud workflows and service endpoints.

The primary customer-facing symptoms included:

  • New cloud user creation may fail, especially for email addresses that had never previously had a Cloud account.

  • Activation emails may not be received for newly created users.

  • Password reset workflows may not complete or may fail to load after clicking the reset button.

  • Some customers may see intermittent cloud connectivity issues or cloud systems temporarily showing as unreachable.

Initial symptoms appeared in Cloud DB-dependent workflows, so the investigation initially focused there. Further review confirmed that the issue was not caused by Cloud DB itself, but by traffic routing behavior in the service layer.

The confirmed issue was in the networking and traffic routing layer. Traffic was not being distributed correctly across available upstream service replicas. As a result, only a limited subset of service replicas was receiving traffic, while other healthy replicas were not being used. This caused overconsumption, resource pressure, and degradation across multiple endpoints.

Two emergency changes were performed to stabilize the environment and reduce customer impact. The first emergency change improved service metrics and reduced customer impact, but it did not fully resolve the issue. The second emergency change restored the previous stable routing path and fully resolved the customer-facing issue

Root Cause

The incident was caused by a service-layer traffic routing issue.

Traffic was not distributed evenly across the available upstream service replicas. Instead, a limited subset of replicas received most of the traffic while other healthy replicas were underutilized. In some cases, multiple replicas running on the same cluster node did not receive traffic as expected, and only one replica per node effectively served traffic. This uneven distribution overloaded the replicas receiving traffic, increased resource pressure, and caused degradation across multiple customer-facing workflows and service endpoints.

The issue was resolved after Traffic Switch was rolled back to ECS and the EKS layer was removed from the active traffic path.

Contributing Factors

Hybrid ECS and EKS Architecture

As part of the platform modernization initiative, the long-term goal is to migrate workloads from AWS ECS to AWS EKS. To make this migration less disruptive and allow services to move in phases, the team implemented a hybrid ECS/EKS architecture. In this model, ECS and EKS workloads can run side by side while traffic is gradually shifted to the new EKS-based path. Correct traffic routing is critical during this transition state. The Traffic Switch project was created to support this phased migration by allowing production traffic to be routed through the new setup while preserving the ability to validate, adjust, and roll back as needed.

At the time of the incident, the Traffic Switch rollout was already in progress, with 5 of 6 phases completed. The active EKS traffic path did not distribute traffic correctly across the available service replicas. This reduced the effective capacity of the affected services and contributed to resource pressure and degradation across multiple endpoints.

Incident Management and Classification

The incident response process did not broaden the investigation quickly enough beyond the initially suspected component. The issue was initially investigated as a potential Cloud DB issue because several customer-facing symptoms and log errors appeared in Cloud DB-dependent workflows. This delayed broader review of the networking, service mesh, traffic routing, and hybrid ECS/EKS layers. As a result, the initial classification was incomplete, and the investigation did not immediately include all of the technical areas needed to identify the root cause.

How We Fixed It

Two emergency changes were performed.

The first emergency change was performed on May 20, 2026 at 5:30 PM PDT. This change included scaling updates, additional autoscaling adjustments, improved pod distribution, and fixes for additional issues identified with the AWS VPC CNI. The purpose of this change was to increase available capacity, improve workload distribution, and help the environment hold the load while the underlying routing issue was still being addressed.

After the first emergency change, metrics showed improvement compared to the same time on the previous day:

  • Cloud DB improved to 0.49s average latency with 0% unhealthy responses, compared to 0.81s average latency and 3.3% unhealthy responses the previous day. At peak, Cloud DB had reached 1.4s average latency and 7% unhealthy responses.

  • Cloud Portal improved to 0.71s average latency with 3.2% unhealthy responses, compared to 0.99s average latency and 4.3% unhealthy responses the previous day. At peak, Cloud Portal had reached 1.4s average latency and 7% unhealthy responses.

The second emergency change was a complete rollback of Traffic Switch back to ECS. This removed the EKS layer from the active traffic path and restored the previous stable routing path.

After the rollback, post-change monitoring showed improvement in key service metrics. Multiple internal teams and Support later confirmed that the reported customer-facing issues were resolved.

Corrective Actions

Short Term

  • An Istio patch will be implemented to address the observed endpoint discovery and routing behavior.

  • Dashboards will be added to show per-replica traffic distribution, request rate, error rate, CPU, memory, and latency.

  • The incident response process will be updated to ensure the right technical owners are engaged earlier when symptoms span multiple workflows or service layers.

  • Incident classification will be reviewed and updated as new evidence becomes available. If the first suspected component is not confirmed as the root cause, the incident title, ownership, and investigation path will be corrected to avoid narrowing the investigation too early.

Long Term

  • The Traffic Switch will continue, along with the phased migration of workloads from AWS ECS to AWS EKS.

  • Recurring incident response training will be introduced for incident commanders and service owners, with focus on incident classification, escalation criteria, technical owner engagement, decision tracking, and resolution validation.