/
[post-mortem] 2026-06-12 6:30 AM PST (1:30 PM UTC) - Cloud Intermittent Issues

[post-mortem] 2026-06-12 6:30 AM PST (1:30 PM UTC) - Cloud Intermittent Issues

Incident Overview

Date: June 12, 2026
Initial Detection Time: 6:30 AM PDT
Primary Area Impacted: Cloud services and upstream service connectivity
Confirmed Root Cause Area: Networking / service traffic routing layer
Severity: Medium
Status: Resolved
Resolution Confirmed: 8:54 AM PDT

Summary of Impact

Beginning on June 12, 2026, users experienced failures with several cloud account management functions. The issue was active for several hours before being resolved through a configuration rollback. All affected functions were confirmed working following the rollback.

Customer Impact

  • New cloud user creation may have failed, especially for email addresses that have never had a Cloud account before.

  • Activation emails may not have been received for newly created users.

  • Password reset workflows may not have completed or may have failed to load after clicking the reset button for customization-specific deployments.

  • Some customers may have seen intermittent cloud connectivity issues or cloud systems temporarily showing as unreachable.

When the incident was declared, our immediate priority was restoring service. We identified all recent changes made in the days leading up to the incident and began rolling them back in order, testing after each step.The routing migration change applied on June 11 was the one that restored normal operation, confirming it as the source of the failure. Once service was restored, we continued the investigation to understand exactly why that change caused the problem.

The deeper investigation produced a fully documented root cause with supporting evidence from service mesh access logs, internal service logs, and direct reproduction of the failure. This confirmed that the incompatibility between Cloud DB's communication pattern and the service mesh's routing rules was the definitive cause.

The investigation also confirmed that this is not an isolated case. Our services were designed and tested against the previous ALB-based routing and there are additional areas where similar incompatibilities with the new traffic routing may exist. We are conducting a full audit of internal service communication patterns to identify all such cases before proceeding further. This includes a review of Cloud DB's communication design and any other service that may behave differently under the new routing path. The scope of these incompatibilities will be fully identified, designed against, and validated before any further routing migration phases are re-enabled

Timeline

June 12, 2026

  • 6:30 AM - First complaints received

  • 7:17 AM - Investigation started

  • 7:20 AM - Investigation narrowed down to a potential root cause

  • 7:24 AM - Multiple Clients issues received

  • 7:58 AM - Issue escalated to the Engineering team for deeper review and support

  • 8:47 AM - The issue has been narrowed. An emergency Update is being scheduled to improve the situation and reduce the customer impact.

  • 8:54 AM - The emergency hotfix has been applied, internal testing confirmed the fix.  Support is working with customer to get confirmation on reported issues are resolved.

Root Cause

We are in the process of migrating its internal traffic routing from AWS Application Load Balancers to a modern service mesh on Kubernetes. On June 11, 2026, we completed a routing migration phase that moved several product customizations onto the new routing path.

This exposed a pre-existing incompatibility in how one of our internal services, Cloud DB, communicates with our authentication service, Oauth2. Cloud DB was making internal service calls in a way that works with the previous ALB-based routing but is incompatible with how the service mesh handles routing. Specifically, the service mesh uses the TLS connection identifier (SNI) to direct traffic to the right destination, while Cloud DB was sending a conflicting identifier in the HTTP layer. The service mesh dropped these calls before they reached the authentication service, causing password reset and user creation flows to fail.

The incompatibility was not introduced by the migration itself. It is a latent issue in how Cloud DB was communicating internally, one that the stricter routing enforcement of the service mesh made visible for the first time.

Contributing Factors

Hybrid ECS and EKS Architecture

As part of the platform modernization initiative, the long-term goal is to migrate workloads from AWS ECS to AWS EKS. To make this migration less disruptive and allow services to move in phases, the team implemented a hybrid ECS/EKS architecture. In this model, ECS and EKS workloads can run side by side while traffic is gradually shifted to the new EKS-based path. Correct traffic routing is critical during this transition state. The Traffic Switch project was created to support this phased migration by allowing production traffic to be routed through the new setup while preserving the ability to validate, adjust, and roll back as needed.

At the time of the incident, the Traffic Switch rollout was in-progress and multiple phases were completed. 

How We Fixed It

The emergency change was performed to roll back the June 11 routing migration phase, restoring the previous ALB-based routing path. All affected functions were confirmed working immediately following the rollback. Post-change monitoring showed improvement in key service metrics. Multiple internal teams and Support later confirmed that the reported customer-facing issues were resolved.

Corrective Actions

Short Term

  • The investigation confirmed that this is not an isolated case. Our services were designed and tested against the previous ALB-based routing infrastructure, and there are additional areas where similar incompatibilities with the new routing path may exist. We are conducting a full audit of internal service communication patterns to identify every such case. Each identified incompatibility will be addressed and validated before the corresponding routing migration phase is re-enabled in production.

  • Additional monitoring and alerting is needed for the class of routing failure that caused this incident so that similar issues can be detected and diagnosed faster in the future.

Long Term

  • The incident and the following investigation has highlighted the need for a broader review of how our services communicate with each other. Our internal service communication design was built for the previous infrastructure and needs to evolve alongside the platform modernization. We are planning a design review of service-to-service communication across our platform to ensure it is built correctly for the new routing infrastructure from the ground up.