[post-mortem] 2026-07-29 10:55 AM - 12:54 PM - Cloud Portal and Connectivity Issues
Incident Summary
Date/Time : Jul 29, 2026 10:55 AM - 12:54 PM PST
Initial Detection Time: 10:55 AM PST
Confirmed Root Cause Area: A production infrastructure security update rolled out
Severity: High
Status: Resolved
Total Duration: 1 hr 59 min (10:55 AM – 12:54 PM PST)
Resolution Confirmed: Services restored 12:54 PM PST; all functionality confirmed 1:04 PM PST, July 29, 2026
Summary of Impact
On July 29, 2026, a routine security update to our production server infrastructure was applied without the controlled, staged process such a change requires. The update itself was successful across our services, which were upgraded to the new image and continued to run normally. One service, OAuth2, turned out to be incompatible with the new image: a configuration difference caused it to fail. This prevented customers from loading the Cloud Portal and connecting to their Sites, and disconnected some already-connected users from their Desktop and Mobile Clients. We resolved the incident by rolling back that one service to the previous, known-good image; all other services remained on the new image and were unaffected. All affected functions were confirmed working following the rollback.
Customer Impact:
Cloud Portal didn't show the UI
Users couldn’t connect to their Sites
Already connected users were partially affected (disconnected from their Desktop / Mobile Clients).
When the incident was declared, our immediate priority was restoring service. We identified the change that had just gone out, confirmed it as the cause, and rolled the affected Oauth2 service replicas back to their previous, known-good image. Once service was restored, we continued the investigation to fully understand why the change caused the problem.
Timeline (PST)
Work log is posted in the Incident Report.
Root Cause
As part of routine security maintenance, we rebuilt a foundational server image, "golden image", that underpins much of our production infrastructure, in order to close a set of critical security vulnerabilities.
The security update itself was legitimate and necessary. The problem was how it was deployed: this security fix rolled out to production unexpectedly, applied broadly and all at once, without a maintenance window or the advance notice that a change of this scope requires.
The updated image differed from the previous one in a way that had not been validated before rollout: it lowered a system resource limit (the number of simultaneous network connections a service may hold). Most services were unaffected by this difference, but our authentication service maintains a very large number of simultaneous connections and exhausted that limit, at which point it could no longer open new connections. This left it unable to reach its database, which is why the Cloud Portal would not load and users could not connect to their Sites. The update also disrupted some internal operational tooling used to manage and inspect servers, which slowed diagnosis.
Contributing Factors
Automatic adoption of the latest image. Our infrastructure was configured to follow the "latest" version of the server image, so a newly published image was picked up automatically across the fleet.
Broad rollout instead of a canary. The change was applied across all services simultaneously rather than a deliberate, controlled staged promotion.
Insufficient testing in the rush to deliver the OS security patches. The update closed critical, recommended OS security vulnerabilities, and the urgency to deliver them led to the change being rolled out with insufficient testing before it reached production, specifically the two gaps below.
Configurations were not fully checked. Testing in a lower environment confirmed the image started and services ran, but didn't check the rest of the image's configuration (the resource limits and tooling that changed).
No load testing was performed. The authentication service ran under light load in the lower environment, well below production connection volume, so it never approached the lowered limit. Therefore, the problem only surfaced under real production load.
Recovery was prolonged by a slow rollout on one large service group. Replacing the servers in our largest service group took significantly longer than the others, extended by a separate, pre-existing issue in how those servers drain during replacement. This lengthened the overall period of instability before full recovery and required manual intervention to speed it up.
How We Fixed It
We rolled the affected services back to the previous, known-good image. Cloud Portal and connectivity were restored at 12:54 PM PST, and all functionality was confirmed working at 1:04 PM PST. Post-rollback monitoring and internal testing confirmed the affected functions were operating normally.
Corrective Actions
Short Term
Rebuild the patched golden image with the correct configuration, validate and roll the corrected image out to complete the security patch for the remaining service.
Enforce change control for any change to production infrastructure images. An approved change record, a scheduled maintenance window, and advance notice are required before rollout.
Pin infrastructure to specific, validated image versions rather than following the "latest" version, and keep the previous known-good version available for immediate rollback.
Roll out infrastructure changes in batches.
Add validation that verifies system resource limits and operational tooling on a new image before it is eligible for production.
Make the resource requirements of each service explicit and independent of the underlying image.
Long Term
Establish a regular, controlled cadence for refreshing golden images so that security updates are applied incrementally and validated continuously, rather than accumulating into a single large, higher-risk change.
Review how our services depend on the underlying server image for configuration, and remove implicit dependencies so that infrastructure updates cannot silently change application behavior.