[post-mortem] 2026-04-24 1:01 - 1:06 PM PST (8:01 - 8:06 PM UTC) Cloud DB Outage

Incident Overview

Date: 2026-04-24
Time (PDT): 1:01 - 1:06 PM
Duration: 5 minutes
Severity: High

Services Impacted:

Cloud Portal
Email Notifications
Push Notifications
Cross-Site Layouts
Cloud Access (Desktop/Mobile Clients)

Customer Impact:

Desktop / Mobile Clients being unexpectedly logged out
Some customers cannot connect to their Sites
Certain API requests are experiencing high latency and increased failure rates

For the complete timeline, please see our Incident Page.

Root Cause

The Cloud DB service experienced an unexpected application crash. The service automatically restarted within seconds, but required additional time to fully reload its operational data before all requests could be served successfully.

This crash is part of a recurring pattern that the backend team is addressing.

How We Fixed It

The service recovered automatically. The container orchestration platform detected the crash and launched a replacement within 30 seconds. No manual intervention was required to restore service.

The extended recovery time is due to the service's architecture, which requires loading a large dataset into memory on startup before it can serve all request types.

Corrective Actions

Short Term

Recover and analyze existing crash diagnostic data to identify the specific software defect.

Long Term

Development team is actively rewriting the core of this service to improve resilience and reduce recovery time.