/
[incident] 2025-09-30 (9:35 AM - 4:00 PM PST) and 2025-10-01 (11:00AM ) Cloud Performance Degradation for Organization Users (Resolved)

[incident] 2025-09-30 (9:35 AM - 4:00 PM PST) and 2025-10-01 (11:00AM ) Cloud Performance Degradation for Organization Users (Resolved)

Incident Summary

Date: Sep 30, 2025
Time: 9:35 AM PST
Service Impacted: Cloud Access Issue for Organization Users
Severity: High
Total Duration: TBA

Customer Impact:

  • Desktop / Mobile Clients being unexpectedly logged out

  • Some customers cannot connect to their Sites

  • Certain API requests are experiencing high latency and increased failure rates

Our Site Reliability Engineering team is working in the investigation and will be providing the updates here.

Incident Timeline (PST)

Sep 30, 2025

  • 9:35 AM - Monitoring detected slow Web Response Times with authentication. Investigation began.

  • 10:32 AM - Customer impacts reported

  • 10:35 AM - Escalated internally, investigation continues.

  • 4:00 PM - Traffic load reduced, all services restored.

  • 4:45 PM - We decided to schedule an emergency update to avoid further spikes.

    Oct 1, 2025

  • 8:50 AM - Web Response Time climbing again, potential customer impact. We are monitoring. Investigation ongoing.

  • 11:00 AM - Web response Time jumped over 2 seconds, Continuing to investigate. Customer errors seen.

  • 12:44 PM - Web Response time decreased to 500 ms. Intermittent slow response times, continuing to investigate.

  • 4:15 PM - We need to perform an emergency maintenance to upgrade the AWS instance running Cloud DB.

  • 5:35 PM - Emergency maintenance began.

  • 6:34 PM - Emergency maintenance failed, rolled back. Investigations continue.

    Oct 2, 2025

  • 8:36 AM - Web Response Times climbing, potential organization user impact. Monitoring. Investigation continuing.

  • 11:00 AM - Web Response Time is still high, but has not spiked, monitoring.

  • 11:09 AM - Web Response Time spiked, customer impact happening.

  • 1:30 PM - Web Response Time stabilized. Services restored.

Oct 6, 2025

  • 6:42 PM - Emergency Patch deployed, monitoring for further issues.

Oct 7, 2025

  • 9:15 AM - Web Response Time is much better than previous week, continuing to monitor.

  • 1:07 PM - Web Response Time continues to be better than previous week, continuing to monitor.

  • 3:57 PM - Web Response Time good, continuing to monitor.
    Oct 8, 2025

  • Resolved.