Incident Overview

Date: Dec 8, 2025
Time: 04:20 AM PST - 04:10 PM PST
Service Impacted: Push Notifications
Severity: Low
Total Duration: 11 hours and 50 minutes

Customer Impact:

Users did not receive any push notifications from Sites.

Root Cause

Push notifications failed for approximately 12 hours due to the database table backing the notification system reaching the maximum value for its AUTO_INCREMENT primary key. Once the integer limit was reached, insert operations into the table began failing, which caused downstream push notification processing to halt.

How We Fixed It

We identified the failing inserts and confirmed that the AUTO_INCREMENT value had reached the MySQL integer maximum. To immediately restore service, we truncated the affected table, which reset the primary key counter and allowed new rows to be inserted without errors. Push notification processing resumed normally after the table was cleared.

Corrective Actions

Short Term:

No immediate short-term actions required. The issue takes several years to reoccur given current traffic volumes and is not expected to recur before the long-term solution is implemented.

Long Term:

Remove the system’s dependency on this database table entirely.