Incident Overview

Date & Time (PST):

Severity: High
Services Impacted:

Customer Impact:

Incidents & emergency updates:

Root Cause

Under peak traffic, multiple threads in Cloud DB accessed the shared cache at the same time, leading to lock waits and thread stalls. This condition led to increased latency.
When the server size was increased, more CPU/threads competed for the same cache. Latency rose as new threads were blocked.
Cross-service connectivity between Cloud DB and Channel Partners experienced connection timeouts under load, which caused retries and increased overall load.

Introduced cache splitting to minimize contention and reduce the probability of threads competing for the same lock.
After caching was split we increased threads to improve throughput.
Deployed a connection pool and in-memory reuse for Channel Partners calls.
Increased targeted timeouts to avoid cascading failures.

Added new monitoring for Cloud DB to Channel Partners connection metrics.
Added new tooling/scripts to monitor additional Host Metrics: CPU, load/run-queue, memory pressure, disk I/O wait.
Added thresholds and alerts to detect resource contention.