Overview

The Rippling app and associated services experienced a major outage from 7:50 AM PST to 8:32 AM PST, with restoration of services occurring immediately after the outage.

Root Cause

There is an open bug with MongoDB that leads to a database deadlock during a switchover under certain conditions. A switchover that met these conditions occurred at 7:50 AM PST due to increased load on the primary database, which led to the database becoming deadlocked during this period.

Resolution Path

‌We were able to force the switchover to complete and bring the service back after about 40 minutes.

Action Items

• Rippling updated its MongoDB configuration so that the open bug can no longer cause deadlocking
• Rippling is updating its database configuration to mitigate the impact of future MongoDB issues under these circumstances

Rippling’s mission is to Free Smart People to Work on Hard Problems, and we can only achieve that by earning the trust of our customers. When we fail to achieve our mission, we strive to be transparent with customers, and to outline our commitment to preventing future occurrences of similar incidents.

We apologize for any impact this incident has had on your company operations.