All interactions between Rippling and third-party apps were disrupted for all customers on October 14 between 8:36 AM PDT and 9:06 AM PDT. During this 30 min window, several customers experienced errors with SSO authentication and several scheduled jobs syncing data between Rippling and third-party apps ended prematurely.
This issue occurred due to a configuration change that accidentally diverted traffic to a new control flow. The most noticeable impact of this change was the SSO authentication error, which was quickly resolved after the configuration change was reverted at 9:06 AM PDT. The incident led to delays in data synchronization between Rippling and third-party systems; however, all critical transactions were completed by 10:12 PM PDT on October 14, and non-critical updates were finalized by 11:45 AM PDT on October 15. Our customer support team is on standby should you have any questions or see any issues not yet resolved. No unauthorized data access or data loss occurred during or as a result of this incident.
To prevent this in the future, we are adding monitors for detecting unusual patterns of traffic based on the status code, especially the most used api’s/critical flows to detect these issues. This effort also includes improving our release process to have additional approvals for configuration changes and written roll-out plan reviewed and approved when rolling out new features.
We sincerely apologize for any inconvenience this incident may have caused to our customers. At Rippling, we take responsibility for the stability and reliability of our systems very seriously. Our team is fully committed to conducting a thorough and comprehensive analysis of this incident to understand its root causes and to implement improvements to our systems and processes. We are dedicated to learning from this experience and taking all necessary steps to prevent this type of incident from happening again in the future.