Update - We are continuing to monitor the incident and have started the RCA. We will keep this status in monitoring and continue to provide updates.
Nov 28, 23:13 UTC

Update - At 14:44 UTC our incident response team started investigating an alarm showing an increase in response times for the Authentication API affecting all customers in the US (PROD) environment. The team started investigating the issue, and at 16:14 UTC we discovered issues in our MongoDB cluster that soon began affecting all customers in the US region. The issue started getting worse, not just impacting response times but also generating errors for customers in different APIs. From 16:30 UTC until 18:35 UTC (the worst period of the incident), 18.88% of all requests failed in the US environment.

With all hands on deck, we found issues related to a couple of Database queries that significantly impacted the cluster. We immediately began applying changes and redirected traffic to a different application cluster, which began to normalize the issue.

We are still working diligently on improving the situation and will provide an in-depth Root Cause Analysis as soon as possible.

We are deeply sorry for the inconvenience this issue has caused you, your users and your customers.
Nov 28, 20:52 UTC

Update - We're continuing to work on a fix. Response times are slowly normalizing and errors are going down; we'll post an update as soon as we're back to normal.
Nov 28, 16:56 UTC

Update - We are continuing to investigate this issue.
Nov 28, 18:56 UTC