June 10, 2020 - Traffic Spike Causes Increased Error Rates

Summary

On June 10, 2020, starting at 10:34 AM UTC, Auth0 received a sudden spike in traffic to the authorization_code flow in our EU Production environment that lasted until 10:40 UTC. This traffic spike generated sufficient load to impair functionality, and resulted in an increased number of HTTP 502 errors being returned to users between 10:35 AM UTC and 10:45 AM UTC.

What Happened

Starting at approximately 10:34 AM UTC Auth0 began receiving a sudden spike in traffic for authentication endpoints in our EU Production environment. This caused load on Auth0 servers to rapidly increase, to a point that nodes became unresponsive before new nodes could be added to handle the increase in traffic. This resulted in customers seeing an increased number of HTTP 502 errors being returned (up to 81% of responses during the peak of the incident).

During this period of increased load our global Rate Limit protections were triggered, but this was not sufficient for handling sudden bursts of traffic.

The nodes were ultimately replaced and increased by automated processes, and health returned to normal.

Mitigation Actions

• Improve global Rate Limit configuration and scaling settings to better handle bursts of traffic. Initially this will include increasing the minimum number of servers available to handle traffic in order to more effectively distribute load.

Annex 1: Events Timeline