Spike in traffic requests caused partial temporary outage
Incident Report for Auth0
Postmortem

June 10, 2020 - Traffic Spike Causes Increased Error Rates

Summary

On June 10, 2020, starting at 10:34 AM UTC, Auth0 received a sudden spike in traffic to the authorization_code flow in our EU Production environment that lasted until 10:40 UTC. This traffic spike generated sufficient load to impair functionality, and resulted in an increased number of HTTP 502 errors being returned to users between 10:35 AM UTC and 10:45 AM UTC.

What Happened

Starting at approximately 10:34 AM UTC Auth0 began receiving a sudden spike in traffic for authentication endpoints in our EU Production environment. This caused load on Auth0 servers to rapidly increase, to a point that nodes became unresponsive before new nodes could be added to handle the increase in traffic. This resulted in customers seeing an increased number of HTTP 502 errors being returned (up to 81% of responses during the peak of the incident).

During this period of increased load our global Rate Limit protections were triggered, but this was not sufficient for handling sudden bursts of traffic.

The nodes were ultimately replaced and increased by automated processes, and health returned to normal.

Mitigation Actions

  • Improve global Rate Limit configuration and scaling settings to better handle bursts of traffic. Initially this will include increasing the minimum number of servers available to handle traffic in order to more effectively distribute load.

Annex 1: Events Timeline

Posted Jul 06, 2020 - 15:53 UTC

Resolved
This incident has been resolved.
Posted Jun 10, 2020 - 11:18 UTC
Update
We are continuing to monitor for any further issues.
Posted Jun 10, 2020 - 11:05 UTC
Update
We are continuing to monitor for any further issues.
Posted Jun 10, 2020 - 11:04 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 10, 2020 - 10:30 UTC
This incident affected: Auth0 Europe (PROD) (Authentication API, Management API, Custom DB Connections & Rules, Email Delivery, CDN (cdn.eu.auth0.com), Custom Domains, Multi Factor Authentication, Search API v2, Search API v3) and Auth0 Europe (PREVIEW) (Authentication API, Management API, Custom DB Connections & Rules, Email Delivery, Multi Factor Authentication, Search API v2, Search API v3).