On October 11th, between 2:19 PM UTC and 2:45 PM UTC, all Auth0 end users in our Europe (EU) environments experienced errors when authenticating using Lock (https://auth0.com/lock).

We apologize for the impact to you and your customers, and would like to explain what caused the problem, how we reacted to the incident, and what we are doing to prevent incidents like this from happening in the future.

What Happened

Each time Lock is loaded, it performs a request to our CDN (https://cdn.eu.auth0.com/) to download client information. Our CDN uses AWS CloudFront, and during a routine CDN update, we configured CloudFront using the wrong SSL certificate. Because the SSL certificate was wrong, Lock initialization failed, causing the following error for all Lock requests: "We could not reach the server. Please try again."

Once we identified the issue, we rolled back the CloudFront configuration, which stopped the errors and restored service.

Timeline

• 2:19 PM UTC: The wrong SSL certificate was configured for AWS CloudFront, causing the errors to start.
• 2:28 PM UTC: The on-call team received a notification from a support engineer.
• 2:29 PM UTC: We identified the wrong SSL certificate being the cause of the errors.
• 2:29 PM UTC: We started to roll back the CloudFront configuration. Logins using Lock started to stabilize.
• 2:45 PM UTC: The configuration update process was completed. All errors ceased and service was fully restored.

What We're Doing About It

We are going to work on the following things in the short term:

• [in progress] Implement monitoring for total CloudFront HTTPS requests received, so we can observe a sudden drop in requests.
• [in progress] Implement monitoring for CloudFront HTTPS errors.
• [pending] Configure our third-party monitoring service to check the Lock URL.

Summary

We realize that Auth0 is a critical part of your architecture, and is a core technology you depend upon daily. We apologize for the impact these errors had on your business. We will continue to work to provide you with the best authentication service possible.

Thank you for your continued support of Auth0.