Issues when creating Custom Domains with Self-Managed Certificates
Incident Report for Auth0

2020-05-28 - Unable to set up new self-managed custom domains


From 9:00 AM UTC May 22nd through 3:00 AM UTC May 28th Customers were unable to set up new self-managed custom domains because their cname-api-key was not visible on our website.

We would like to apologize to anyone who was unable to verify their custom domains during this time.

What Happened

On Friday, May 22nd our Engineers released an A/B test for new features in our user interface. The change duplicated the components involved in the test, however it also created a duplicate name for the custom domain component that displays the API Key. This meant that the system didn’t know which component to display, because there was more than one, and that is why the pop up window that displays the custom domain API Key was not visible.

This issue persisted for much longer than it should have because we failed to check it during our manual review of the change. The cname-api-key window is only displayed once when setting up custom domains for the first time and it is only used for a specific type of custom domain that is not commonly used. Unfortunately, some customers did encounter this and we are truly sorry for the inconvenience and frustration this caused.

Mitigation Actions

  • We are researching end-to-end testing improvements to discover problems like this sooner.

Annex 1: Events Timeline

Friday May 22nd

09:00 AM UTC - Released the interface A/B test that caused this problem

Thursday May 28th

01:24 AM UTC - Discovered the API KEY was missing

02:30 AM UTC - Identified the cause of the issue

03:24 AM UTC - Fix merged

03:45 AM UTC - Fix deploy completed and confirmed the API KEY was visible again

Posted Jun 23, 2020 - 17:47 UTC

This incident has been resolved.
Posted May 28, 2020 - 04:12 UTC
We have implemented a fix and verified it has fixed the issue. We are currently monitoring the results.
Posted May 28, 2020 - 03:49 UTC
The issue has been identified and a fix is being implemented.
Posted May 28, 2020 - 03:29 UTC
We are currently investigating an issue where creating a Custom Domain with Self-Managed Certificates does not display the CNAME API Key required to finish the configuration. This does not affect customers with Custom Domains that were previously set up, but only customers trying create new ones with the indicated conditions. Our Engineers are currently looking into this, and we will update you as we have more news.
Posted May 28, 2020 - 02:12 UTC
This incident affected: Management Dashboard (