From 9:00 AM UTC May 22nd through 3:00 AM UTC May 28th Customers were unable to set up new self-managed custom domains because their cname-api-key was not visible on our website.
We would like to apologize to anyone who was unable to verify their custom domains during this time.
On Friday, May 22nd our Engineers released an A/B test for new features in our user interface. The change duplicated the components involved in the test, however it also created a duplicate name for the custom domain component that displays the API Key. This meant that the system didn’t know which component to display, because there was more than one, and that is why the pop up window that displays the custom domain API Key was not visible.
This issue persisted for much longer than it should have because we failed to check it during our manual review of the change. The cname-api-key window is only displayed once when setting up custom domains for the first time and it is only used for a specific type of custom domain that is not commonly used. Unfortunately, some customers did encounter this and we are truly sorry for the inconvenience and frustration this caused.
09:00 AM UTC - Released the interface A/B test that caused this problem
01:24 AM UTC - Discovered the API KEY was missing
02:30 AM UTC - Identified the cause of the issue
03:24 AM UTC - Fix merged
03:45 AM UTC - Fix deploy completed and confirmed the API KEY was visible again