On February 25th 2020 at 13:39 UTC Auth0 began receiving customer support tickets with regard to issues related to AD/LDAP connections. Customers with credential caching disabled were seeing attempts to login via AD/LDAP fail. Calls to the /usernamepassword/login endpoint would respond with a 404 error.
The root cause of the issue was a network configuration change made as part of an operating system upgrade. This caused some assumptions which are made in our application code to no longer be valid.
For tenants with credential caching enabled, the connection would fall back on the cache, which would prevent any login issues.
This incident was resolved at 17:15 UTC.
Thank you for your understanding and patience during this incident.
Auth0 has rolled back the operating system upgrade, and created a high priority task to fix the underlying application code, and implement automated testing to prevent a recurrence of this issue. The work is expected to be completed by 4th March, 2020.
Auth0 has also embarked upon a high priority project to implement additional monitoring and alerting for our AD/LDAP capability, in order to detect these issues sooner.
13:39 UTC - First customer report of the issue was received.
13:53 UTC - The appropriate team was paged and triaging was performed.
15:30 UTC - It was identified that the issue was related to the operating system upgrade. The rollback procedure was started.
17:15 UTC - Incident was marked as resolved.