A defect has been introduced in a production deployment that unfortunately has not been caught by the automated testing pipeline since the issue affected only the production environment. This error has broken the authentication mechanism of the application, blocking user access to all backend services.
Although the services continued running normally, the authentication mechanism prevented all users from accessing the application via the UI. The API has not been affected and continued to operate normally. No data has been affected.
The team has reverted the deployment to a previous versions. Then, the defect has been fixed and a new deployment took place.
Improvements have been identified for the automation pipeline. New mechanisms to detect production issues right after new deployments will be put in place, so that similar situations can be reverted much quicker.