The root cause of the issue was the expiration of certificates on our Cluster Connectivity Manager (CCM) instances. The certificates were renewed earlier this year however did not get fully installed across all services which caused the breakdown of communication between the services.
To address this, we have enhanced our monitoring and devised a comprehensive remediation plan to detect and address similar incidents ahead of time.
We regret the inconvenience this may have caused and strive to ensure every measure is taken to avoid similar issues in the future.