Service in our Sydney (SYD) location was severely degraded and may have been completely unavailable from 1:16 UTC to 3:45 UTC.
Some ISPs in Australia route to our other Sydney PoP (qsyd2), so only some ISPs were affected.
We have identified the root cause as a bug within our monitoring and automated remediation processes, which is the system that determines if servers are healthy and should announce (be in production). We are working on improving this processes immediately to prevent similar situations in the future.
If you are still experiencing issues in Sydney, please reach out to support@quad9.net
Issue Start: 1:20 UTC, September 14th Issue Stop: 6:55 UTC, September 14th
Posted Sep 14, 2023 - 07:08 UTC
This incident affected: Recursive DNS Services (Current Status).