Quad9 experienced a significant outage in some points of presence today while replacing/updating core software components in our resolver stack.
We repeatedly performed this update on smaller portions of our network in preceding days without experiencing any problems, as part of a normal, incrementing change cycle. However, when we moved to this larger group of systems, there were issues with automatic health checks, which caused withdrawals of service in some locations. The outage duration varied, but some sites were sporadically or entirely unavailable during the window of 12:50 UTC-13:45 UTC. Most Quad9 locations were not in the set, but notable cities that were in the change group: Bogota, Frankfurt, Turin, Miami, Athens. This issue created delays or missed query responses for roughly 12% of our user community.
We apologize for the outage to those users who experienced problems, and we have identified and resolved the root issue so this is not a repeated failure condition.