We received a large traffic spike (DDoS0 around 15:00 UTC, which targeted a specific type of resource exhaustion. When we deployed additional capacity earlier this week in Ashburn, the new systems had a very low ceiling configured for this particular resource due to an accidental regression in our deployment automation templates last week, and they were unable to recover without manual intervention.
Normally, Quad9's systems would've been able to handle this traffic spike without service degradation and there wouldn't have been resource exhaustion.
After performing a global audit, only these new servers in Ashburn were affected.
We fixed the configuration/regression issue in our deployment system and on these new Ashburn servers.
We apologize for this issue. Ashburn is one of our busiest global PoPs, and this affected a lot of users.
Issue Start: 15:00 UTC Issue Stop: 20:20 UTC
Posted Mar 20, 2025 - 20:33 UTC
Investigating
We're seeing major packet loss to our systems in Ashburn.
We are investigating.
Issue Start: 15:00 UTC
Posted Mar 20, 2025 - 19:28 UTC
This incident affected: Recursive DNS Services (Current Status).