Service Degredation - Manchester, Warsaw, Boston, Marseille, Sofia, Lisbon
Identified - After replacing/updating core software components in our resolver stack yesterday, a few locations started experiencing service degradation.

We have identified the root cause and are working on a resolution.

* Marseille Issue Start: 13:30 UTC, Jan 30th
* Boston Issue Start: 19:30 UTC, Jan 30th
* Sofia Issue Start: 20:40 UTC, Jan 30th
--> Sofia Issue Stop: 16:10 UTC, Jan 31st - Taken offline until a server upgrade can be organized.
* Lisbon Issue Start: 23:00 UTC, Jan 30th
* Warsaw Issue Start: 23:30 UTC, Jan 30th
* Manchester Issue Start: 5:00 UTC, Jan 31st
Jan 31, 2025 - 10:59 UTC
Chicago (ORD) - Occassional Timeouts and Slow Resolution
Investigating - We've seen evidence of occasional timeouts and slow resolution specific to one Chicago location (ORD). One other chicago location, QORD5, is not affected.

We are investigating.
Dec 21, 2024 - 01:21 UTC
Mumbai (BOM2) - Degraded Performance
Investigating - Our Mumbai (bom2) location is experiencing increased latency and some low amounts of packet loss.

Our provider in Mumbai is investigating.
Feb 20, 2024 - 10:27 UTC
Uptime over the past 90 days. View historical uptime.
Recursive DNS Services ? Operational
90 days ago
99.98 % uptime
Today
Current Status ? Operational
90 days ago
99.96 % uptime
Today
Past incidents (last week) Operational
90 days ago
100.0 % uptime
Today
Scheduled Maintenance Operational
90 days ago
100.0 % uptime
Today
Threat Intelligence API ? Operational
90 days ago
100.0 % uptime
Today
Website ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Past Incidents
Feb 2, 2025
Singapore (qsin1) - Some Packet Loss
Resolved - We experienced some packet loss due to one of our systems going down and not triggering an alert. We are investigating the lack of alerting so this does not occur again.

Issue Start: 13:20 UTC, Feb 1st
Issue Stop: 9:50 UTC, Feb 2nd
Feb 2, 09:58 UTC
Feb 1, 2025

No incidents reported.

Jan 31, 2025

Unresolved incident: Service Degredation - Manchester, Warsaw, Boston, Marseille, Sofia, Lisbon.

Jan 30, 2025
Zurich (ZRH) - Services Down (Blackholed)
Resolved - This has been resolved. Some .CH traffic is routing to Frankfurt, but not much. We will aim to bring our ZRH back online as soon as possible.

Issue Start: 17:10 UTC
Issue Start: 18:00 UTC
Jan 30, 20:46 UTC
Monitoring - Our service in our ZRH PoP have been shut down, and traffic is re-routed to ZRH2, Geneva, or Frankfurt.

We are monitoring and investigating.
Jan 30, 18:04 UTC
Investigating - Some networks in Switzerland which route to our "ZRH" PoP, including Sunrise, currently cannot reach Quad9; the traffic is blackholed.

We have escalated this matter to our network partner as a Priority 1 issue.

This is not related to the Quad9 outages that occurred earlier today.

Issue Start: 17:10 UTC
Jan 30, 17:57 UTC
Global - Availability/Outage Issues in Some Sites (ex: Frankfurt, Miami, Athens, Bogota)
Resolved - Quad9 experienced a significant outage in some points of presence today while replacing/updating core software components in our resolver stack.

We repeatedly performed this update on smaller portions of our network in preceding days without experiencing any problems, as part of a normal, incrementing change cycle. However, when we moved to this larger group of systems, there were issues with automatic health checks, which caused withdrawals of service in some locations. The outage duration varied, but some sites were sporadically or entirely unavailable during the window of 12:50 UTC-13:45 UTC. Most Quad9 locations were not in the set, but notable cities that were in the change group: Bogota, Frankfurt, Turin, Miami, Athens. This issue created delays or missed query responses for roughly 12% of our user community.

We apologize for the outage to those users who experienced problems, and we have identified and resolved the root issue so this is not a repeated failure condition.

For any questions, please contact support@quad9.net

Issue Start: 12:50 UTC, January 30th
Issue Stop: 13:45 UTC, January 30th.
Jan 30, 15:42 UTC
Jan 29, 2025

No incidents reported.

Jan 28, 2025
Palo Alto (PAO) - Arelion (AT&T, etc) traffic routing to Los Angeles (+15 ms)
Resolved - This was resolved by Arelion on January 22nd. We apologize for the late notice.
Jan 28, 22:52 UTC
Investigating - Starting around 6:10 UTC on January 17th, Arelion (AS1299) started routing traffic in Silicon Valley to our Los Angeles PoP.

This affects AT&T, and potentially Verizon and Cox traffic as well.

We are conferring with our network partner in PAO.
Jan 19, 09:46 UTC
Jan 27, 2025

No incidents reported.

Jan 26, 2025

No incidents reported.

Jan 25, 2025

No incidents reported.

Jan 24, 2025

No incidents reported.

Jan 23, 2025

No incidents reported.

Jan 22, 2025

No incidents reported.

Jan 21, 2025

No incidents reported.

Jan 20, 2025
Mexico City (qmex2) - Offline (+20-50ms to US)
Resolved - Service has resumed in Mexico City as of 17:00 UTC.
Jan 20, 20:07 UTC
Identified - This PoP was taken offline due to a physical connection issue with our uplink to our network partner.

We are coordinating troubleshooting efforts with the facility.
Jan 16, 17:12 UTC
Jan 19, 2025
← Incident History Powered by Atlassian Statuspage