Identified - We've implemented "Phase 1" of performance improvements, increased capacity, and mitigation efforts in this Ashburn PoP.
Phase 2 is planned for later this week or early next week.
We are going to monitor during the next traffic spike and provide an update.
We apologize for this ongoing issue and are trying to make the necessary adjustments to mitigate any performance and reliability issues as soon as possible.
Mar 18, 2025 - 23:01 UTC
Investigating - We're experiencing large-scale attacks / traffic spikes once or twice daily that are resulting in service degradation (timeouts) during these times.
We apologize for the inconvenience and are working on a plan to eliminate the negative effects of these traffic spikes.
We will update this page as soon as we have made the appropriate changes in Ashburn.
Mar 05, 2025 - 19:32 UTC
Update - We have eliminated almost all of the packet loss, but there is still 2-3% packet loss during local, peak traffic. We are seeing if it's possible to eliminate the packet loss in the short term.
Mar 17, 2025 - 22:23 UTC
Monitoring - We are testing a fix and appear to be back at full performance/capacity as of 16:45 UTC.
We are monitoring.
Mar 13, 2025 - 17:11 UTC
Investigating - We are experiencing about 20% packet loss in our Bogota PoP.
This appears to be software-related, and we are investigating, but we are unable to bring this system back into performance for the next 8-12 hours at least.
Investigating - We've seen evidence of occasional timeouts and slow resolution specific to one Chicago location (ORD). One other chicago location, QORD5, is not affected.
Resolved -
While performing a routine maintenance to replace a server in Amsterdam, which was not service impacting itself, a Layer-2 configuration issue with the switch caused our BGP sessions for all of Amsterdam to shut off. This caused all Amsterdam traffic to spill over to Rotterdam, which got overloaded and resulted in severe service degradation (timeouts or retries).
We are working with our network partner in Rotterdam to schedule a major capacity upgrade so it can serve as a fully redundant PoP should there be any outages in Amsterdam.
We apologize for the inconvenience, and for the delayed notice.
Issue Start: 10:00 UTC, April 15th, 2025 Issue Stop: 10:45 UTC, April 15th, 2025
Apr 15, 08:00 UTC