This issue is now resolved as of 21:20 UTC on September 18th.
Posted Sep 19, 2023 - 14:26 UTC
Monitoring
We are testing a fix, and Rio and Sao Paulo currently do not suffer from any performance issues or packet loss. We are continuing to monitor.
Posted Sep 18, 2023 - 20:58 UTC
Update
We have shifted traffic back to Sao Paulo from Rio, but are still seeing packet loss, though about %10 compared to 70% before. We are continuing to investigate.
Posted Sep 18, 2023 - 18:07 UTC
Update
We have rerouted Sao Paulo traffic to Rio de Janeiro while we investigate further, which only adds ~6ms of RTT to devices routing to Sao Paulo. There should not be any packet loss at this time.
We may have to temporarily re-enable Sao Paulo with the packet loss occurring to continue troubleshooting, but this will be performed during off-hours and in small windows.
Posted Sep 15, 2023 - 19:32 UTC
Investigating
Our Sao Paulo location is currently experiencing heavy packet loss.
We are investigating and will provide updates as they come.
Posted Sep 15, 2023 - 18:32 UTC
This incident affected: Recursive DNS Services (Current Status).