APAC Network Degradation

Minor Incident Asia Pacific Global Network and Backbone
7 hours, 45 minutes

Update

Resolved

We are no longer seeing packet loss or high levels of latency and as a result are resolving this incident, even though it remains under monitoring for repeat events.

Mar 14th, 2020 18:29 UTC
Monitoring

We have seen route changes for NTT showing that Europe <> APAC connectivity has been restored to normal. We are still awaiting a notification from NTT and will provide further updates as they become available.

Mar 14th, 2020 17:28 UTC
Monitoring

Pending updates from NTT on backbone restorations between Asia and Europe, we are observing that traffic within the region is within acceptable levels.

We will keep this incident open and update as soon as further information is available.

Mar 14th, 2020 15:31 UTC
Investigating

TATA reports that their TGN-EA cable capacities have been restored. We are awaiting further updates.

Mar 14th, 2020 14:04 UTC
Investigating

NTT confirms they also have multiple major outages between EU and APAC. They are waiting for updates from their underlying cable providers.

Total Uptime engineers have moved the vast majority of traffic to alternate links and are continuing to monitor routes and capacities.

Mar 14th, 2020 13:47 UTC
Investigating

Telia Carrier has also confirmed backbone issues which has also impacted APAC region traffic.

Mar 14th, 2020 11:11 UTC
Investigating

TATA has confirmed backbone capacity issues between Singapore and Los Angeles as well as Mumbai to Marseilles. We have moved most of our traffic away from TATA and continue to seek alternate paths for further mitigation.

Mar 14th, 2020 10:50 UTC
Issue

We are currently investigating reports of degraded network performance including additional latency and packet loss across portions of our network in the APAC region. Engineers are actively investigating the issue at this time. We will post updates here as they become available. Thank you for your patience as we work to restore service to normal levels as quickly as possible.

Mar 14th, 2020 10:48 UTC