We have identified a soft-lockup condition on one of our core switches.

This condition has resulted in the affected core switch to pass health checks, however, fail to correctly route traffic.

We have worked with the equipment vendor, and have replaced the affected core switch.

Telemetry indicates this is a result of the power feed issues on 06-Oct-2024.

We have since observed a full restoration of service.

What happened?

Tuesday 08-October-24 at

• 19:00: We received alerts from our monitoring systems that our primary power feed had dropped offline. No disruption to service availability at this point.
• 19:20: We receive alerts that several access (feed) switches have gone offline and are unreachable internally.
• 20:59: We receive alerts that the offline access switches have come back online and are carrying traffic correctly.
• 21:05: We manually verify that workloads are back online...

We are aware of an issue in Telehouse North affecting multiple networks (Google, AWS etc).

This issue is caused by high temperatures, causing equipment to go into over-temperate protection.

Our services are fully online, and traffic has been gracefully routed away through alternative points of presence.

All of our equipment is currently running within normal operating temperatures.