Executive Summary

On 25th October at 09:15 GMT, monitoring systems detected a loss of network availability to several services connected to access switches in a Virtual Chassis configuration, in racks A2, A3 and A4, triggering a major incident response.

Upon investigation, the pair of access switches had moved into a ‘soft-lockup’ state, which meant the switches were actively advertising their availability to carry traffic, but were unable to do so.

To rectify the situation, our on-si...

What happened?

Tuesday 08-October-24 at

• 19:00: We received alerts from our monitoring systems that our primary power feed had dropped offline. No disruption to service availability at this point.
• 19:20: We receive alerts that several access (feed) switches have gone offline and are unreachable internally.
• 20:59: We receive alerts that the offline access switches have come back online and are carrying traffic correctly.
• 21:05: We manually verify that workloads are back online...

We are aware of an issue in Telehouse North affecting multiple networks (Google, AWS etc).

This issue is caused by high temperatures, causing equipment to go into over-temperate protection.

Our services are fully online, and traffic has been gracefully routed away through alternative points of presence.

All of our equipment is currently running within normal operating temperatures.