Incidents | CilixCloud Incidents reported on status page for CilixCloud https://status.cilix.cloud/ https://d1lppblt9t2x15.cloudfront.net/logos/3a70572ce4abf82e367d9ca1145b1cd8.png Incidents | CilixCloud https://status.cilix.cloud/ en Scheduled Maintenance: VPS Migration to THN2 https://status.cilix.cloud/incident/902608 Fri, 22 May 2026 08:00:57 -0000 https://status.cilix.cloud/incident/902608#3228df63383cff584eaaadab88a589ac3da7d490f3335e53ea9cf76e82f1bd46 We will be migrating VPS services to THN2 from 0900 to 1300 on 22nd May 2026. We expect services to be restored in under 2 hours, however, we have allowed for 4 hours. VPS Availability Issues https://status.cilix.cloud/incident/845999 Tue, 10 Mar 2026 11:25:00 -0000 https://status.cilix.cloud/incident/845999#ac9b4c727f9d5bdf05740690d906527b78d9ba93904a85e284b9f4575a14cd97 This has been traced to a NIC issue in the impacted hypervisor host. The on-site team has replaced the impacted NIC, and affected services are now back online. VPS Availability Issues https://status.cilix.cloud/incident/845999 Tue, 10 Mar 2026 11:00:00 -0000 https://status.cilix.cloud/incident/845999#5daf661be2d746f91e4a098235c83e444ce1fca8b9ee48e4bc439ce2bc9bc824 We are aware of an issue impacting the availability of Virtual Private Servers on node lon1-a5-u18. We are currently investigating. Broadband Services https://status.cilix.cloud/incident/757840 Wed, 05 Nov 2025 19:08:00 -0000 https://status.cilix.cloud/incident/757840#4783329e4555f302a83de22cf7826302e9f9c389811b6da902893a1a89b57b0d Broadband services have been stable over the last 24 hours. Broadband Services https://status.cilix.cloud/incident/757840 Tue, 04 Nov 2025 18:42:00 -0000 https://status.cilix.cloud/incident/757840#d06f3971ed4b7273f2a8e33ec04b240b468d1b87ccc3bf0336e35372f1766378 We have seen a recurrence of the session & connectivity drop. We have informed ICUK (the wholesale provider), who are currently working towards service restoration. Broadband Services https://status.cilix.cloud/incident/757840 Tue, 04 Nov 2025 15:28:00 -0000 https://status.cilix.cloud/incident/757840#34e82d59dd66a26c4bdcddf213521d629ee1ddae51a298c94b50f91472afc594 All affected services are currently back online. We are working with ICUK closely to mitigate any further impact. Further updates will be provided as required. Broadband Services https://status.cilix.cloud/incident/757840 Tue, 04 Nov 2025 15:00:00 -0000 https://status.cilix.cloud/incident/757840#202d652d0b9fac42699cdd004ef356d3b8d33adeb845e687b51034363c1739e3 We are experiencing intermittent service as ICUK appears to be handling several denial-of-service attacks. Further information will be provided as required. Broadband Services https://status.cilix.cloud/incident/757840 Tue, 04 Nov 2025 11:40:00 -0000 https://status.cilix.cloud/incident/757840#2cf64e5d50aafd262a8d513929d6dfd2fb811de24ec93f62c67d55da73b9073e We are aware of a fault affecting broadband services via ICUK starting at 11:40 AM. We will update this page as more information is available. WAF Service Availability https://status.cilix.cloud/incident/744286 Wed, 15 Oct 2025 20:37:00 -0000 https://status.cilix.cloud/incident/744286#269563829c7417181642d7291b4a7bf298ed7b18672cf7df3b18f7000805c6fd #RFO# ##Date and Time of Incident:## 15-Oct-2025, 20:28 - 20:49 ##Timeline of Events## - 20:28: Availability alerts triggered, indicating a loss of Web Application Firewall cluster 1. - 20:32: Technical teams begin fault investigation. - 20:40: Issue identified as a locked-up worker node. - 20:45: Worker node removed from WAF cluster. - 20:49: Service availability returns to normal. ##Summary of impact## One of the WAF worker nodes in cluster 1 experienced a software failure that caused the TCP session state table to fail synchronising correctly across the WAF cluster. This, in turn, led to the cluster falling out of synchronisation and failing to pass traffic correctly. Services routing through the affected WAF cluster were unavailable for 21 minutes. ##Resolution Status## We have multiple redundancies in place to prevent a total outage from happening should a worker node fail entirely; however, in this case, a worker node experienced a very unique partial failure, where it was sending OKs to health checks, but wasn’t healthy. We have engaged with our WAF vendors TAC, and will deploy remedial fixes as soon as they are made available. As a precaution, we will be reloading all worker nodes overnight. No downtime is expected from this change. WAF Service Availability https://status.cilix.cloud/incident/744286 Wed, 15 Oct 2025 19:49:00 -0000 https://status.cilix.cloud/incident/744286#1af03ef9ab6be097b4d5197fa364c644dd4c06aedf4776077de200fead070f3e Services are now back online. WAF Service Availability https://status.cilix.cloud/incident/744286 Wed, 15 Oct 2025 19:42:00 -0000 https://status.cilix.cloud/incident/744286#d136ce697763457c233ab395125f6ef88318ecdc5dfe0f84d65668a20791e4a5 We have identified the issue and are working to bring the services back online. WAF Service Availability https://status.cilix.cloud/incident/744286 Wed, 15 Oct 2025 19:32:00 -0000 https://status.cilix.cloud/incident/744286#a4ae49ddd1b57e2b6c3a8cf840c53f7aea065ba4477727f650ddccc4b4700d21 We are investigating availability issues with our Web Application Firewall. Broadband: L2TP Provider Issues https://status.cilix.cloud/incident/738761 Thu, 09 Oct 2025 07:45:00 -0000 https://status.cilix.cloud/incident/738761#d462e097a6b02c6ef1d93324a7767889df67bc1e2125cd2e211b74a19e2e8b17 The provider has confirmed this disruption was a result of a targeted DDoS attack against their core network. They have taken measures to filter the malicious traffic out and have seen full service restoration. Broadband: L2TP Provider Issues https://status.cilix.cloud/incident/738761 Mon, 06 Oct 2025 21:23:00 -0000 https://status.cilix.cloud/incident/738761#a6493a5918c5a4a1c5ef691c5831f33ca7f7f4a00958b0edb527399974132aaf Services have been stable for the past 40 minutes. We will follow up with an RFO shortly. Broadband: L2TP Provider Issues https://status.cilix.cloud/incident/738761 Mon, 06 Oct 2025 20:40:00 -0000 https://status.cilix.cloud/incident/738761#569c1262f7883866a78ffc76bc2314f7ce71d3048a86d0cd6faf33dc8b5f7cd5 We are seeing the majority of broadband sessions back online. Broadband: L2TP Provider Issues https://status.cilix.cloud/incident/738761 Mon, 06 Oct 2025 19:25:00 -0000 https://status.cilix.cloud/incident/738761#9818d4c6014c8f59de4192e6ae5d9c2fc9c01d3dd8e0538c82ec39f6da077c3f The provider in question has acknowledged the issue and is working towards a resolution. Broadband: L2TP Provider Issues https://status.cilix.cloud/incident/738761 Mon, 06 Oct 2025 19:00:00 -0000 https://status.cilix.cloud/incident/738761#14625414c5648ea3a52265d73b1925bed9156ac78d6a7614b3b555a6efd988bc We have identified the source of this fault to an upstream provider. We have engaged with the L2TP provider for further information. Broadband: L2TP Provider Issues https://status.cilix.cloud/incident/738761 Mon, 06 Oct 2025 18:42:00 -0000 https://status.cilix.cloud/incident/738761#0c4f4dfcdee82e54de6fca664d280241456a918633c3ef3510b8123d447a6406 We are aware of a fault affecting L2TP dynamic broadband services starting at 19:42. L2TP BGP sessions are currently flapping, resulting in sporadic broadband connectivity. Ethernet Circuits are currently unaffected. We will update this page as more information is available. Scheduled Maintenance: Web Application Firewall https://status.cilix.cloud/incident/725069 Sat, 20 Sep 2025 02:00:58 -0000 https://status.cilix.cloud/incident/725069#fbaafdacaef57548aecf970c37472c8c4e3ef3c5e9d8d1fcd3c9833093609340 We will be performing code upgrades on our Web Application Firewall cluster. Traffic will be moved away from nodes while upgrades are applied. Applications routing through the WAF cluster may experience up to 5 minutes of disruption. Planned Maintenance: Sentinel Web Application Firewall Capacity Enhancements https://status.cilix.cloud/incident/705964 Sat, 16 Aug 2025 04:00:00 -0000 https://status.cilix.cloud/incident/705964#63b829fe68138606e3235fe848b02514bcb99517812c6537d174f50133fec649 Maintenance completed Planned Maintenance: Sentinel Web Application Firewall Capacity Enhancements https://status.cilix.cloud/incident/705964 Sat, 16 Aug 2025 00:00:00 -0000 https://status.cilix.cloud/incident/705964#4a1d49dd2061557af73b4e73f01795ec614ffb5892a70c4decb7e6e284c2c60e ##What's Happening?## On 16-Aug-25, we will be making changes to our Sentinel Web Application Firewall system to increase its request processing capacity significantly. These changes will involve adding additional compute capacity to our load-balancing and scrubbing infrastructure components. Works are scheduled to begin at 1 AM BST and are expected to be complete by 5 AM BST. ##What's the impact?## Traffic that flows through the Sentinel WAF will experience up to 4 consecutive instances of disruption, each lasting up to 5 minutes. During each instance, customer service using the WAF will be unavailable for up to 5 minutes. We have taken steps to minimise disruption where possible; however, due to the nature of the changes being made, we require all traffic to be halted while configuration is synchronised between infrastructure components. ##Updates## Updates will be provided as required. LON2: LNS Configuration Changes https://status.cilix.cloud/incident/604720 Sat, 28 Jun 2025 02:00:25 -0000 https://status.cilix.cloud/incident/604720#b616b2d3178b1fb8326ae4b43b13929406c13e80678fa632484f70b24c32449b ##What's Happening?  On 28-June-25, we will be performing configuration changes on the following L2TP Network Servers (LNS). These changes are to enable a number of technologies used for providing IPv6 over broadband sessions.  -lns1.lon2.as215638.net -lns2.lon2.as215638.net Any active sessions will be dropped when we apply configuration changes, the LNS' will require reloading.  These works are scheduled to begin at 3AM. ##What's the impact?  Existing broadband sessions will be terminated as the configuration changes are applied and the LNS appliance reloads. Any customers terminating on the above LNS' will experience a brief loss in internet connectivity while their session redials to an available LNS. ##Updates: We will provide updates as required on this page. If you have any queries, please get in touch.  Regards,  CilixCloud Team LON2: Scheduled Maintenance https://status.cilix.cloud/incident/582823 Sat, 31 May 2025 02:00:14 -0000 https://status.cilix.cloud/incident/582823#07dbb1aa0ff3d23b5a37b118973877f5366a3e479c3f2074932cfd52b83fd135 ##What's Happening?## On 31-May-25, we will be performing code upgrades on the following L2TP Network Servers (LNS). * lns1.lon2.as215638.net * lns2.lon2.as215638.net Any active sessions will be dropped before the code upgrade starts. These works are scheduled to begin at 3AM. ##What's the impact?## Existing broadband sessions will be terminated before the code upgrade begins. Any customers terminating on the above LNS' will experience a brief loss in internet connectivity while their session redials to an available LNS. LON1: Dedicated Severs SVI Migration https://status.cilix.cloud/incident/524353 Sat, 08 Mar 2025 03:30:44 -0000 https://status.cilix.cloud/incident/524353#f2ad1cbfa3aea980883752fd38ecdfae506ab370814d6bcfaa133e18b9eb9621 ##What's Happening?## We will migrate the SVIs used for self-service dedicated servers away from our core and onto a dedicated VC switch stack in rack A4. This is part of an ongoing project to migrate our network to a BGP-free, L2-free core, improving performance, reliability and scalability. ##What's the impact?## There will be a brief disruption in connectivity while the SVI is dropped from core switches, and brought up on the new switches. This should last a maximum of 1 minute, while our iBGP mesh reconverges. LON1: Cisco Access Switch Firmware Upgrade https://status.cilix.cloud/incident/523766 Sat, 08 Mar 2025 03:30:00 -0000 https://status.cilix.cloud/incident/523766#1d9e3b799c7ee5a1e3549cd09769cc2a89d6aa0d35f83b2709892b53871780da ##What's Happening?## We will be performing firmware upgrades on Cisco access Switches in racks A3 & A4 to patch a recently discovered CVE vulnerability.  These works are scheduled to take place on 08-March-2025, starting at 03:30 AM. ##What's the impact?## Switches will continue to pass traffic while the new firmware is installed. Once complete, they will reload, which will take approximately 5 minutes. During this time, network connectivity will be unavailable.  Switches will be updated sequentially, to minimise disruption. Customers who take redundant Layer 3 connectivity from us will failover to their backup uplinks automatically. We will cull BGP sessions for customers who advertise their routes to us using BGP before starting the upgrade process. This will gracefully route traffic away from the affected uplinks. Broadband Issues: Telehouse West https://status.cilix.cloud/incident/516796 Sat, 22 Feb 2025 07:21:00 -0000 https://status.cilix.cloud/incident/516796#8a9291b34636b245d9b1c12622fa638e7e0572eb4fd8532424887a3ba429a64e We have observed an extended period of stability, all broadband services are online. We'll update this incident with further details as we receive them. Broadband Issues: Telehouse West https://status.cilix.cloud/incident/516796 Fri, 21 Feb 2025 15:34:00 -0000 https://status.cilix.cloud/incident/516796#a619c249d77321b21e3492bdb4e32b4090528bc11d592abd4b608cb956fab9f5 All broadband sessions are back online - We have not had confirmation that the issues on the provider's side have been resolved, as such, we are still considering broadband services at risk of further disruption. Broadband Issues: Telehouse West https://status.cilix.cloud/incident/516796 Fri, 21 Feb 2025 15:02:00 -0000 https://status.cilix.cloud/incident/516796#619d7b8f4dc8bf0fd1afeeaa00c31244bcff21760ab8549f36a76da8c4480769 We can see broadband sessions are now dialing via alternative paths, with the majority of sessions back online. Broadband Issues: Telehouse West https://status.cilix.cloud/incident/516796 Fri, 21 Feb 2025 14:44:00 -0000 https://status.cilix.cloud/incident/516796#c9510b1aad89a5662b5627cf9c12ae323498e5183074bcbdf8b8138d43b5f3ad We are aware of a network device failure in BTs POP in THW. Sessions are diverting via alternative paths, however, continued issues in the provider's THW POP are causing disruption. We are working to isolate the THW NNIs, and route L2TP traffic via alternative paths. Broadband Issues: Telehouse West https://status.cilix.cloud/incident/516796 Fri, 21 Feb 2025 14:40:00 -0000 https://status.cilix.cloud/incident/516796#a8df48bdec0737c1bda6fb620bc88ef41540f11e15de9a3080322c24c9fe5a18 We are aware of an issue with broadband sessions connecting via BT's Telehouse West POP. We are working with the BT to isolate the root cause. BT Wholesale Connectivity Drop https://status.cilix.cloud/incident/489760 Thu, 02 Jan 2025 11:27:00 -0000 https://status.cilix.cloud/incident/489760#e7d0851844d2f2ddd7495741b4e291c9d69a5e5c1f3689dfab2ce197cc63204c BT NOC has provided the following information: _________________________________________________________ Date and Time of Incident: 02/01/2025, 10:04 AM Summary of Impact: At 10:04 AM on 02/01/2025, a brief disruption occurred, impacting all services routed through the THW POP. Leased Line services restored almost immediately. Broadband services experienced varied recovery times, depending on customer router PPP dialer frequency, with some requiring a manual reboot to re-establish connectivity. Current Status: We are actively investigating the root cause of the incident. Preliminary analysis suggests a potential hardware fault. As part of our investigation, we are consulting with our vendor, Cisco, to determine the specific nature of the issue and identify any required remedial actions. Resolution Status: Pending further investigation and vendor feedback. We apologise for any inconvenience caused and are committed to resolving this issue as quickly as possible. BT Wholesale Connectivity Drop https://status.cilix.cloud/incident/489760 Thu, 02 Jan 2025 10:06:00 -0000 https://status.cilix.cloud/incident/489760#bdcc2f752a0c3b9c7729d5e25bd5984251b0665daca9a069f4c101ebd6e31821 All affected services are back online and experienced approximately 90 seconds of disruption before failing over to alternative infrastructure. We are investigating the issue with BT and will provide updates as required. BT Wholesale Connectivity Drop https://status.cilix.cloud/incident/489760 Thu, 02 Jan 2025 10:04:00 -0000 https://status.cilix.cloud/incident/489760#1bd124f5b7d26ced28ca2145e0cb861ad7e84291dbf1fffbbda1c99e99b7771d We are aware of connectivity issues for BT Wholesale broadband and leased line circuits terminating in Telehouse West. We are investigating these issues and will provide updates in due course. DC1: Power Feed Issues https://status.cilix.cloud/incident/440763 Tue, 08 Oct 2024 14:18:00 -0000 https://status.cilix.cloud/incident/440763#02d13cf898dd84cdb81177b74d444a119176cc0f9cb58759a3c6b360de6a2d97 # What happened? ### Tuesday 08-October-24 at * 19:00: We received alerts from our monitoring systems that our primary power feed had dropped offline. No disruption to service availability at this point. * 19:20: We receive alerts that several access (feed) switches have gone offline and are unreachable internally. * 20:59: We receive alerts that the offline access switches have come back online and are carrying traffic correctly. * 21:05: We manually verify that workloads are back online and functioning as expected. # What went wrong? After our primary power feed dropped, all equipment successfully failed over to the backup power feed and was performing as expected. In an effort to bring service back online for non-power redundant customers, our colocation provider enabled a transfer switch, that connected the primary and secondary power feeds together. This in turn meant that all the previously offline equipment in the data centre began to power back online. This resulted in a voltage drop from the standard 240v, to approximately 160v. Our core, border and aggregation network equipment feature power supplies that are able to function within a 100-240v range, and as such weren't affected by the voltage drop. However, our access switches feature power supplies that are validated for 220-240v. This led to our access switches powering down automatically, which in turn caused a loss of connectivity to any services not directly connected to our core switches. Once the voltage drop condition cleared, the access switches powered back online automatically, and service availability was restored. # What changes are we making? The service disruption was caused by a failed power feed, which then resulted in out-of-range operating conditions for our access layer switches. As a result, we will be implementing changes to our access layer switches that include: * Firmware configuration changes to adjust the automatic shutdown voltage range. * Replacement power supplies, validated for a 100-240v voltage range. The above changes are scheduled to take place this Saturday and Sunday (12-13 October 2024) We will also provide further updates once we have received a full Root Cause Analysis from our colocation provider. DC1: Power Feed Issues https://status.cilix.cloud/incident/440763 Mon, 07 Oct 2024 19:45:00 -0000 https://status.cilix.cloud/incident/440763#d76da99b0fe530cede900de31ff5386c845f97a6f8c5deaecfd8850ece9ba478 We have seen a full restoration of power at DC1. We are currently manually verifying all equipment has powered on correctly. DC1: Power Feed Issues https://status.cilix.cloud/incident/440763 Mon, 07 Oct 2024 19:30:00 -0000 https://status.cilix.cloud/incident/440763#315846d2c15036a30d6adc57f691c86c988db832126c905a18b2e6b3e72f39b9 We have received an update from the data centre that the issue is related to power systems. They believe they have identified the cause of the issues, and are working on remediation. DC1: Power Feed Issues https://status.cilix.cloud/incident/440763 Mon, 07 Oct 2024 18:59:00 -0000 https://status.cilix.cloud/incident/440763#497ee5768c5a3022a7cf25b8026e2925f0bc23f614cd047c91390eb676c6890e We are aware of an issue affecting the availability of our infrastructure. Maintenance: Network Infrastructure Upgrades https://status.cilix.cloud/incident/427496 Sat, 14 Sep 2024 04:00:00 -0000 https://status.cilix.cloud/incident/427496#581f5d60181913f27aa8c23a1449e131619a2417f2f665e1633a4780ec36477a We will perform maintenance on our network infrastructure in our London 1 location to upgrade our access layer switches to 25Gb capable equipment. As part of this, we will be re-patching hypervisor nodes into the new access switches, however, we will re-route traffic through alternate uplinks beforehand. Dedicated server customers with the prefix "lon1-a4" will be unaffected by the maintenance. We expect disruption to be minimal and will update this page with more information as required. Self Service Portal: Software Upgrades https://status.cilix.cloud/incident/406059 Thu, 01 Aug 2024 14:00:00 -0000 https://status.cilix.cloud/incident/406059#374e776c479932d0e06af797b78d0a0ac61e4e206b444aa884f43456179c7ceb We will be deploying a software update to our self-service portal. During the update works, the portal will be offline. *Please note, that customer virtual and dedicated servers will remain online and aren't affected by these maintenance works.* DDoS Protection System Integration https://status.cilix.cloud/incident/351449 Sat, 06 Apr 2024 06:00:00 -0000 https://status.cilix.cloud/incident/351449#c10ed78d595d3ff622d9cbe29b48fa3697260f8960fb31fcb8726c00f5b9813a As part of an ongoing project to enhance the security of our infrastructure, we will be integrating Cloudflare's Magic Transit DDoS protection into our infrastructure. As part of these works, we must deploy changes to our routing infrastructure and traffic monitoring systems. These changes will be made during a 2-hour window on the 6th of April 2024, resulting in up to 10 minutes of downtime. We will provide further updates as necessary.