COMPLETED: Major Engineering Works – Services at Risk

DataCenter Major Engineering Works

Power Testing, Upgrades and Replacements

What is being done:

  • Replacement of faulty UPS
  • Checking of Breakers and Wiring
  • Failover Testing of Power Systems

Following the unexpected power issues at Level3/CenturyLink on 8th and 12th of June which caused major disruptions to all datacentre tenants, they are performing major testing and remedial works to identify, isolate, replace and live-check the UPS, power failover and on/off breakers to the 2nd (data-carrier) and 3rd (colocation) floors.

As part of this testing power will be going on and off, hopefully in a controlled manner, throughout the weekend.

This is a significant work, and there is possibility of widespread and *without-notice* power fluctuations, power cuts/terminations, and power related issues such as reboots, systems being depowered, and over/under voltage problems (brownouts).

Astutium staff will be onsite to minimise any outages, reroute power, shutdown and restart (where possible under controlled conditions) and rewire/migrate systems as necessary during the 2 days of engineering works.

It is unfortunate that power may be removed *without warning* from any/all systems as either a repeat of the fault, or as part of the investigation and remedial works, in order to return the datacentre to the redundant and capable levels expected.

Who is affected:

All clients are potentally at risk during this work, as systems/racks/services (and upstream providers/routers) may lose power at an time for any amount of time. This includes

  • Shared Hosting
  • Reseller Hosting
  • Virtual Private Servers
  • Virtual Dedicated Servers
  • Backhaul/Transit Clients
  • Managed Dedicated Servers
  • Colocations
  • VoIP/Telephony Systems

which are all at risk.

When is this maintenance:

Starting: 13/10/2020 22:00
Completing: 15/10/2020 10:00

Why do we need to do this:

It is important for the continued provision of power to the datafloors that the faulty systems which have cause 2 periods of outages to some Astutium systems be checked, replaced and tested to prevent or limit future recurrences and escalating power related problems.

Power sensitive systems (which required physical replacement/repair last time) will be off during any obvious UPS changeovers, tp limit “power shock” causing repeat hardware failures.

IF staff are provided notice by the datacentre at each stage of testing (as we expect and as we have people onsite to try and ensure) then systems will be professionally closed down and restarted cleanly rather than simply power-cycled, which will minimise possible data loss.

We apologise in advance for this issue, and for the inevitable downtime (no matter how minimal) that will occur – this is completely unavoidable and beyond our control.