Emergency Maintenance: cPanel11 Hosting Server
Replacement Drives, Migration of Data and Hardware Upgrades
What is being done:
Replacement of SSD Hard Drives and Migration of all existing data to the new disks, in order to stabilise an intermittent Disk-IO/Load issue, addition of more RAM and reconfiguration of caching, intrusion-detection systems, extra firewalling and other tweaks.
Who is affected:
Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:
- cPanel11
The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.
During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc
The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.
This will involve a maximum of 12 hours of outage of all services (mail, web, databases etc) on server cp11 and some limited access for an additional 48 hours whilst the impact and improvements are monitored and tweaked.
We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.
Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.
Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.
FTP upload/download access will return once we have confirmed all data is available.
Access to other services/ports where supported will then be made available again.
When is this maintenance:
Starting Saturday 01 November 2014 22:00
Completing Monday 03 November 2014 10:00
Why do we need to do this:
Clients are reporting issues with ‘load’ related problems with MySQL DB driven websites and intermittent mail access at some times.
Investigations into these issues have found a number of causes, some which have been mitigated to the extent they can be, by a variety of means:
– attempts to ddos certain clients
– client setup issues triggering the IDS
– customers with ‘service abusing’ processes
– use of technologies banned from our shared hosting service
etc
However there appears to be some which will require more advanced analysis, which itself is increasing the load and IO, so inadvertently extending the problems – a ‘fix’ is to make hardware improvements to solves many of the underlying issues, and to allow a much greater suite of tools to be used simultaneously to find ‘abusers’
We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that whilst further improvements, checks, logging and investigation go on, services continue to operate for our clients at the usual high-availabilty ultra-fast levels we aim to always provide.
updates
2014-11-02 10:00 – Server returned to service with staff access only
2014-11-02 10:30 – All Email Services returned to service – imap/pop3/smtp – exc. webmail
2014-11-02-12:00 – All static websites returned to service
2014-11-02 13:00 – Webmail returned to service
2014-11-02 13:30 – All MySQL DB services returned to service after full mysqlcheck
