Mar 30

RFO 150320: Reason For Outage – 20/March/2015

Astutium would like to apologise for the unplanned outage to some services earlier this morning.

What went wrong?

At approximately 8.56am, a management card failed and then proceeded to stop switching and passing traffic which affected some customers:

  • on single homed connections
  • on our shared hosting platform
  • on the legacy virtual private server OVZ
  • on the virtual dedicated server Xen based systems
  • some dedicated servers
  • single upstream connection co-location customers

Clients using BGP or with multiple connections to multiple switches, or where we offer transit and routing were not affected.
The problem was limited to equipment utilised by customers in rack rows C, D, and E in Global Switch 2.
All other customers in Global Switch 2 as well as in Telehouse and our overseas data centres were not affected.

Whilst the network remains fully operational and world wide accessible during this time, unfortunately as affected the switch was not correctly passing traffic onto some racks, the devices within those racks were uncontactable.

By 9.18am engineers were on site and had diagnosed the problem was with the switch and at which point they decided to replace the management card. The existing card was removed and the new card inserted.

Software was loaded onto the switch by 9.25am and operations started to resume, although it did take between 7 -12 minutes for different servers and different customers arp requests and customer switches and hardware to pick up the new system and start taking traffic again.

By 9.41am all operations should have reverted to normal.

What can we do about this in the future?

Unfortunately unplanned events do happen and hardware does fail.

Whilst the Astutium policy is to replace all equipment before the end of its’ projected working life, as well as to actively monitor all servers and services throughout the network, it is unfortunate that some things just blow up when not expected to and can take some time to fix.

There are some things that can be done at the customer end in order to minimise the impact of a single item failing, such as a BGP connection from multiple routers or multiple cables to diverse switches to duplicate connections, it is not always practical or cost effective.

If you would like to talk about the options for a Higher Availability service to take you…
• from 99% to 99.5% (or to 99.95%)
• from 99.5% to 99.9% (or to 99.95%)
• from 99.95% to 99.99%
• or even to a full 99.999%
H-A service please contact the Sales Team to discuss the options.

Mar 20

COMPLETED: Planned Maintenance: cPanel02

Planned Maintenance: cPanel02 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel02
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp04 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Friday 3 April 2015 18:00
Completing Monday 6 April 2015 06:00

This is the resheduled maintenance from 19th Dec 2014 which was postponed due to hardware incompatibility

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

updates

Mar 05

COMPLETED: Planned Maintenance – CPanel04

 

Planned Maintenance: cPanel04 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel04
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp04 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Sunday 15 March 2015 18:00
Completing Tuesday 17 March 2015 06:00

This is the resheduled maintenance from 19th Dec 2014 which was postponed due to hardware incompatibility

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

updates

2015-03-16 19:00 – All data restored to new drives and hardware swapped over
2015-03-16 19:30 – All Email Services returned to service – imap/pop3/smtp – exc. webmail
2015-03-16 19:45 – All static websites returned to service
2015-03-16 20:00 – Webmail returned to service
2015-03-16 20:15 – All MySQL DB services returned to service after full mysqlcheck

Maintenance is completed – if you are currently seeing any issues with your service on hosting server cpanel08 please open a ticket through the client portal

Feb 18

COMPLETED: Planned Maintenance – CPanel08

Planned Maintenance: cPanel08 Hosting Server

Planned Maintenance: cPanel08 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel08
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp08 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Friday 20 February 2015 18:00
Completing Sunday 22 February 2015 06:00

This is the resheduled maintenance from 19th Dec 2014 which was postponed due to hardware incompatibility

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

updates

2015-02-20 20:15 – Data restore completed – server and all services returned to active work pending software and control panel upgrades tomorrow
2015-02-21 07:40 – OS Updates, Security Patches, CPanel/WHM Updates and Softaculous Updates all completed with no outages/problems

Maintenance is completed – if you are currently seeing any issues with your service on hosting server cpanel08 please open a ticket through the client portal

Feb 13

COMPLETED: Planned Maintenance – CPanel10

Planned Maintenance: cPanel10 Hosting Server

Planned Maintenance: cPanel10 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel10
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp08 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Saturday 14 February 2015 22:00
Completing Monday 16 February 2015 06:00

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

updates

2015-02-15 23:15 – Data restore completed – server returned to service with staff access only
2015-02-15 05:40 – All Email Services returned to service – imap/pop3/smtp – exc. webmail
2015-02-15 06:25 – All static websites returned to service
2015-02-15 06:55 – Webmail returned to service
2015-02-15 07:15 – All MySQL DB services returned to service after full mysqlcheck

Maintenance is completed – if you are currently seeing any issues with your service on hosting server cpanel10 please open a ticket through the client portal

Dec 12

CANCELLED: Planned Maintenance – CPanel08

Planned Maintenance: cPanel08 Hosting Server

Planned Maintenance: cPanel08 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel08
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp08 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Friday 19 December 2014 22:00
Completing Sunday 21 December 2014 06:00

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

updates

Nov 28

COMPLETED: Planned Maintenance – CPanel09

Planned Maintenance: cPanel09 Hosting Server

Planned Maintenance: cPanel09 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hotsing services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel09
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 6 hours of outage of all services (mail, web, databases etc) on server cp09 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Saturday 29 November 2014 22:00
Completing Monday 01 December 2014 06:00

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

updates

2014-11-03 03:15 – Data restore completed – server returned to service with staff access only
2014-11-03 05:40 – All Email Services returned to service – imap/pop3/smtp – exc. webmail
2014-11-03 06:25 – All static websites returned to service
2014-11-03 06:55 – Webmail returned to service
2014-11-30 07:15 – All MySQL DB services returned to service after full mysqlcheck

Maintenance is completed – if you are currently seeing any issues with your service on hosting server cpanel09 please open a ticket through the client portal

Nov 16

COMPLETED: Planned Maintenance – CPanel06

Planned Maintenance: cPanel06 Hosting Server

Emergency Maintenance: cPanel06 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis from Emergency Maintenance – CPanel06 on 16/November, and reconfiguration of caching, intrusion-detection systems, extra firewalling and other tweaks.

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel06

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 6 hours of outage of all services (mail, web, databases etc) on server cp06 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Monday 17 November 2014 05:00
Completing Tuesday 18 November 2014 11:00

Why do we need to do this:

A disk-corruption has occurred, which required previous maintenance and downtime to repair. On completing a full scan of all drive arrays, datacentre technicians decided that a full replacement woudl be necessary at some point in the future. In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and elimiate the ‘pending fail’ possibility of the drive array.

updates

2014-11-17 07:55 – Data restore completed – server being returned to service with staff access only
2014-11-17 08:00 – All Email Services returned to service – imap/pop3/smtp – exc. webmail
2014-11-17 08:10 – All static websites returned to service
2014-11-17 08:20 – Webmail returned to service
2014-11-17 09:20 – All MySQL DB services returned to service after full mysqlcheck
2014-11-17 22:00 – All services have been fully operational for 4+ hours load is down, peformance is up

Maintenance is completed – if you are currently seeing any issues with your service on hosting server cpanel06 please open a ticket through the client portal

Nov 16

COMPLETED: Emergency Maintanence – CPanel6

Completed: Emergency Maintenance: cPanel11 Hosting Server

What is being done:

At present the drives are being checked for consistency and being reorganised due to potential issues being reported by the drives.

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel6

 

When is this maintenance:

Starting Sunday 16 November 2014 19:00
Completing – Presently unknown

Why do we need to do this:

The current disk drives in this server are reporting potential problems.  We are taking this action at this stage to prevent further issues later on.  Depending on the outcome of the initial work, the drives may need to be replaced.

Updates:

  • 16/11/2014 20:53 – Services have been restored at present.  Another window will be scheduled to replaced the disks.

Nov 01

COMPLETED: Emergency Maintenance – CPanel11

Emergency Maintenance: cPanel11 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, in order to stabilise an intermittent Disk-IO/Load issue, addition of more RAM and reconfiguration of caching, intrusion-detection systems, extra firewalling and other tweaks.

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel11

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 12 hours of outage of all services (mail, web, databases etc) on server cp11 and some limited access for an additional 48 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Saturday 01 November 2014 22:00
Completing Monday 03 November 2014 10:00

Why do we need to do this:

Clients are reporting issues with ‘load’ related problems with MySQL DB driven websites and intermittent mail access at some times.

Investigations into these issues have found a number of causes, some which have been mitigated to the extent they can be, by a variety of means:
– attempts to ddos certain clients
– client setup issues triggering the IDS
– customers with ‘service abusing’ processes
– use of technologies banned from our shared hosting service
etc

However there appears to be some which will require more advanced analysis, which itself is increasing the load and IO, so inadvertently extending the problems – a ‘fix’ is to make hardware improvements to solves many of the underlying issues, and to allow a much greater suite of tools to be used simultaneously to find ‘abusers’

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that whilst further improvements, checks, logging and investigation go on, services continue to operate for our clients at the usual high-availabilty ultra-fast levels we aim to always provide.

updates

2014-11-02 10:00 – Server returned to service with staff access only
2014-11-02 10:30 – All Email Services returned to service – imap/pop3/smtp – exc. webmail
2014-11-02-12:00 – All static websites returned to service
2014-11-02 13:00 – Webmail returned to service
2014-11-02 13:30 – All MySQL DB services returned to service after full mysqlcheck