Oct 28

COMPLETED: Planned Maintenance: EU Minecraft Server Maintenance

Minecraft Server Maintenance
from: Friday 30th October 2015 12:00
to: Friday 30th October 18:00

Astutium Multicraft

 

As part of our regular service refresh and ongoing updates to improve performance, we are in the process of replacing a number of acquired, legacy and older minecraft systems with faster and more capable new hardware.

This will allow for “in-place” upgrades to larger and more powerful plans instantly and on-time through our client portal, and for you to get the fastest response and highest uptime possible for your Minecraft Server(s).

When our hardware gets updated/replaced, all files from the MineCraft servers of our clients are moved by our staff to the new equipment, 1-by-1 manually. This ensures that in general you won’t notice anything different about your minecraft server.

However due to the server behind the scenes being different, and the way migrations are managed by the MultiCraft Control Panel it is normal that your IP Address or Port will change.

On Friday afternoon we will …

  • Shut down your Minecraft Server / Minecraft Virtual Machine
  • Manually copy over all your files and settings to the new servers
  • Confirm the copy has completed correctly
  • Assign your new IP/Port and update multicraft
  • Email you the access details for your MineCraft Server
  • Restart the MC server on the new hardware

Notes on Launcher settings …
# If you currently access your Minecraft Server through a launcher using (domain):(port)
and the domain uses our DNS/nameservers, then all changes will be done for you in the dns systems

# If you currently access Minecraft Server through a launcher using (domain):(port)
and the domain uses a different companies DNS/nameservers, then you will need to update your zone A record to the new IP address

# If you are using one of our sub-domains to access your MC-Server, this will be updated for you automatically.

# If you currently access Minecraft Server through a launcher using (ip):(port)
you will need to update the launcher to use a new IP and/or PORT

Details of your new IP and/or new PORT will be sent to you by email as soon as your server has completed being moved.

We expect the actual time to copy your minecraft server to a new machine to be around 30 minutes (this depends a little on the size of your worlds, numbers of installed plugins and setting for number of players) and this will be undertaken on Friday 30th October 2015

We already have a backup copy of the server content on our Cloud platform in London, so any migration can be “undone” as quick as possible if the copy fails for any reason (it is not expected to, but we like to be prepared).

These steps are being taken to improve the performance of your service, and to allow for the anticipated processing requirements/changes needed by 1.9

Additionally, we are now officially retiring all variations of the 1.5.x branch of Minecraft (1.8.x is the current version), so if you’re using the really old 1.5.x versions currently, we advise you upgrade immediately to a newer version through your control panel.

Oct 18

COMPLETED: Downtime: MiniVPS OVH SSD Server

Downtime: MiniVPS OVH SSD Server

The minivps ovz ssdfr1 server is currently offline/not responding

What is being done:

We are waiting on OVH support to restart the server and determine the status of the machine

Who is affected:

Clients with …

  • OVZ-SSD-FR services from MiniVPS
    on servers/systems: ssd1fr

Once the machine is returned to service we will begin the process of migrating virtual private servers to more reliable hardware

When is this maintenance:

Starting 18/October/2015 – 09:02
Completing unknown at this time

Update

OVH Technical Support attended the machine @ 09:52 19/October/2015 and brought the machine back online, IPv4 routing and IPv6 routing was re-established shortly afterwards and all VPS are available again.

If you are still experiencing connectivity issues to yoru virtual machine, please open a ticket for the Support department through the Client Portal at https://www.astutium.com

We will now make plans to migrate the virtual machines to more stable hardware and decomission the current minivps-ssd-fr1 server – detail will be sent individually to affected clients over the coming weeks.

Oct 03

COMPLETED: Emergency Maintenance – DNP1W2K8

COMPLETED: Emergency Maintenance: DotNetPanel Hosting Server DNP1Win2K8

What is being done:

Follwing a fault with service not correctly starting in windows a hard reboot of the server was necessary.
Server is now failing at boot unable to read from sections of the drives/array.
Replacement of SSD primary drives necessary before reinstallation/repair of Windows Core 2k8 packages

Who is affected:

Clients with DotNetPanel Windows .NET Hosting Services (web, email, personal, business hosting packages) on server:

  • DNP1Win2k8

When is this maintenance:

Starting Saturday 03 October 16:00
Completing – Presently unknown

Why do we need to do this:

The current disk drives in this server are reporting physical problems.  We are taking this action at this stage to fix the current issue and to prevent further issues later on.  After the initial diagnosis it was determined that the need to be replaced immediately before attempting to restore service.

Updates:

  • 04/10/2015 01:18 – Data migrated to new SSD drives has reached 50% after 5h32min
  • 04/10/2015 07:35 – All data migrated to new drives, windows repair started
  • 04/10/2015 09:50 – Server fully repaired and updated – returned to service

Sep 30

COMPLETED: Planned Maintenance: US Shared Hosting / Minecraft Servers

Planned Maintenance: US Shared Hosting / Minecraft Servers

What is being done:

Transformer maintenance to prevent inner windings of transformer core from rubbing on outer windings / CET’s (Critical Environment Technicians) to repair a manufacturer defect.

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanelA2

cpanel logo

Clients with US Minecraft Servers:

Multicraft

When is this maintenance:

Starting Tuesday 13 October 2015 10:00 (BST)
Completing: Tuesday 13 October 2015 13:00 (BST)

Why do we need to do this:

Repairing manufacturer defect to prevent an unplanned outage.

Aug 21

COMPLETED: Planned Maintenance: cPanel03

Planned Maintenance: cPanel03 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel03
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp04 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Friday 28 August 2015 18:00
Completing Saturday 29 August 2015 06:00

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

Jun 23

COMPLETED: Planned Maintenance: cPanel01

Planned Maintenance: cPanel01 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel01
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp04 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Friday 28 August 2015 18:00
Completing Saturday 29 August 2015 06:00

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

Jun 04

COMPLETED: Planned Maintenance: cPanel05

Planned Maintenance: cPanel05 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel05
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp04 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Friday 19 June 2015 18:00
Completing Monday 22 June 2015 06:00

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

May 13

COMPLETED: Planned Maintenance: cPanel07

Planned Maintenance: cPanel07 Hosting Server

Replacement Drives, Migration of Data and Hardware Upgrades

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, following the check/repair/analysis of other hosting servers (see Emergency Maintenance – CPanel06 on 16/November) we are continuing the rollout of new SSD drive arrays for shared-hosting, reseller hosting, secure hosting and email hosting services

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel07
cpanel logo

cpanel logo

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 8 hours of outage of all services (mail, web, databases etc) on server cp04 and some limited access for an additional 24 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Friday 29 May 2015 18:00
Completing Monday 1 June 2015 06:00

This is the resheduled maintenance from 19th Dec 2014 which was postponed due to hardware incompatibility

Why do we need to do this:

Several hosting servers have reported disk-corruptions over recent months, which required previous maintenance and downtime to repair on some. On completing a full scan of all drive arrays on all servers, datacentre technicians decided that a full replacement would be necessary at some point in the future for each machine (around a 1 year anniversary from last upgrade). In order to minimise any potential loss of client data, we have decided to do that in a controlled manner as soon as possible.

We are therefore going to be migrating to new drives to increase the IO capabilities, add more ram to upgrade caching for overall performance benefits, and some reconfiguration – so that the server(s) can be returned to service at full capacity and eliminate the ‘pending fail’ possibility of the drive array.

Apr 29

COMPLETED: Emergency Maintenance – cPanel02

Emergency Maintenance: cPanel02 Hosting Server

Replacement Drives and Migration of Data

What is being done:

Replacement of SSD Hard Drives and Migration of all existing data to the new disks, in order to proactively replace them prior to complete failure.

Who is affected:

Clients with cPanel/WHM Linux Hosting Services (web, email, personal, business hosting packages) on server:

  • cPanel02

The server will need to be completely offline during the maintenance to physically swap hard-drives, and restore all data as quickly as possible.

During the maintenance window you will not be able to access any aspect of your hosting accounts through the Client Portal, via WHM/cPanel or by http/ftp/pop3/smtp/imap etc

The servers will need to be powered down, have all drives removed, duplicated and replaced, and other hardware changed before bringing up with limited access whilst services are started in a controlled manner and the reported disk-io and load investigations are completed.

This will involve a maximum of 12 hours of outage of all services (mail, web, databases etc) on server cp11 and some limited access for an additional 48 hours whilst the impact and improvements are monitored and tweaked.

We are unable to provide an exact time for you to regain any access to your hosting account as it will depend on the type of services you use – email only will be back first, followed by ftp, web, databases – if any ‘prioritisation’ of clients is necessary, then those with the advanced-support levels will be 1st, followed by Enterprise & E-Commerce clients, then Business, Personal and finally any trial and subsidised/discounted/internal services.

Incoming email during this time will ‘queue’ at the sender, Outbound emails will stay on your local machine (where using a mail client) – webmail access will be unavailable until the work is completed.

Websites will be down until the data has been restored, Websites which rely on a database will start to work once the mysql databases are restored and repaired.

FTP upload/download access will return once we have confirmed all data is available.

Access to other services/ports where supported will then be made available again.

When is this maintenance:

Starting Wednesday 29 April 2015 19:25
Completing <Unknown at this time>

Why do we need to do this:

Our internal monitoring systems have reported an ‘imminent failure’ of one of the primary disks in this server.  We have taken this action to prevent complete loss of customer data.

Updates

29/April 21:15 Hardware has been replaced and data is being migrated over to the new system
29/April 23:30 All data has been restored and server is being returned to active duty
30/April 04:45 Maintenance and followup testing all completed – all services 100% operational

Mar 30

RFO 150320: Reason For Outage – 20/March/2015

Astutium would like to apologise for the unplanned outage to some services earlier this morning.

What went wrong?

At approximately 8.56am, a management card failed and then proceeded to stop switching and passing traffic which affected some customers:

  • on single homed connections
  • on our shared hosting platform
  • on the legacy virtual private server OVZ
  • on the virtual dedicated server Xen based systems
  • some dedicated servers
  • single upstream connection co-location customers

Clients using BGP or with multiple connections to multiple switches, or where we offer transit and routing were not affected.
The problem was limited to equipment utilised by customers in rack rows C, D, and E in Global Switch 2.
All other customers in Global Switch 2 as well as in Telehouse and our overseas data centres were not affected.

Whilst the network remains fully operational and world wide accessible during this time, unfortunately as affected the switch was not correctly passing traffic onto some racks, the devices within those racks were uncontactable.

By 9.18am engineers were on site and had diagnosed the problem was with the switch and at which point they decided to replace the management card. The existing card was removed and the new card inserted.

Software was loaded onto the switch by 9.25am and operations started to resume, although it did take between 7 -12 minutes for different servers and different customers arp requests and customer switches and hardware to pick up the new system and start taking traffic again.

By 9.41am all operations should have reverted to normal.

What can we do about this in the future?

Unfortunately unplanned events do happen and hardware does fail.

Whilst the Astutium policy is to replace all equipment before the end of its’ projected working life, as well as to actively monitor all servers and services throughout the network, it is unfortunate that some things just blow up when not expected to and can take some time to fix.

There are some things that can be done at the customer end in order to minimise the impact of a single item failing, such as a BGP connection from multiple routers or multiple cables to diverse switches to duplicate connections, it is not always practical or cost effective.

If you would like to talk about the options for a Higher Availability service to take you…
• from 99% to 99.5% (or to 99.95%)
• from 99.5% to 99.9% (or to 99.95%)
• from 99.95% to 99.99%
• or even to a full 99.999%
H-A service please contact the Sales Team to discuss the options.