Emergency Maintenance : Hosting : Toby-DH/PleskWin3

Server PleskWin3 (toby) is under maintenance following a failed PSU @ appx 8.45am

Power supply has been replaced, but disks are not being recognised. It may be necessary to replace the entire machine and restore data from last good backup – more details will be added to the thread as they become available.

Update

The drives have been replaced and RAID array rebuilt – we are now performing a bare metal restore.

Update

The restore is currently at 21%.

Update

The restore is currently at 38%.

Update

The restore is currently at 56%.

Update

The restore has now completed – we are performing some final checks before bringing the machine back up.

Update

The server Toby-DH/PleskWin3 is now back online.

If you are still experiencing issues please raise a support ticket at http://support.othellotech.net/

Update

We have received scattered reports of additional problems with this server. At this point we have not been able to verify the cause, more updates will be provided as they are available.

Update

Remote diagnostics have not assisted, an engineer has been dispatched for physical diagnostics.

Update

There was more degredation to the drive arrays for this server. It is currently back in service, however we expect to be performing a hardware migration very soon.

More details of the migration will be provided as they happen.

Update

The server has failed again and destroyed the data on the RAID.

We have replaced the server and the restore is in progress.

Update

The restore is currently at 14% – we will post further updates as they are available.

Update

The restore is currently at 42%.

Update

The restore is currently at 59%.

Update

The restore is currently at 67%.

Update

The restore is currently at 74%.

Update

The restore is currently at 89%.

Update

The restore is completed and the disk verify is at 56%.

Update

The disk verify phase has now completed. Engineers have now turned their attentions to repairing the software differences due to the replaced hardware.

Update

Despite our best efforts, recovery using the CDP (Constant Data Protection) Block Level Backup has failed. This is most likely due to the incremental nature of the original disk failures.

The server is being re-provisioned onto another set of disks and we will attempt to re-provision and recover whatever files are possible from the block restored disks. Unfortunately this means that some customer data may be unrecoverable.

Again, we will provide updates as work progresses.

Emergency Maintenance : Hosting: cPanel4

All,

We apologise for the outage in service, this host has developed some filesystem corruption and we were forced to take it out of service to correct it.

At this stage engineers are actively working on the issue but there is no estimate available as to how long it will take to complete.

We will post additional updates as they become available.

Update

The first pass repair has completed, however due to the type of corruption detected additional passes may be required.

Emergency Maintenance : Hosting : DesignerHosting (DC Hosting)

At 09:32 we detected a fault with the datacentre systems for the old DesignerHosting / DC-Hosting servers, this was an unscheduled outage – an engineer is current en-route to investigate. As this was unscheduled, we do not have an ETA for it to be working again. We will keep you apprised as the engineer reports back.

Update: 09:56
Power issue at Cable & Wireless datacenter affecting a number of services.

DC-Hosting
cp.dc-servers.com
mail.dc-servers.com
ns1/2.dc-servers.com
smtp.dc-servers.com
web*.dc-servers.com
win*.dc-servers.com

DesignerHosting:
duck.designerhosting.co.uk
edward.designerhosting.co.uk
percy.designerhosting.co.uk
emily.designerhosting.co.uk
toby.designerhosting.co.uk
thomas.designerhosting.co.uk

Racks: E10, E09, E08, E07, E06, E05

Service has been restored to all services listed above except:

thomas.designerhosting.co.uk
win4.dc-servers.com
win7.dc-servers.com

Update

Service has been restored to all services listed above except:

win4.dc-servers.com

Emergency Maintenance : Hosting : Percy

This server has suffered an unknown failure an engineer has been dispatched to investigate.

At this point we have no ETA for service restoration. More information will be posted as it becomes available.

Update

This server had developed a filesystem inconsistency again, which has now been repaired. We apologise for the unexpected outage.

Emergency Maintenance : Hosting : Web4 (DC Hosting)

This server has suffered an unknown failure, engineers are working on this at present.

Additional checks are being made due to its recent replacement – we hope to have this back online before the morning.

Update

A recent Microsoft Update for this server required manual intervention with the console to restore service. We apologise for the inconvenience, and were unaware of this requirement.

Emergency Maintenance : HostAssist

We apologise for the unscheduled outage of the HostAssist system, this was due to an unexpected update which required a rollback.

Engineers are working on the task and all e-mail tickets are being queued. If you require non-urgent support, please e-mail support@othellotech.netand the engineers will respond once the update is complete.

Update

HostAssist is now available again. Tickets that were submitted whilst it was unavailable are filtering in. Any tickets that were submitted whilst the upgraded version was in place (ticket IDs start 9) will be added back shortly.

Emergency Maintenance : Hosting : cpanel3

Customers with services provided by CPanel3 will shortly be experiencing an extended downtime this afternoon. We have been alerted to a failing drive and this must be replaced to prevent data loss.

At this point we do not have an exact length of the outage, however must stop several services to maintain customer data.

We will keep customers up to date via this forum posting.

Update

Replacement equipment is ready; we are waiting on the completion of the data migration from the existing disks. This is running slower than normal due to the failing disk.

We will continue to provide updates when available.

Update

The data migration is now completed. The equipment is in the process of being swapped, the the restoration can begin. We expect this to be quicker than the migration as the disks are not faulty.

Update

Data restoration is now in progress. We will update further when this is complete.

Update

Data restoration is now complete. There are a couple of observed issues which will be cleared up in the next 30 minutes.Services should now be restored to all users.

Planned Maintenance: Hosting : R4B – Migration (Chorver)

Friday 25/06/2010 20:00

Due to failing hardware on this server we will be migrating clients over to our more modern cPanel servers.

All clients have been notified of their new cPanel login details and they should contact support if they require any assistance setting up mail accounts before the migration process.

Emergency Maintenance : Hosting : cpanel4 (Disk)

Customers with services provided by cpanel4 will be experiencing an unexpected, unplanned outage due to essential disk maintenance on this server.

At this point we do not have an exact time for services to be restored, this will ultimately be decided by how much corrective action needs to be taken after the integrity scans complete.

Engineers are working on this as this is posted and are working to restore service as soon as possible. We apologise for any inconvenience in the mean time.

Update

The disk inconsistencies have been repaired. Final checks are being made and services will return of the next 5 minutes.

We apologise again for any inconvenience.

Emergency Maintenance : Domain Registration : [.eu .es .cn .at]

We have received a notification of unscheduled hardware updates from the domain registries regarding

.eu domains
.es domains
.cn domains
.at domains

Start date: 2010-04-30, 15:00
End date: 2010-05-01, 07:00

During the maintenance, you may expect the following:

– No new registrations of domains
– No nameservers updates of affected ccTLD’s
– No renewals of of affected ccTLD’s
– No WHOIS services for the affected ccTLD’s
– No Transfers-IN of domains
– No Unlock/Lock of affected ccTLD’s within the Othello Client Portal
– No EPP Transfer Code requests within the Othello Client Portal

DNS services, hosting services, email services etc are not affected and will continue as normal.

Please accept our apologies for informing you at this short notice.