Dec 16

COMPLETED: Emergency Maintenance : Hosting : Plesk1 (Duck)

Server Plesk1 (Duck) has failed with a corrupt RAID array.

DataCentre technicians are onsite investigating if it can be repaired and returned to service.

If the current server is unrecoverable a new plesk server will be deployed and all sites/emails etc restored from backups.

ETA for server availability / next update 12:30 16/December/2011

Update 1:
We apologise for the lateness of the update; repairs are continuing to progress, data recovery is at approximately 10%.
Next update: 15:00 16/December/2011

Update 2:
This server has now been restored to service.  We apologise for the extended downtime, this machine is being scheduled for replacement in the near future.  E-mail will begin to be delivered over the next few hours, however this depends on the sending server’s retry schedule.
If you have any further issues, please contact support via the usual channels.

Dec 13

COMPLETED: Planned Maintenance: Hosting : CPanel/WHM Upgrades

What:

Planned roll out of the latest version of centos, cpanel/whm and script-installers to shared-hosting, managed-dedicated, managed-cloud and managed-vps servers.

Who (affects):

This planned maintenance affects all

When :

starting Saturday 17/December 2011 01:00 (GMT)
completing Saturday 17/December 2011 07:00 (GMT)

Why:

There has been a significant number of increases in attempted hacks and attacks on shared-hosting systems in the last few weeks – this update will add additional security and logging features, as well as a few minor interface bug-fixes (cosmetic only) and a roll-out of the latest tested PHP and Apache Updates

These updates are not expecting to be service affecting beyond a 1-2 minute server restart at the end of the updates/compiles/installations.

Nov 24

COMPLETED: Emergency Maintenance : Hosting : Percy

This server has suffered an unknown failure, engineers have been dispatched to investigate.

At this point we have no ETA for service restoration. More information will be posted as it becomes available.

Update

This server had developed a filesystem inconsistency, which has now been repaired. We apologise for the unexpected outage.

Sep 13

Emergency Maintenance : Hosting : Plesk2-Duck

Server Plesk2-Duck has failed with a corrupt RAID array.

DataCentre technicians are onsite investigating if it can be repaired and returned to service. If the current server is unrecoverable a new plesk server will be deployed and all sites/emails etc restored from backups.

ETA for server availability currently: 12:30 14/September/2011

Update

We apologise for the lack of updates this afternoon.

Duck/Plesk2 has physically been repaired however some configuration files and other data was corrupted during the initial equipment failure. These are having to be found and corrected individually to avoid user data loss.

At present there is no firm ETA on exactly when this will be completed, but we do expect it to be completed tonight.

Update

The corrupted configuration files and databases have been restored from the backups (2011/09/12 as the backup from 2011/09/13 had not completed).

If you continue to have problems with your account please raise a support ticket at http://support.othellotech.net/

Sep 03

Planned Maintenance : Hosting

Starts: 03/September/2011 16:00 GMT
Ends: 24/September/2011 22:00 GMT

Details:
Essential server and systems maintenance on OthelloHosts.net shared hosting servers will be taking place throughout the weekend.

  • Upgrade to latest version of Control Panel
  • Upgrade to latest o/s version
  • Upgrade to latest script installer
  • Patch/fix Apache and PHP for latest exploits
  • Add new RAM to upgrade all to min 8Gb

Affects:
All shared hosted clients on systems at IPHouse…

  • Linux Hosting (Plesk, CPanel, DirectAdmin, CP+) Clients
  • Windows Hosting (Plesk, HELM, DotNetPanel, WSP) Clients
  • Video Streaming Services Clients
  • Radio Streaming Services Clients
  • Magento Ecommerce Hosting Clients
  • WordPress Enterprise Hosting Clients
  • Joomla SaaS Hosting Clients
  • HSphere DCH Cluster Clients
  • Acquired MidnightSoftware Clients

There should only be minimal disruption to any websites, for the periods the machine is powered down (for hardware updates) or restarting services.

Should you experience any issues _before_ the server is updated, please open a ticket for support with all the details.

Should you experience any issues _after_ the server is updated, please open a ticket for support with all the details.

The forum will be updated as each machine is started and completed, please do not raise any “my site is down” support tickets whilst the server is under maintenance as
[a] we know and
[b] it’ll be back in 15 minutes

Update

System: PleskWindows1 updates started.
System: PleskWindows1 updates completed.

Update

System: MailFail1 updates started.
System: MailFail1 updates completed.

System: MailFail2 updates started.
System: MailFail2 updates completed.

Update

System: Plesk7 updates started.
System: Plesk7 updates completed.

System: Plesk9 updates started.
System: Plesk9 updates completed.

System: PleskLinuxReseller9 updates started.
System: PleskLinuxReseller9 updates completed.

Update

System: Plesk8 updates started.
System: Plesk8 updates completed.

System: PleskLinuxReseller8 updates started.
System: PleskLinuxReseller8 updates completed.

Update

System: CPanel1 updates started.
System: CPanel1 updates completed.

System: CPanel3 failed.
System: CPanel3 replaced with new machine.

System: CPanel5 updates started.
System: CPanel5 updates completed.

System: CPanel6 updates started.
System: CPanel6 updates completed.

System: CPanel7 updates started.
System: CPanel7 updates completed.

System: CPanel2 updates postponed.
System: CPanel4 updates postponed.

Update

System: CPanel2 updates started.
System: CPanel2 updates completed.

System: CPanel4 updates started.
System: CPanel4 updates completed.

System: Ensim1 updates started.
System: Ensim1 updates completed.

System: DNP1 updates postponed.

Aug 23

Emergency Maintenance : Hosting : cpanel2

Customers with services provided by CPanel2 will be experiencing downtime this afternoon. This server has suffered an unknown failure without warning.

At this point we do not have an exact length of the outage, however an engineer in en-route to diagnose the fault.

We will keep customers up to date via this forum posting.

Update

This server is back in service, we apologise for the inconvenience. The fault was a damaged power supply which has been replaced.

Jun 20

Emergency Maintenance : Hosting : Stream2 – NIC Failure

We are aware of the issue regarding stream2.othellohosts.net/CheapFFMPEG services.

At this stage it appears to be a NIC failure, an engineer in en-route to diagnose and correct. We will post more information once we have it.

Update

We apologise for the extended outage, the failure has been repaired.

Apr 08

Planned Maintenance : Midnight Software : Migration Status

We apologise to all former Midnight Software customers for the inconvenience, this hastily arranged migration was not our original plan and was a result of the business situation left by Midnight with their suppliers.

The current migration status is: –
HyperV VPS – being worked on
OpenVZ – being worked on
Cpanel Hosting – sites all moved, ip addresses being updated now ETA 11am
HELM Hosting – servers being repaired ETA 1pm
Dedicated Servers – please open a ticket at http://support.othellotech.net

Update

A OpenVZ Linux-based VPSs have now been restored to operational status. Owners have been sent the new details to their billing e-mail address. You can also access these in the My E-mails section of the Client Portal at http://my.othellotech.net.

Update

All cPanel-based virtual hosting is now available, again we apologise for the inconvenience.

Update

Windows VPS are now back up, details of the new IP addresses can be found via the client portal at http://my.othellotech.net.

In some cases we will need authentication details to update the IP address, if you do not have access to your emails please check My Emails via the client portal.

Update

All servers are now returned to operational status.

There are still some issues with Windows-HELM based hosting accounts, and the Technical Team are continuing to work on bringing the remaining sites back online.

Update

Mail service on the Helm server is now restored – should you have any further problems please create a support ticket

Mar 05

Emergency Maintenance : Hosting : Toby-DH/PleskWin3

Server PleskWin3 (toby) is under maintenance following a failed PSU @ appx 8.45am

Power supply has been replaced, but disks are not being recognised. It may be necessary to replace the entire machine and restore data from last good backup – more details will be added to the thread as they become available.

Update

The drives have been replaced and RAID array rebuilt – we are now performing a bare metal restore.

Update

The restore is currently at 21%.

Update

The restore is currently at 38%.

Update

The restore is currently at 56%.

Update

The restore has now completed – we are performing some final checks before bringing the machine back up.

Update

The server Toby-DH/PleskWin3 is now back online.

If you are still experiencing issues please raise a support ticket at http://support.othellotech.net/

Update

We have received scattered reports of additional problems with this server. At this point we have not been able to verify the cause, more updates will be provided as they are available.

Update

Remote diagnostics have not assisted, an engineer has been dispatched for physical diagnostics.

Update

There was more degredation to the drive arrays for this server. It is currently back in service, however we expect to be performing a hardware migration very soon.

More details of the migration will be provided as they happen.

Update

The server has failed again and destroyed the data on the RAID.

We have replaced the server and the restore is in progress.

Update

The restore is currently at 14% – we will post further updates as they are available.

Update

The restore is currently at 42%.

Update

The restore is currently at 59%.

Update

The restore is currently at 67%.

Update

The restore is currently at 74%.

Update

The restore is currently at 89%.

Update

The restore is completed and the disk verify is at 56%.

Update

The disk verify phase has now completed. Engineers have now turned their attentions to repairing the software differences due to the replaced hardware.

Update

Despite our best efforts, recovery using the CDP (Constant Data Protection) Block Level Backup has failed. This is most likely due to the incremental nature of the original disk failures.

The server is being re-provisioned onto another set of disks and we will attempt to re-provision and recover whatever files are possible from the block restored disks. Unfortunately this means that some customer data may be unrecoverable.

Again, we will provide updates as work progresses.

Feb 19

Emergency Maintenance : Hosting: cPanel4

All,

We apologise for the outage in service, this host has developed some filesystem corruption and we were forced to take it out of service to correct it.

At this stage engineers are actively working on the issue but there is no estimate available as to how long it will take to complete.

We will post additional updates as they become available.

Update

The first pass repair has completed, however due to the type of corruption detected additional passes may be required.