Nelly Server Unscheduled Maintenance

[GlowSteve]

Dear Valued Customers,

Emergency maintenance is required and has been scheduled for Nelly server. This machine's hard drive went read-only at 5:30PM by EST and we had to reboot this server to bring it back online. At this time Nelly is running a FSCK. We will notify you as soon as problem is completely resolved.

Please read these details carefully:

Server Name: Nelly
Service Date: Monday, 11 November, 2013
Service Window: 5 PM to 6 PM
Details: Nelly Server Unscheduled Maintenance
Expected Downtime: 25 minutes

We are currently working on this server and will have service restored as soon as possible. Please do not call or open support tickets regarding the downtime as floods of tickets at our helpdesk will only slow response times for other users. We will post updates as they become available in this forum thread.

Please post any questions that you may have in this forum thread so that common questions that may arise surrounding this issue can be answered in a central location instead of individual support tickets which will slow down response times for everyone involved.

To receive instant notifications via email about updates which are posted in this thread:
Please click "Thread Tools" at the top right of the first post in the forum thread, then select Subscribe. This will send out instant email notifications of any updates to this thread. Below is what it looks like if you are having trouble finding this option. (Click image for a larger version)Attachment 268
Note: You will need to be registered on the forums in order to receive these notifications.

We sincerely apologize for the unexpected inconvenience and are working as fast as we can to restore service.

Thank you for choosing GlowHost.

[GlowHost-James]

At this time, the server is still undergoing an FSCK.

We will continue to update as we progress.

We sincerely apologize for the outage and thank you for your patience and understanding.

[GlowHost-James]

We are still running through the FSCK at this time and will continue to update as we make progress and have new information available.

Thank you for your continued patience.

[GlowHost-James]

Unfortunately, the Auto-FSCK failed. We have initiated a manual FSCK which will attempt to fix the errors on the filesystem and check the integrity of the disks.

We currently do not have any ETA on this, but will continue to keep you informed with the latest developments as they occur.

Thank you for your patience and understanding, it is highly appreciated.

[GlowHost-James]

The FSCK did not completely fix the problems and some services failed to start.

We have rebooted the system to try again. We will keep you informed on the progress.

Thanks again for your continued patience.

[GlowHost-James]

We have been unable to fully resolve the issues on Nelly. It appears that the filesystem is badly corrupted.

We are now evaluating our options and making a decision based on the assessment of the server as to what to do next. We should reach that decision within the next half hour and updates will follow.

Thank you for your patience.

[GlowHost-James]

Unfortunately, we have been unable to recover the filesystem due to the integrity of the file structure being too corrupted. We have been left with no option but to reload the entire server OS, reconfigure and restore customer accounts from backups.

We have already begun this task now and further updates will follow.

We sincerely apologize for the inconvenience and thank you for your patience.

[GlowHost-James]

Just a quick update to let you know we are still working on reloading and configuring the OS and cPanel. Unfortunately, this does take several hours since we must configure everything from scratch, including all the additional extras GlowHost offers as standard.

We will keep you informed as we progress. Thanks for your patience.

[Jeff Nikkel]

This is completely unacceptable. Why have you not given a status update in the last 8 hours?!

[Alexander]

Currently we are restoring the accounts starting from p.