The server became unresponsive and our NOC techs are looking into it. Updates will follow.
Printable View
The server became unresponsive and our NOC techs are looking into it. Updates will follow.
Here is an update from our NOC:
We will keep you posted as the events unfold.Quote:
This is a pretty major OS issueWe are trying to use a rescue CD to get the system back to a boot state.
The techs are not having much luck getting the OS to recover so we are going to work on it for another hour or so and if we still cannot get it to fire up we are going to have to install a fresh operating system and then restore the cPanel accounts from last nights backups.
The Operating System is not looking good, so we have already started the process of installing a new OS and backups will begin to be restored shortly after the OS and cPanel have been configured.
The OS is installed and cPanel is over 50% complete. We will begin configuration and restoration very shortly.
cPanel froze during the install procedure. We have restarted the process...
cPanel is installed. Apache and PHP are being compiled now, and the root settings for this machine are being configured. Accounts will be unpacked as soon as the webserver is done.
Apache is done. Sergey is unpacking the accounts now. I am updating settings for the server. We will update you when all accounts have been unpacked.
The Server and OS have been configured. The cpanel accounts are 25% complete and some of you have web service available again. When this is complete we will post again and ask for anyone who still has problems to open support tickets.
The cPanel accounts are now 50% complete. Please note that there is more than one page for this thread. This is the final post on the first page...
Seems like its back up but I suffered around 14 hours of rollback with my forum. Anyways, thanks for the effort.
The backups are run once per day so if you are missing data, it is from the point when your last backup was taken to the time when the server crashed. If you have local backups you should be able to import those to recover the data.
Ok forum was back up. All be it with about 12 hours lost. But guess what, OLGA is back down again!
ImageShack - Hosting :: scr53ebf1ake5.jpg
This is pretty poor - almost an entire day down. I also cannot see glowhost.com through my connection, having to use a proxy. Can i please have my account permanently moved to a different server? This is shocking. Almost a day down.
ok slowly coming back together. Only ssl issue now.
I'm sure it's just a matter of time before it crashes again though.
ImageShack - Hosting :: scr55777e6vf9.jpg
All accounts have been restored. Now we are settling the dedicated IP addresses and SSL certificates. If you still have any problems with your sites please open a trouble ticket.
All SSL certificates have been reset. There were some problems with email services and they are rectified now too.
If you still have any problems with the server/sites please let us know.
Both of my sites are still down. It looks like its giving me a php error.
Fatal error: phpSHIELD Loader - script was encoded for another version of zend engine
Please open a trouble ticket and include the site name/URL.
This is rediculous, yet again olga is not functioning correctly.
DNS is down resulting in my pages not loading correctly. I'm starting to get so annoyed.
ImageShack - Hosting :: scr1930288uo0.jpg
Please tell me, due to the continous downtime and poor functionality, are we going to be receiving credit for this month?
Don't you think it would be a good idea letting us know what other issues we're likely to face over the next few days? At the moment I think everything that could go wrong, has. Christmas or no Christmas, is this server so unstable that it needs continuous attention?
DNS on olga is up&running, service was restarted several times to recover some aftermaths.
Service or your server? I think you are confusing service with a problem on a piece of machinery which we are servicing.Quote:
Originally Posted by dazkeirle
As I explained to you earlier, there is no way to predict an operating system failure and unfortunately, when you are dealing with high-end hardware and the newest operating systems, things require a bit of tweeking to get them streamlined. Any outages are intermittent while we optimize the servers performance.
The reason we have such a good reputation is because this sort of thing rarely happens because we take preventative measures, but with an OS that explodes, there isn't much that can be done than to fix it when it happens. I am sorry that you do not appreciate our commitment to you over Christmas Eve and Christmas Day to get the issue resolved.
Your DNS thing is fixed. You may some restarts but that is quite different than an outage. Was you site affected when it showed down on netstatus or are you just watching the status monitor for red lights?
rubbish, could you not see my screenshot?
My site is currently down.
edit* site is back up.
So please can somebody just answer my damn question. How long am i supposed to expect the server to go up and down for? a day, two days?
Does olga require someone to give it a sharp bang every so often to keep it ticking over lol. I used to have a tv like that.
Excuse me, but can you look at the real live monitor and not your screenshot? they are currently all green.
http://glowhost.com/netstatus/
Even if you see a red light does not mean the server is down. if you see a wall of red lights then it is down. You may see some services blink from time to time during a restart or something like that.
What is your domain name or ticket number if your site is still down?
whenever I look its either green or red. Sometimes a mixture of the two.
If i was submitting a ticket every time something on the site was down I'd have to have my cpanel open permanently.
Like I said netstatus is not 100% accurate and just because some lights are red sometimes does not mean the service is down. you can verify the service using a telnet to the server IP to verify it for yourself, next time you see a potential issue. If you see a whole row of red lights then it is a pretty good indicator of a server-wide issue. But a single light can be a timeout or a restart (not reboot) of a service.
In answer to your other question, no, no one is beating it with a hammer. As a matter of fact, no one is even looking at the server. It was re-racked yesterday. Right now all the software and services are being configured and repaired remotely.
The machine was stabalized yesterday. What we are doing now is fine tuning the services like Apache and DNS and whatever else turns up to be throwing errors. Sometimes in order to fix an error, modifications need to be made to the Linux or application configuration and a service restart is needed for the new configuration to be loaded. I am sure you are famiiar with the process on a windows machine where you install something and the entire machine needs a reboot. With linux, typically you do not need to shut the whole thing down, but you need to restart the service.
hang on, if we keep editing past posts then i cant tell what I'm supposed to be replying to.
Of course im not watching the netstatus permanently. But i do have it open as when i'm browsing my forum every so often everything just goes tits up. Of course my first port of call is to check the netstatus where usually olga then has some reds.
My problem is not your commitment, its the server. It's supposed to be fixed but there are still issues. If you'd just give us some visibility of what we are to expect and for how long in the coming future (resets, downtime etc) then I wouldn't have to post here to find out whats going on.
I have to post because everything goes back up and is fine and then it hits the fan again. I never know if your doing something to reset it etc, or it the hardware is just dying again. If you updated the thread with issues as you found em I wouldn't feel the need to keep informing you.
Thanks for the visibilty, as i know its being worked on I wont keep posting. Just please let us know when you think it's stable and then if it goes down after that I'll let you know.
You do not need to inform us, we also have access to netstatus but more importantly we see it before netstatus does and are working to fix it. We do not have enough time to come in here and post every time we are going to be making a change otherwise this will never get done.
But for the record, ffmpeg and other streaming video software is about to be loaded and Fantastico is about to be re-installed. neither of these require service restarts and after that we will be dealing with errors in customer tickets which may or may not require service restarts but i have no way of telling until the customer tickets come in.
dazkeirle
I am sure that many of us users on olga who would beg you to stop berating and badgering the techs so that the server can be patched as soon as possible. Many of us feel that you are only delaying the issue, and if they inform you everytime a service has to be reset, it will just delay things. So I would suggest, as someone sharing this server with you, that you do not make it worse for everyone please.
OK sorry all. :(
Matt, while at the moment i do think you and your team are legends. I don't appreciate you publically posting my sites address in that screengrab. edit* screengrab was removed - thanks
p.s. I didnt realise forum support were the same guys as who actually perform maintenence on these servers, was told that was NOC. Didnt realise i was taking time away from them.
I have removed the screen shot. The NOC handled the hardware and OS reloads, we handle server configuration and performance when they are online.
Hi, I'm one of the people with a site on Olga... thanks for working through the holiday on this issue, guys, I appreciate it, the place I used to be with would not have been as attentive... I just wanted to know, as I was about to log into cpanel and got the 'certificate expired' warning, is it still safe to log in? Sorry, I'm not as up on the tech stuff, so I just wanted to be sure it was all good before logging on. Thanks!
I was wondering as well if PHP5 is working properly now? Because it seems like some of my scripts that use PHP5 are not working.
Yes, PHP% is working well. Please open a trouble ticket and mention your domain/URL.
id like to throw my hat in the ring and say i was pretty dissapointed with what has happened - my site had a php error due to the server crashing and was down for around 24 hours. Not sure if this would have been possible as it would depend on how many websites are actually on Olga but i would suggest next time staff take the time to check all sites are back up manually, if there are errors they couldnt probably fix then and there. To be honest im very annoyed that it took 24 hours.... :angry:
if i didnt happen to check my site quickly for something i would only assume it would still be showing a php error - this is just not suitable, I dont have time to check my site everyday just to make sure the server hasnt crashed!!!!
lastly i would have though it appropriate for an email to be sent to all users on Olga letting them know of the outages - it would have been such a simple bit of effective customer service.
Having said all that i am glad it seems to be working again, christmas would have to be the worst time for a server crash so well done and good effort in getting her up and running again.
lets hope it doesnt happen again. :eek:
edit: in awstats you cannot update or refresh it (there used to be a little update link at the top) can anyone help?
Thanks for the suggestions. As for the link in AW Stats, try it now.
Please open a trouble ticket and include your domain name. I have spot checked a few sites on that machine and they all have the link. If anyone else has issues please open a ticket. I am going to lock this thread now.