Resolved- Olga "has" had a bad hard drive

[andreadoria]

Please get it back soon. I'm missing a lot of traffic already.

Seriously. 3am was one thing, but people will start getting home from work soon, and error pages are not good for business. Not in the slightest.

[cornnfedd]

ok now im annoyed, im at work, cannot submit a ticket although i did last night, AND MY WEBSITE IS STILL NOT WORKING.

seriously i was patient before but now its getting stupid.

MATT AN UPDATE?

[Matt]

The problem is that the firewall is seeing a spike in pop3 connections and does not like them and when that happens it shuts down everything instead of just the people popping to frequently.

Without the firewall on, the load goes through the roof. We are still looking for a solution and are trying to keep thiings up as much as possible but it is a fine line between staying up and overloading the machine and being able to maintain our terminal sessions while we try to diagnose the problem.

[Matt]

Okay this thing seems to be holding stable now since about right after my last post.

The firewall and DNS departments seem to be doing what they are supposed to now.

The load is still very high and this can lead to some performance hits, basically meaning it will be running a bit slower than normal while the RAID array is rebuilding itself.

We ask you guys to play nice for the next 24 hours and if you can hold off on doing load intensive things like testing scripts, making 20 backups in a row, excessive cron jobs, etc please refrain from doing so, if possible.

The less strain on the CPU the faster the array will rebuild and that will bring you back to the normal transfer speeds.

Other things you can do is turn down the frequency of your email program's automated checks for new email, stay out of SSH if you have it enabled, and anything else that lessens connections to the server.

If things go south again we will keep you posted.

[AsdaLegend]

Nope, not yet stable. This has become extremely irritating. It's time for me to move on..

[cornnfedd]

yup not working for me either.

[Matt]

I am sorry to hear that AsdaLegend. All we can do at this point is offer refunds for those that request them, upgrades to other shared hosting plans, and apologies to the rest of you. I know it is frustrating no one is more frustrated than me and I hope soon I will be able to find a few minutes to get some rest...

Our goal here was to preserve your data from what could have a catastrophic drive failure. I think we succeeded in that department. Now we are stuck dealing with the hardware limitations of the raid array and waiting for it trying to rebuild itself.

If any of you would like to be migrated to another type of hosting plan with a higher uptime SLA please open a ticket and we can move your account to any type of hosting account you like, typically anything on the site other than budget or advanced hosting will put you on a package with a higher uptime SLA and off of the Olga server. We can also credit you a free month of hosting should you prefer an upgrade option.

[klikmouse]

Hello all. First post here.

Frustrating, yes. But "thank you" to everyone who worked hard to get us back up and running.

[Matt]

Thanks and welcome to the forums.

Just so there are no false promises here, I do not think we are out of the woods entirely yet. I am seeing some disk errors no on the other drive in the array and as you may have noticed, the server is not as peppy as usual. We may need to put yet another new drive in here after this array has rebuilt itself. Obviously hoping for something much smoother than the first time around....

I will be sure to keep you posted with any news as it unfolds.

[Matt]

Okay good news and bad news.

The good news is the original machine has been outfitted with new hardware and is ready to take these drives and raid array back.

The bad news of course is we have to take thie servers offline to do it. We expect about 30 minutes for this changeout. We are going to do this tonight at 2 am.

This should clear up some of the load issues, and get us heading on the right track again. The machine running now is much less powerful than the orignal machine and it was the closest thing we had on hand that could be used as an emergency machine to keep your sites up why we took the original down for diagnosis and hardware replacement.

The way this has been going so far I would expect 1 hour of downtime where there are no services at all. If things go well it should be only 30 mins or so.

We will update you as services return to normal.