There is some packet loss affecting some servers at GNAX DC1 in Atlanta. we are looking into this now.
Printable View
There is some packet loss affecting some servers at GNAX DC1 in Atlanta. we are looking into this now.
More servers are back online, the last update we received is that they are still working on the problem. We don't have much information other than that but we do see more servers coming back online every minute. Please standby for updates.
We will post them as soon as we get any more information.
Twitter also available for updates if you prefer
So as the preliminary reports advise, a nearby lightning strike was to blame. This was enough to knock most everything offline. The servers that are still down are most likely undergoing automatic fsck.
We are going to check them each to make sure that they are truly waiting for fsck to complete or if there is any electrical damage which would prevent the remaining machines from coming back online automatically when fsck completes.
Hey Matt -
Do you have any additional info regarding anticipated online status for Missy?
Nothing as of yet, I just put in another ticket a few minutes regarding the remaining servers and it looks like they are plugging a crash cart into the ones you see remaining on netstatus.
Okay thanks! Just trying to keep my clients updated as they are all concerned.
We are sorry for the delays for the handful of you that are still waiting are experiencing. We definitely are making our point with GNAX about the issues and they assure us they are scrambling to get information to us and those of you who are still down, back and running.
The list of servers with problems is getting smaller, we will keep at this until they are all back up.
Given that this is a somewhat recently recurring problem, are things becoming slack there at the DC?
I just had a client switch to me last week and today we're down all day. It's frustrating but at least GH deals with it nicely, just saw a post on another forum where a host just buried themselves with what seems to be a flipant response to the problem. lol
http://www.webhostingtalk.com/showth...=932654&page=5
Has GNAX and NetDepot always been one of the same company?
Lightning strikes?
The servers that are not online all appear to have hardware issues related to the strike. We are still confirming this and making a plan of attack for the best way to get these units back online, there will be additional downtime for these devices are replacement parts are installed.
Matt -
I've forwarded the info to my clients, thanks so much for the update! I know the next question will be what the anticipated repair time will be...
Do you have any idea how many servers are requiring repair at the facility or are they giving an ETA at all?
It depends on what the issue is for the affected device, I don't have a list here yet so I can't anticipate things yet, but I will be posting here as soon as I do know so that you can notify your customers as soon as possible.
Missy and Penny have bad power supplies, they are being replaced now. I expect 30-60 minutes on those machines and then fsck may be required after they have power running again.
I didn't mean to be critical of Gnax. Of course it's understandable if the hardware is fried, that's way out of their control.
You guys, Glowhost, do a great job of response. I am always impressed by your service. I just have no reference or idea about them (GNAX). All I see is that for 5 years I haven't ever had this major of an interruption, so hence the question, that's all.
I hope that all our data is in-tact. That's my biggest worry and keeps me up at night, even though I have a physical backup drive. That doesn't do too much good if the server goes up in a puff of smoke...
By the way, Lynne has over 400 posts and I have around 150. We're both Master Glow Jedis. I think Lynne needs a new title when all this is over.
Thanks for the update, it helps to give my clients something about what happened. :)
Well in all fairness I think there could be more done to mitigate a strike like this. How it was able to get past all the fancy new electrical components they have put into this new datacenter I am surprised that the surge was powerful enough to make is so far in that it was able to fry multiple power supplies.
I am not too happy about it right now because in addition to the hardware that smoked, the UPS that are supposed to prevent any power loss and also add another layer of clean power to the machines before they hit the servers also failed (well, one of them did) and that too should have never happened. That caused nearly everything we had in there to have unclean shutdown, which in turn causes Lunx to run filesystem checks which can take hours to do.
In general I happy happy with GNAX over the years. We've had servers in most all of the neutral datacenters and they all have issues like this, some more than others. The reason we have been in GNAX so long is because they seem to have less of these issues than most of the others.
But they have some explaining to do about this electrical problem if you ask me.
I am with you about data and backups. Hopefully you guys all have your data in at least 3 places. If not, now would be a good time to consider it. Typically you wont loose your primary drive and backup at the same time even if they are on the same machine unless the machine itself physically catches fire (or similar physical meltdown) but anything is possible when we are talking about these types of issues.
Lynne needs 100 more posts to get the next title I believe :)
Er, scratch that, only 9 more to go for her :)
Thanks for the good customer service. Hard to come by.
I wasn't trying to get any level! lol I was just trying to give you some encouragement from a faithful gh customer!
I just read a long post from gnax about the outage, and they keep plugging their a/b servers but reiterrating that they have all this protection in place. I don't know enough about the technicalities but I do know GH will address what they need to in order to provide the level of service they normally provide.
I hope you get some answers Matt, and would definitely be interested in what the resolution is for future "lightening strikes".
Yeah you can count on that. Just look for "myusername" in that thread if you want to see me behind the scenes....
ohhhhh.... good stuff Matt.
8 more to go. :)
Go get'em Matt! :thumbsup2:
Any word on Missy status Matt?
Okay - looks like Missy is coming back!!!!!
e-mail is still struggling though.
Matt,
Any word as to the status of Arjuna? I've got quite a few more than concerned clients right now.
Thanks,
Hi,
Waiting for update from DC. I'll update the thread as soon as we get any info regarding Arjuna and other servers.
Arjuna is now back online. The others will follow shortly.
hopefully Penny will be back soon. We are assuming that in case of data loss all our files are safe on the backup drive? You guys are doing great but keeping us in the loop :thumbsup2:.
Looks like Aegir is at the end of the list. 21 hours down. :mad: If I lived closer. I'd be knocking on the DC door.
I wish we could do a better job of updating you faster, but thank you for the kind words. :)
The DC has been pretty quite with everyone tied up on the floor fixing various hardware issues.
As for the data, it should be safe if you have a backup drive. It really depends on what the techs find when they plug in the cart and see what all is failed. typically you wont loose your backup drive and primary drive at the same time.
Yes, I was talking about the netstatus but referencing being last to be green not last to be repaired.
As always Matt. You do an AWESOME job of keeping us informed. At the end of the day, that’s what people want and need. Honest answers, good or bad. I really appreciate it.