Network problems at GNAX datacenter

[bdominick]

Gnax have released an official explanation for this, im sure glowhost will update shortly.

Hey Andy, where do you find notices from GNAX? I'd like to see the long version of this explanation.

From the short version, it sounds like GNAX has been deceiving its customers, which is very disconcerting. If you have 2 backups and they both fail, you actually had zero backups. You can't promote backup systems that aren't working. And how did the two grid connections both fail in the first place? Probably because there was only one...

[jmarcv]

You can't promote backup systems that aren't working. And how did the two grid connections both fail in the first place? Probably because there was only one...

GNAX has a forum.

I visited the datacenter, and it is pretty damn impressive! The guys I met are all real people.
However, if you read the long version you will see the long list of comedy of errors.

I can relate to their issue, it was a great embarrasment for them. When I visited they were so proud that they had it all figured out.

I have been working on a failover system for a client with 17 servers. I can relate. The only way to test it is to flip the switch, which is hard to do when you are live. So you wait and pray. Then judgement day comes, and you find all the loopholes. And then you patch them and lick your wounds.

However, when both battery units failed they fired up the backup generator and much of the power was back online.
But .... when a computer goes down unexpectedly, bad things can happen to the file system as I said, delaying the boot. When it is bad enough, your OS may get hosed.
When the power fluctiuates on and off from load spikes, routers can lose configuration settings. Suddenly you have power but data is not getting routed.

I am a moderator and a glowhost client. I've done my share of ragging on GNAX. But I think we will see a much more stable DC after this. They had every employee called in to deal with this. From my experience with hardware, this was probably pretty ugly for them.