DC outage

[jmarcv]

From GH's point of view I'm small fish

But I have to admit, my enthusiasm has waned a bit. An understanding of the technical issues does not help alleviate the frustration factor. Better communication would be a help.

I dont think anyone is small fish in the eyes of Glow. If Glow cannot communicate, it is because they are down, and maybe for the same reasons you cannot warrant the expense of 2 dedicated servers with load balancing. Glow has cost issues also. And frankly, we have all been told from the DC that they have bulletproof failover in place. Well, maybe not today. But hey, 3am to 10am EST is better than 12 to 7 dont you think?

So hopefully your enthusiasm will wax. Amazon was down for how many days?

[papp]

Perhaps I have not made it clear enough that I am perfectly aware of the fact that GH is not to fault for this downtime.
Let me add that on the WHT board I was offered to contact the helpdesk to receive a coupon. I didn't even consider doing so, since I see no reason to penalize GH for something that is beyond their control.

I merely tried to express my frustration over the lack of alternative communication channels. I feel that this is a legitimate concern and judging from the responses here it seems that I am not alone with this opinion.
I think that this constitutes as constructional feedback that might benefit GH, wouldn't you agree?

And yes, I am glad about the time frame of the outage. Could've been worse.

[Raven]

I am a reseller that has been with GH since 2003. My main issue with the way the outage was handled is that I phoned into GH around 9:00 am CST requesting a Call Back as the message stated to do. Since there was no way to contact GH via email nor website it was the only avenue available to any of us. I was not expecting an instant reply, however I received no call back at all. As a reseller I had absolutely nothing to tell any of my clients as to what the nature of the problem was. With some, especially the eCommerce resellers and clients who depend on their sites for their income, this did not go over too well.

A suggestion for the future would be to change the voice message when there is anything major happening and GH cannot be reached. That way it's simply a matter of calling the hot line and receiving an up-to-date status on whatever the issue is. Now, I'm not suggesting this for anything but major issues that keep us from reaching the GH web site to obtain information. I also think that Planned Outages should be available via phone. In addition to updating a web site forum we could get the info via the toll free line.

Thanks.

[rlhanson]

In the meantime, "I'll have my emergency email get in touch with your emergency email..."

CharlesH

That's exactly what I put in place for my clients yesterday! lol

I took a proactive approach with my clients, called each one and let them know that we were aware of the outage and although we didn't have information as to when it would be resolved, I could assure them that it was being addressed and would be resolved at the earliest time possible.

I then called them back to let them know it had been resolved.

I have had no fall-out because I communicated with my clients...even though I didn't know what was going on, I did know that GH was living up to their reputation and that they were handling it as expediently as possible.

I like the recording idea for the toll-free number - it would also be easy enough to set-up a google calendar we could all bookmark...

I did think GNAX could have been a little more forthcoming on the status originally which probably would have helped with the anxiety everyone was feeling.

[trickyBizzness]

I hate to churn the pot, but there are some comments in this thread that I find humorous that I would like to point out:

Whithout a monitoring service I might have never learned of the outage. Had I received a timely mail notification I would have had the chance to reroute to another server through my registrar.

If you would not have noticed without a monitoring service, it seems to me like you could probably live with some downtime. I mean, if you need a monitor to tell you that your site down, is it really being used that much?

However, potential buyers that come to my portfolio site are considering spending 4 to 5 figure sums, and that\'s where downtime becomes an issue, because it immediately communicates a lack of professionalism.

If you have people wanting to spend 4 and 5 figures why on earth are you using anything else besides a dedicated server?

I doubt that my last sales contact would have materialized if the buyer had been unable to reach the site on repeated attempts.

Is this normally what you do when searching for a specific product on a specific website? I don\'t. if my bank is offline or PayPal is under maintenance, I always go back later. Maybe I am different.

I phoned into GH around 9:00 am CST requesting a Call Back as the message stated to do. ... I was not expecting an instant reply, however I received no call back at all.

So you would rather take the techs of the issue they are working on, which is getting the sites back online, or have them calling people and telling them there is a problem and it is being fixed?

I run several e-commerce sites and uptime is just as important to me as everyone else. In fact, it is my primary and only source of income. While I don\'t want to see repeats of an outage like this, I\'ve been around the block a few times and quite frankly, it happens to everyone. As a previous poster said, even the big boys like Amazon, who were down for days. Do you think they are going to call you back? That\'s a joke!

Has anyone ever tried to get in touch with google about anything? Good luck!

I think GH did what they could in light of a bad situation and posting on WHT is a good solution for those of us that know where to look for industry news like this. Their business is in GNAX and if GNAX s down, including the GH site, then the only alternative I see for GH is to host in a competitor\'s datacenter, and that is kind of silly. GH is clearly a player in the hosting industry, and they stand behind their product. I think it would send a mixed message if they were to move their primary sites to a secondary provider...but maybe that is just me.

[Matt]

Hello Everyone,

Thanks for your comments and suggestions. We are still looking into alternatives right now for the best way to deal with such a situation. There are some logistical issues that we have to consider for a secondary lines of communication. For example, our customer database is of course located at our primary hosting facility, and if that goes offline, so does our access to customer records. Sure, these are backed up elsewhere but there is the problem of backups sometimes being older than the days current records.

It is possible that we may take all or portions of the GlowHost site over to the secondary datacenter in Dallas, though at this stage I am still hesitant to put any devices in there because the rumor mill tells me that the GNAX DC2 in Dallas is still running a skeleton crew and we would prefer to have a full crew of techs available for a major issue like this versus a skeleton crew for many small issues.

I am still waiting for reports and general feedback from the web hosting community as to how things are going over at DC2 and do not want to be the trailblazer at this time.

There are some good points though in this thread. Perhaps we can start a facebook profile and all of you can be invited as our friends. Twitter is another possible option. Or perhaps a combo of the two.

The phone systems voicemail is another idea, though reprogramming the message is not as simple as one might think. We have been working on updating those as well.

To those of you who did not receive a call back we do apologize but the high volumes made it simply impossible to return everyone's call. After the sites came back online I instructed our customer care department to stop making calls and to redirect efforts to handling the massive influx of tickets at the helpdesk that came in (and are still coming in) regarding the downtime from yesterday morning.

We thank you for your understanding in the matter.

[Raven]

I hate to churn the pot, but there are some comments in this thread that I find humorous that I would like to point out:

So you would rather take the techs of the issue they are working on, which is getting the sites back online, or have them calling people and telling them there is a problem and it is being fixed?

I never said anything like that. First of all the tech's hands were tied from what I understand. They were in a complete wait state and could do nothing but call for statuses from the DC. So in this case I believe a call back was in order.

[omarfilip]

Perhaps we can start a facebook profile and all of you can be invited as our friends. Twitter is another possible option. Or perhaps a combo of the two.

I vote for this - simple and effective.

[omarfilip]

With today's slowdown, I searched Twitter for GlowHost, but only found a few GH customers mentioning their sites were down. Seems like a good time to fire up a Twitter account for GlowHost. What do you say, Matt?

[Matt]

Its already there. @glowhost though we didnt have time to write much there as we were busy fielding other issues. For extended outages twitter is where to find the info.