| Posted by MartHUK, 09-23-2006, 05:03 AM | |
It appears I posted this at the same time as AlphaRed posted an update - see http://www.webhostingtalk.com/showthread.php?t=549369
Last edited by MartHUK; 09-23-2006 at 05:08 AM.
|
| Posted by MSV_AR, 09-23-2006, 05:03 AM | |
Alpha Red Inc is currently experiencing a primary transfer switch failure (as diagnosed at this time), which is causing utility feed to our datacenter to have ceased.
We are aware of this issue, and are currently working on resolving it as soon as possible.
Please be patient, as we do realize that your uptime is important, and we will have this rectified as soon as humanly possible.
If you have any questions, you can reach me on my cell phone at: 713-962-5791
|
| Posted by MartHUK, 09-23-2006, 05:10 AM | |
Thanks for the update, much appreciated.
|
| Posted by Abobo, 09-23-2006, 05:12 AM | |
Thanks for the info. Any idea how long this will take to resolve? I mean is this going to be a few more minutes, or are we talking hours?
Please keep us informed. Thanks.
|
| Posted by MSV_AR, 09-23-2006, 05:32 AM | |
Hello,
I have an update on situation:
The transformer room for our building, where our transfer switch is located is flooded.
From what I've been able to gather, is that the building's fire suppression system has malfunctioned and is pumping water into the underground transformer room, and has caused our equipment, as well as our utility provider's equipment Centerpoint, to automatically shutdown to avoid damage.
The Houston Fire Department is at the scene now attempting to shut off water flow and pump the water out, so that Centerpoint can begin work on restoring power.
I will update further as I can.
Please be sure we are working on this as fast as possible - it is 4AM on a Saturday, and we are unflinching in making sure this is getting resolved ASAP.
Thanks
|
| Posted by dreaz, 09-23-2006, 05:33 AM | |
nevermind, thanks for keeping us updated
|
| Posted by Abobo, 09-23-2006, 05:48 AM | |
So, does this mean all the servers lost power? Does that mean even after this is resolved I'm going to have to wait for my servers to run fsck to make sure the HD's didn't die?
|
| Posted by MSV_AR, 09-23-2006, 05:55 AM | |
Yes, all servers have lost power, a situation like this was never even imagined.
Your server may or may not have to FSCK when it comes back online.
Our UPS system's have built-in power conditioners which should've kept any current spikes from reaching server's, I don't expect that many failed HDD's.
|
| Posted by Abobo, 09-23-2006, 06:11 AM | |
The damage sounds pretty huge. Do you have any idea yet about how much longer this will be?
|
| Posted by MSV_AR, 09-23-2006, 06:14 AM | |
I don't have an ETA.
I have learned that the entire city power grid that we are on is now down as well.
We are working on alternative means of bringing power backup - we arn't going to wait on the building, Centerpoint or the fire department at this point.
We are actively pursuing alternative power means, including re-wiring our feed away from our transfer switch direct to our generator, so that we can get power restored before the rest of the building/city grid.
Last edited by MSV_AR; 09-23-2006 at 06:17 AM.
|
| Posted by Abobo, 09-23-2006, 06:18 AM | |
Wow, you guys took out the city! 0_o
I hope this can be resolved quickly, thank you for all your hard work.
|
| Posted by RH Swaroop, 09-23-2006, 06:21 AM | |
man, that's bad.. thanks for the update and hope this gets resolved soon!
|
| Posted by MSV_AR, 09-23-2006, 06:29 AM | |
Not the entire city, just the grid we are on is out. Also, we wern't responsible for the issue, though we are directly affected.
We are all working on this as fast as possible guys - don't forget that this happened around 4AM CST on a Saturday, it is now 5.30AM CST on Sat. and we are trying to get all sorts of people to answer their phone.
I am having good luck getting the neccesary people on the ball though, so I am hopeful that we will be up shortly. (relatively speaking, of course)
Plans have already been set in motion, and our own electricians are on their way.
|
| Posted by MSV_AR, 09-23-2006, 07:15 AM | |
Guys, I am heading downtown to the datacenter - the entire city grid is still out, so I don't know what type of access I will have, I am bringing my laptop with VerizonWireless internet, so I should be online once I get there.
Please call me if you have any questions - though we are doing what we can to make sure we are back up ASAP.
I will take pics while I am there, there is flooding like 20 feet high.
|
| Posted by MattJB, 09-23-2006, 07:23 AM | |
I'd also like to mention that you can reach Tech Support directly at 713-589-3238. They are offsite but will be current with the situation and, once power is restored, they can escalate any server problems directly to the onsite staff. They are also available on ICQ. You may also reach me on ICQ or gtalk.
Alphatech01: 235-435-304
Alphatech02: 224-215-392
--
MattJB
Alpha Red, INC
ICQ: 133076358
Gtalk: alpha.red.admin
|
| Posted by BLazeD, 09-23-2006, 08:05 AM | |
Just wanted to say that as an OF customer you guys seem to really be on top of this one - good job on the updates!
|
| Posted by marmoset01, 09-23-2006, 08:23 AM | |
From what I see (in this thread) , this will take days to fix......
Last edited by marmoset01; 09-23-2006 at 08:36 AM.
|
| Posted by HackNo-Alex, 09-23-2006, 08:24 AM | |
Thanks for the update MSV_AR
|
| Posted by dreaz, 09-23-2006, 08:29 AM | |
Where did you hear that? I doubt this is the case, but if it is please someone from OF confirm it, I cant afford days of downtime on 6+ servers.
If not, is there any ETA or just a rough estimate?
|
| Posted by forumtalk, 09-23-2006, 08:30 AM | |
are u sure
|
| Posted by MattJB, 09-23-2006, 08:31 AM | |
I just received word from the workers onsite. The situation is very close to being resolved. At this point we're looking at 30 minutes to 1.5 hours at the maximum, from what I've been told. I'm going in now, the techs are still available at the numbers above.
|
| Posted by marmoset01, 09-23-2006, 08:33 AM | |
By "hear" I mean my speculation from the posts Ive seen in this thread.
Flooded equipment......water 20 feet high..... that seems pretty bad.
|
| Posted by MattJB, 09-23-2006, 08:34 AM | |
This is absolutely not true.
|
| Posted by marmoset01, 09-23-2006, 08:34 AM | |
I hope you are right.
|
| Posted by MattJB, 09-23-2006, 08:38 AM | |
Yes the damage to the buildling equipment is extensive and will take several more hours to fix. However we're not relying on building power, the problem is our connection to the generator which will be fixed very soon. Once we are on the generator everything will be back to normal.
|
| Posted by frus_trated, 09-23-2006, 09:28 AM | |
Update? "very soon" was posted almost an hour ago.
|
| Posted by MartHUK, 09-23-2006, 09:37 AM | |
That was posted an hour ago so they've got another 30 minutes yet. Don't forget this is just what they've been told - don't be surprised if it takes longer.
Flooded equipment sounds like one of the worst scenarios. I can't imagine what these pics will look like. (Although they should help me explain the situtation to other people...)
|
| Posted by grapple, 09-23-2006, 09:38 AM | |
Hello, is anyone else having a problem with Orange Fiber. They have been down for over 6 hours now, their website is not responding, IM or ICQ is off and of course no one answers their phone. Did they run away?
Thanks;
|
| Posted by F430, 09-23-2006, 09:45 AM | |
same here.
My servers were down.
Can't access their support site too.
|
| Posted by MSV_AR, 09-23-2006, 09:46 AM | |
Servers should start coming back up within the next 30-40minutes. Our electricians had to sever live wires, and re-run them to our generator to get everything back up.
Once I get a chance to post the pictures (when I get home tonight) you will see all the water and the extent of the damage.
I arrived to late to catch the firetrucks for pictures though.
Centerpoint (the eletric utility) had to use a BOAT to get in the water and make their way to main power room, where they have only at this time finally disabled power, and the water is non-electrified anymore.
Either way, we are not relying on the status of the building for us to be operational, I mentioned that very early on - we analyzed the situation, and this is the quickest way to get every one back online.
THe building itself will probably be offline without power for another 24 hours+
There is a lot of water, that they have not been able to pump out yet.
If updates in this thread are slow now, I apologize, everyone is now at the datacenter working and getting ready for when the servers come back up.
|
| Posted by music, 09-23-2006, 09:50 AM | |
http://webhostingtalk.com/showthread.php?t=549369
looks like they are working on it.
|
| Posted by HackNo-Alex, 09-23-2006, 09:53 AM | |
Within that post, there are various numbers that you can call to get the status of things.
|
| Posted by HackNo-Alex, 09-23-2006, 09:55 AM | |
Thanks for the update MSV_AR
|
| Posted by LoveM, 09-23-2006, 10:08 AM | |
Very Long Time.I think Alphared indemnify to customer for this cause
|
| Posted by funkywizard, 09-23-2006, 10:14 AM | |
One of my servers is back up, but my other one on the same subnet isnt... weird.
|
| Posted by forumtalk, 09-23-2006, 10:21 AM | |
backup now
|
| Posted by nt99, 09-23-2006, 10:21 AM | |
Both of my servers appear to be back online, but none of the sites on either one are working. And the Cpanel for one is saying Unactivated License File.
|
| Posted by Bukowski, 09-23-2006, 10:27 AM | |
Ok .. my website back up
|
| Posted by frus_trated, 09-23-2006, 10:30 AM | |
"cannot find server"
|
| Posted by HackNo-Alex, 09-23-2006, 10:31 AM | |
My rack is up fine as far as I can tell.
|
| Posted by joeventura, 09-23-2006, 10:31 AM | |
Relax and give them some time, I know time is money but they are in constant contact with us, so lets give it a few more hours. I have a server myself that is still down, you should all expect another 24 hours before each and every problem is addressed is my guess.
|
| Posted by Alpha Red Tech, 09-23-2006, 10:32 AM | |
The power is backup now. We are currently waiting for dns to put back online. Other servers are back online now. We are making sure every service run on the server. Thank you.
|
| Posted by kkikki, 09-23-2006, 10:34 AM | |
their name servers are still not up yet.
cause all of our boxes with own DNS setup are fully back up 30 min ago.
|
| Posted by RH Swaroop, 09-23-2006, 10:36 AM | |
Thankfully our servers are back. I appreciate the effort OF put into this. Constant updates was just keeping us much more relaxed.
|
| Posted by SPB-Jeff, 09-23-2006, 10:39 AM | |
Alphared back online. Power feeders replaced.
|
| Posted by MSV_AR, 09-23-2006, 10:40 AM | |
Working on the DNS servers, was first priority, the servers are rebuilding tables
|
| Posted by SPB-Jeff, 09-23-2006, 10:41 AM | |
Dear Memebers,
This email to inform you. All Alphared new power feeders placed in and activated.
Power back up now and all servers will be turned back on one by one as i write this post.
Jeff
|
| Posted by MSV_AR, 09-23-2006, 10:41 AM | |
Our Cisco phones are acting up, if we do not answer, PLEASE do not get discouraged ... we know your server is down, we monitor EVERY server in the datacenter and we will get it back up ASAP.
|
| Posted by Airkat, 09-23-2006, 10:43 AM | |
I've got DNS running on ANOTHER seperate server and I still can't reach my box. Through IP either...
|
| Posted by kkikki, 09-23-2006, 10:43 AM | |
Nobody likes outage, but they did the best they possibly can to get servers back on line in the shortest time, and we've been well-informed throughout. I have to say: Good work, guys.
|
| Posted by joeventura, 09-23-2006, 10:46 AM | |
DNS and my server just came back up
|
| Posted by biggerboy, 09-23-2006, 10:47 AM | |
Sounds pretty bad, I would like to see the pictures too.
|
| Posted by joeventura, 09-23-2006, 10:49 AM | |
Just FYI, 6 hours of downtime for a problem as horrific as has been described is pretty darn impressive. Well done guys!!
|
| Posted by Destray, 09-23-2006, 10:57 AM | |
How many peoples servers are still down?
I cant get a hold of anyone at OF so I'll just say it here - both my servers are still down - 1674 and 1675. Just wondering whats up...
|
| Posted by mainchoicemedia, 09-23-2006, 10:59 AM | |
Mine is still down too.
I contacted them via ICQ . Still waiting for a reply.
|
| Posted by dragonhawk, 09-23-2006, 10:59 AM | |
My server's still down
And I only got the server a bit over a week ago. That's a high percentage of downtime for how long I've had the server :p
|
| Posted by softtech, 09-23-2006, 11:00 AM | |
Well not all servers power back on after a power failure. Some might need to be switched back on manually.
I have a rack colo at AR and only my supermicro based motherboards auto pwoered on. The older servers I had to open a ticket to get open. I just told them to check my rack and ensure that all servers were powered on and one by one the others came back. But be patient with them they are probably flooded with tocket requests right now.
|
| Posted by MSV_AR, 09-23-2006, 11:02 AM | |
Got it - will try and get to them ASAP.
|
| Posted by ewhost, 09-23-2006, 11:08 AM | |
Judging from http://webhostingtalk.com/showthread.php?t=549369
You guys are handling this situation quite well... providing cell #s, telling updates like it is, etc.
I'm impressed
|
| Posted by Coolraul, 09-23-2006, 11:22 AM | |
Well done OF.
Bad stuff happens but you have been handling it well. People looking at this as extended downtime or something to complain about should really rethink their view.
|
| Posted by Eleven2 Hosting, 09-23-2006, 11:25 AM | |
I am very pleased in the way OF has handled this. I just have a VPS though a reseller with OF and got alerts early this morning and by the time I checked back here all is up and are pleased with the constant updates OF/AR has provided!
Thanks Guys!!! You Rock! I owe you a cold one!
|
| Posted by Dennis Nugent, 09-23-2006, 11:30 AM | |
Questions to add to the list of questions to ask potential data center / colo providers:
1. Is your meet-me room located in the basement?
2. Is your power equipment located in the basement?
3. Do you have an ON-SITE generator for power back-up?
if so, how often do you completely test it?
We looked at a very large data center two years ago, that had their meet-me room in the basement, near the street corner, with no drains in the floor.
|
| Posted by Abobo, 09-23-2006, 11:39 AM | |
My servers are still down.
server 1153 and my shared hosting account on virtual2.alphared.com
|
| Posted by MSV_AR, 09-23-2006, 11:43 AM | |
1153 is being rebooted now
|
| Posted by nt99, 09-23-2006, 11:43 AM | |
I'm still down.
server 1531
edit: just called in. Mike said it'd be looked at right now.
Last edited by nt99; 09-23-2006 at 11:47 AM.
|
| Posted by SPB-Jeff, 09-23-2006, 11:46 AM | |
Engineering bringing servers up one by one.
If your servers still down during next 30min or so.
please contact the online techs or drop a call and it will be taken cared by.
|
| Posted by Destray, 09-23-2006, 11:47 AM | |
I'm still down - 1674 + 1675.
edit: i also called, its being looked into.
|
| Posted by dragonhawk, 09-23-2006, 11:57 AM | |
Mine's still down too... Server 1961...
|
| Posted by Airkat, 09-23-2006, 11:57 AM | |
Seems everytime there's a problem with a host we get these kinds of posts in the threads.
1. How is your post helping the current situation
2. I understand it's a sneaky way of advertising, but no matter how annoyed I am with OF I'd never go with a company that does stuff like this for attention (have turned them down in the past and will continue to do so).
3. Step aside and let OF handle this thread so that everyone is happy
|
| Posted by funlol, 09-23-2006, 11:58 AM | |
Wow, I'm just having bad luck this week
my ev1 server was down for 3 days... and I finally managed to get it back up last night. I sent an email to all my members saying my site was back and to start sending my pages to their friends again. And now my media server goes down, making my site useless again.
OF, what's the status of 67.15.190.14?
|
| Posted by MSV_AR, 09-23-2006, 12:04 PM | |
That isn't our IP, update me
|
| Posted by F430, 09-23-2006, 12:05 PM | |
Sites for server 1682 down...please check
|
| Posted by MSV_AR, 09-23-2006, 12:07 PM | |
your server is backup and pinging, please login and verify
|
| Posted by nt99, 09-23-2006, 12:07 PM | |
Can anyone update me on 1531? I called a while ago and Mike said it'd be looked at right away, so the fact that it's nto back up tells me there may be a more serious problem.
ANyone know?
|
| Posted by Destray, 09-23-2006, 12:15 PM | |
I called 30 minutes ago reguarding 1674 and 1675, both of which are still down. Any updates?
|
| Posted by MSV_AR, 09-23-2006, 12:20 PM | |
both are backup
|
| Posted by unknownstyle, 09-23-2006, 12:26 PM | |
Please look at server1685 . Thanks!
|
| Posted by Host Ultra, 09-23-2006, 12:27 PM | |
Your support site is still down and the tech on icq shows he is online but isnt responding.
My servers 1095 1096 1098 are still down
|
| Posted by Kim Yeah, 09-23-2006, 12:29 PM | |
Please check 1412...
It's still down
|
| Posted by joeventura, 09-23-2006, 12:29 PM | |
So since we are in the mood, what do all you OF/Alphared folks do for backups?
I am using DirectAdmins backup function and downloading a big tgz file every now and then
|
| Posted by funlol, 09-23-2006, 12:32 PM | |
my fault 64.72.125.137
|
| Posted by Abobo, 09-23-2006, 12:35 PM | |
Thank you OF staff, my servers are back online. I can relax again
I'm totally impressed with the OF's level of service. Both of my servers were back online within an hour of contacting them, and not only that, but they called me back to let me know when each one was brought back online. Simply amazing. Keep up the good work.
|
| Posted by MSV_AR, 09-23-2006, 12:36 PM | |
All 3 should be coming up now
|
| Posted by Host Ultra, 09-23-2006, 12:42 PM | |
Wow that was fast, they are all back up thanks.
|
| Posted by Destray, 09-23-2006, 12:46 PM | |
dont forget about me and nt99! Heeeeeellllp us!
|
| Posted by smolarek, 09-23-2006, 12:47 PM | |
Two of our servers are up but all our servers on our private switch are still down - any possibility those could be put back on ?
Servers still down are:
1930 1931 1932 1522
Thanks
|
| Posted by nt99, 09-23-2006, 12:49 PM | |
Maybe they sold our servers during this.....to a fireman.
|
| Posted by Destray, 09-23-2006, 12:50 PM | |
hope not! That fireman isnt gonna be very happy when he see's whats on my hard drive! Well, depends what he's into
|
| Posted by lingates, 09-23-2006, 12:55 PM | |
Hey Mike, its Lin.
Still down for me, you know my server numbers.
Thanks!
|
| Posted by Dennis Nugent, 09-23-2006, 12:55 PM | |
Posted by Airkat:
Seems everytime there's a problem with a host we get these kinds of posts in the threads.
1. How is your post helping the current situation
2. I understand it's a sneaky way of advertising, but no matter how annoyed I am with OF I'd never go with a company that does stuff like this for attention (have turned them down in the past and will continue to do so).
3. Step aside and let OF handle this thread so that everyone is happy
First off, I did not post that in this thread -- I posted it in the dedicated servers section. You can confirm that with the moderators.
Secondly, I am not advertising, I am trying to help the community. I have been in this industry for ten years now, and am confident that if you review the majority of my threads you will see that the intent is to educate.
Third - apologies to OF for the moderators putting my post into this thread
Finally -- how does it help the current situation? The current situation could have been prevented.
|
| Posted by MSV_AR, 09-23-2006, 12:59 PM | |
1531 coming back up
|
| Posted by supafly, 09-23-2006, 01:04 PM | |
Server1956 still down.
|
| Posted by JoHnY77, 09-23-2006, 01:14 PM | |
server 832 is still down
|
| Posted by funlol, 09-23-2006, 01:16 PM | |
What's the deal with 64.72.125.137 ? I still haven't heard anything
|
| Posted by Destray, 09-23-2006, 01:20 PM | |
Any updates on 1674 and 1675? I keep calling and I keep getting told the '30 more minutes' deal..
|
| Posted by lingates, 09-23-2006, 01:37 PM | |
server1816, 1403 still down
|
| Posted by Destray, 09-23-2006, 01:51 PM | |
You guys arent answering your phones...
I was told a tech was looking at my servers 2 hours ago..
Whats the deal?
|
| Posted by softtech, 09-23-2006, 02:10 PM | |
welcome to the internet. I think ar/of is doing a fine job. sure it would be nice if it didnt happen but well sometimes things happen dispite anyones best efforts.. Hopfully in the future a situation like this wont happen.
|
| Posted by joeventura, 09-23-2006, 02:12 PM | |
Right and for those who think that this or anything else can be prevented, you give me a data center design and I will give you a scenario that will cause unexpected downtime. No one except billion dollar companies like VISA and Mastercard can afford what is called TRUE redundancy.
|
| Posted by Kim Yeah, 09-23-2006, 02:22 PM | |
**** happens, personally I like to prevent downtime myself, so I have a backup server which I'm uploading to and waiting for the DNS to change over to.
I'm sure OF are doing what they can
|
| Posted by dreaz, 09-23-2006, 02:23 PM | |
1577 is still down for me, all my other servers are up. I talked to a tech on ICQ, he said he'd check it but not he doesn't seem to be there anymore and the server's still down. Could you please have someone look at it?
|
| Posted by Basura, 09-23-2006, 02:35 PM | |
Hi, could someone take a look at 1889?
|
| Posted by proust, 09-23-2006, 02:53 PM | |
Yeah please check out all these servers after you check out server 575
Coffee anybody?
|
| Posted by Kim Yeah, 09-23-2006, 02:56 PM | |
Nah, I'm on beer... This is too frustrating to drink coffee
|
| Posted by funlol, 09-23-2006, 03:18 PM | |
I'm ready to kill someone... 12 hours of downtime
|
| Posted by merhan, 09-23-2006, 03:22 PM | |
Downtime hurts businesses
|
| Posted by Xam1, 09-23-2006, 03:28 PM | |
Pfff what a day....
Good news is 6 of my 7 servers are up...
But my most important server 1574 is still down. Can some one please look at that server and boot it.
Strange thing is... Server 1574 was 2hr ago online for just 5 minutes... then the power plug was pulled out again.
Please update us ..
Roger & Guru - The xadultmasters
|
| Posted by Kim Yeah, 09-23-2006, 03:36 PM | |
Yeah... an update would be good....
|
| Posted by Kim Yeah, 09-23-2006, 03:40 PM | |
Well, mine are comming up
|
| Posted by supafly, 09-23-2006, 03:43 PM | |
It's frustrating I can access server1956 via ssh, but the kernel as far as i've worked out is booting up in 'read only' mode as it's detected something wrong on the disk. This means cpanel, apache basically everything is not working.
|
| Posted by funlol, 09-23-2006, 03:47 PM | |
how do you know?
|
| Posted by Kim Yeah, 09-23-2006, 03:56 PM | |
http://www.****stars.net/
One of the sites I haven't moved to the backup server yet, but it comes up
|
| Posted by Kim Yeah, 09-23-2006, 03:59 PM | |
lol, damn censur
|
| Posted by smolarek, 09-23-2006, 04:05 PM | |
All our servers are now back up.
Considering all the damage - i think AR did extremly well to recover as many servers as they have so far and were were willing to be so open and communicative about the problem.
|
| Posted by MSV_AR, 09-23-2006, 04:27 PM | |
We've got about 32 servers left that are having issues causing them to be hard down.
There might be more that have services down, but the server is up.
|
| Posted by Global2, 09-23-2006, 04:59 PM | |
Now that this disaster is nearly over~ you should absolutely and without a doubt be ashamed of yourself!
Not only did my techs and myself stay clear of disrupting this thread - we lent a true helping hand to several of their customers and provided them with free emergency service to get them back up again on our network until their units were restored. AND NOT ONE COMMENT WAS ALLOWED TO BE MADE TO ANY OF THEIR CUSTOMERS ABOUT US TAKING THEIR BUSINESS I FORBID MY PEOPLE FROM DOING SUCH A THING!
We have also been in this biz about the same amount of time you have and shame shame on you for not being truly HELPFUL when disaster comes about.
Our company shipped water and food to all of our hurricane customers during Katrina and Rita no doubt you were sitting in your chair eating cookies watching cnn !
Next time get off your butt and help someone instead of making sales pitches and nasty comments!
Sure they are our competitor just like they are yours, but that doesnt mean that some day we might have a 10 quake in downtown LA and I can only hope when that happens that a company like OURS will be there to lend a hand to us and our customers as we have done for others in situations such as this one today!
Take a lesson in Nettiquite from this post please and try to be more helpful instead of harmful when your competitor is down on their luck.
|
| Posted by Energo, 09-23-2006, 05:07 PM | |
Can you please check my deda server - HOU0567
I have a bad feeling that I'm the last one with the server still 100% down
I asked the support via ICQ about 2hrs ago but still nothing... and both alphatech1&2 are offline now.
|
| Posted by proust, 09-23-2006, 05:25 PM | |
HOU0575 is also still down .. we're in the same rack
We've made it to the last 32!
|
| Posted by MSV_AR, 09-23-2006, 05:33 PM | |
will check on 567 and 575 now ..
|
| Posted by nuke, 09-23-2006, 05:35 PM | |
One FSCK later and HOU567 is back online. We appreciate your patience and please let us know if there is anymore we can do. I will check on 575 now.
Regards,
AR-Brandon
|
| Posted by dreaz, 09-23-2006, 05:42 PM | |
Again, could you please check 1577? It was the first server down 5 minutes before everything went down and it's not up yet. I have talked to 2 techs hours ago, can't mail support because alphared.com and orangefiber.com are down, I'm still waiting to hear what is going on, it's been 10+ hours now.
|
| Posted by supafly, 09-23-2006, 05:47 PM | |
Can you please check Server 1956.
|
| Posted by RyanD, 09-23-2006, 06:13 PM | |
MSV:
I can't hit orangefiber.com / alphared.com are the sites still offline?
|
| Posted by proust, 09-23-2006, 06:26 PM | |
Hi Brandon,
Any news on 575? Its still dead here
|
| Posted by Destray, 09-23-2006, 06:51 PM | |
Mike...Ive had over 24 hours of downtime in this week alone now....
Can I get any word on 1674 and 1675...?
edit: alright they seem to be back online
Last edited by Destray; 09-23-2006 at 06:56 PM.
|
| Posted by Mark L, 09-23-2006, 06:58 PM | |
orangefiber.com is down for me too, wiresix.
|
| Posted by MSV_AR, 09-23-2006, 07:07 PM | |
these have been up for a while please double check
edit: nevermind, see your edit
|
| Posted by boyla, 09-23-2006, 07:09 PM | |
Please bring Server1229 back online, it down for over 10 hours.
|
| Posted by MSV_AR, 09-23-2006, 07:11 PM | |
other servers are taking priority right now.
Our support ticket system is and has been up since the beginning:
https://rockstar.alphared.com/support/
|
| Posted by Basura, 09-23-2006, 07:12 PM | |
HOU1889 is still down, could someone take a peek?
Thanks
|
| Posted by dragonhawk, 09-23-2006, 08:32 PM | |
server 1961 still down... Been down for almost 16 hours now. I hope we do get some sort of compensation for this.
|
| Posted by chandro, 09-23-2006, 10:13 PM | |
Any idea when those pictures will be up?
|
| Posted by MSV_AR, 09-23-2006, 10:34 PM | |
in a little bit once i get settled
Last edited by MSV_AR; 09-23-2006 at 10:40 PM.
|
| Posted by proust, 09-24-2006, 07:51 AM | |
HOU0575 is up , however everything added after april 2006 is missing
|
| Posted by MSV_AR, 09-24-2006, 09:04 AM | |
I know you are talking with support on chat, proust - they're working on it
|
| Posted by MSV_AR, 09-24-2006, 09:09 AM | |
Just woke up... man was so tired, and body aching all over.
All server's are back up at this time, and hardware issues resolved / data restored / migrated, etc.
Some pics:
http://blog.alphared.com/floodpics/
|
| Posted by joeventura, 09-24-2006, 09:33 AM | |
Well done!!
|
| Posted by MartHUK, 09-24-2006, 10:34 AM | |
@ http://blog.alphared.com/floodpics/IMG_1739.jpg
|
| Posted by RyanD, 09-24-2006, 01:06 PM | |
why use a metal john boat in electrified water?
Doesn't look like they have any rubber insulation either... why aren't they toast?
|
| Posted by SSA-Kris, 09-24-2006, 02:11 PM | |
I think AR/OF did a fantastic job, considering the situation. Most of all, I appreciate the communication and regular updates. Most companies who experience this sort of unexpected disaster will release a single statement usually no more helpful than to confirm they are having a problem - and you won't hear from them again until it's been resolved, again with nothing more than a "we're back, go about your business" brush-off message.
Thanks for the excellent customer service here AR/OF!!
|
| Posted by proust, 09-24-2006, 03:11 PM | |
Any news when the backup systems will be accessible? Server 575 is missing all uploads from april25 onwards and I "wondering" if they are backed up at all
Last edited by TradeViceroy; 09-25-2006 at 08:47 PM.
|
| Posted by Basura, 09-24-2006, 10:20 PM | |
I second that one, inspite of considerable downtime. Taking into account the nature of the problem, the in general responsive nature of AR/OF during this ordeal, I am quite impressed.
|
| Posted by MSV_AR, 09-25-2006, 11:43 AM | |
I've updated http://blog.alphared.com with a summary
|
| Posted by FHDave, 09-25-2006, 11:49 AM | |
perhaps the paint gives an insulating layer and furthermore, anti-static paint?
|
| Posted by supafly, 09-25-2006, 12:55 PM | |
One of my servers booted up fine. The other booted up with lots of errors and is still ****ed up. For example virtually everything in cpanel reports [an error occurred while processing this directive]. And two sites have disappeared from webhostmanager.
|
| Posted by MSV_AR, 09-25-2006, 01:37 PM | |
What server # is this?
|
| Posted by supafly, 09-25-2006, 01:57 PM | |
Server 1956.
Tech Support Ticket:
Ticket ID: IPL-203250
Tech Support Response:
Dear Client,
The errors on cpanel should be fixed now as well as the services (imap, pop and eximstats).
Ticket ID: HTA-839374
--------------------------------------------------
imap, pop, eximstats have been fixed.
But ALL cpanel/webhostmanager issues detailed in ticket remain.
|
| Posted by MSV_AR, 09-25-2006, 02:21 PM | |
Should be fixed now
|
| Posted by supafly, 09-25-2006, 02:47 PM | |
Fantastic, just logged into cpanel for all domains and it's all fixed.
Just one little problem remains, in webhostmanager 'List Accounts' it's not listing two accounts, the readers***********.com site and viewers*****.net site.
|
| Posted by MSV_AR, 09-25-2006, 03:54 PM | |
fixed that too
|
| Posted by supafly, 09-25-2006, 04:41 PM | |
Great thanks
|
| Posted by tenten, 09-25-2006, 06:45 PM | |
MSV_AR
I really need your words, cannot pm you. This is chris, I had a colocation package, and need some words from you. I sent many emails to you, but it seemed you were busy with recent data center issue. Please let me know what should I do to with my sever...
|
| Posted by MSV_AR, 09-25-2006, 07:33 PM | |
Already emailed you back
|
| Posted by tenten, 09-25-2006, 07:52 PM | |
Thank you once again.
|
| Posted by softtech, 09-26-2006, 10:06 PM | |
Actually the water wasnt "electrified".
The power was off and even if it was on Iam sure it would have long ago tripped one of those really big fuses that they use in transformers. That water was perfectly safe so long as some idiot didnt try reconnecting the power at some transformer somewhere...and if somsone did that guy putting the fuse in would be in for a surprise of his life as the thing explodes in front of him. When those fuses go they really go and you dont want to be right next to them when they do. They often put those fuses in using a long pole. Well assuming downtown Houston power works like those in smaller cities like where I live.
|
| Posted by MSV_AR, 09-26-2006, 10:08 PM | |
It was "charged" with 120v actually
|
|
 Add to Favourites
 Print this Article
|
