| Posted by Angelo, 06-29-2009, 04:28 PM | |
seems like rackspace.com down.
any others who can access their servers?
|
| Posted by Scott.Mc, 06-29-2009, 04:29 PM | |
Yeap we have multiple systems down too, started exactly 15 minutes ago.
|
| Posted by Steven, 06-29-2009, 04:29 PM | |
i have 2 clients whom are down.
|
| Posted by Scott.Mc, 06-29-2009, 04:34 PM | |
Only appears to be servers in the DFW datacenter which just happens to be where the vast majority of ours and most peoples servers are.
|
| Posted by sieb, 06-29-2009, 04:37 PM | |
Yep, all of our CloudSites are offline.
|
| Posted by BizMarquee, 06-29-2009, 04:37 PM | |
Anyone know what happened? Their phones are down as well!
|
| Posted by pedrojose, 06-29-2009, 04:43 PM | |
yes, rackspace is also down from my end (portugal)
Does the email exchange is also down?
Does anyone knows what is the url of the rackspace network status / noc site?
|
| Posted by Scott.Mc, 06-29-2009, 04:44 PM | |
They updated twitter with, "We are having an issue that is affecting part of our DFW data center. No details yet. Will update as we get more information."
|
| Posted by DigitallyM, 06-29-2009, 04:49 PM | |
Pretty poor they don't have an alternative status page that shows up on a google search for rackspace status.
I only noticed because I use Rackspace's email service, which I didn't expect to go down, ever. Stupid of me to be that optimistic I guess.
|
| Posted by blahrus, 06-29-2009, 04:52 PM | |
holy crap, sieb is on WHT
|
| Posted by r3volt, 06-29-2009, 04:55 PM | |
Word from inside is that it is another power issue, of which they've had 2-3 in the past month.
|
| Posted by jyarnold, 06-29-2009, 04:55 PM | |
they do have a status page, status.rackspacecloud.com
|
| Posted by JayNL, 06-29-2009, 04:59 PM | |
ah damn, that's why apture.com is down too. That's just terrible, RackSpace is having more and more problems lately.
|
| Posted by pedrojose, 06-29-2009, 05:02 PM | |
that site is only for the mosso/cloud network.
i guess rackspace does not have a similar site for they core-business!
very bad...
|
| Posted by pedrojose, 06-29-2009, 05:03 PM | |
back online for me
|
| Posted by jyarnold, 06-29-2009, 05:05 PM | |
They do for rackspace but funny enough it's down too.
|
| Posted by r3volt, 06-29-2009, 05:09 PM | |
here too! Power related again. They've been doing a bunch of UPS and Generator testing the last number of weeks so perhaps it is related.
|
| Posted by JayNL, 06-29-2009, 05:11 PM | |
Generates LOADS of buzz on the net atm:
http://twitter.com/#search?q=Rackspace
http://www.techcrunch.com/2009/06/29...avorite-sites/
http://news.cnet.com/8301-17939_109-...=2547-1_3-0-20
http://twitter.com/jtimberlake/status/2392959228
|
| Posted by jNive, 07-02-2009, 09:10 PM | |
just to follow up on this incident
http://www.datacenterknowledge.com/a...of-25-million/
Interesting lol - wouldn't want to be writing those sort of cheques myself
|
| Posted by Scott.Mc, 07-07-2009, 12:10 PM | |
They are out again, doesn't seem to effect as many of our systems as the last one did but it's certainly another wide spread issue, including their own website.
|
| Posted by Scott.Mc, 07-07-2009, 12:33 PM | |
Another power issue,
# uptime
11:25:22 up 4 min
|
| Posted by speedcolo, 07-13-2009, 05:54 PM | |
Rackspace's explanation is roughly:
• 13:51: utility breaker trip. Critical load on UPS. Gen cluster starting.
• 14:35: synchronization failure in gen cluster. Critical load back on UPS.
• 15:15: UPSes exhausted, load off.
They don't explicitly say that the UPSes operated from generator for a time, but imply it; and 84 minutes would be very long for battery run-time. Assuming the generators did parallel and power UPSes from 13:51 to 14:35, that would be 40 minutes run-time, which is believable assuming 2N UPSes.
But: if the generators were running in parallel, how did they lose synchronization? It's not as if a generator needs to "try" and stay synchronized. The current on the emergency bus, and in the windings of each generator, forces the generator to stay in sync. I have read that you could shut off fuel supply to a generator, and if the parallel cluster is large enough, the energy on the emergency bus and in the windings will change it from being a generator into being an electric motor: the rest of the cluster will drag it along, making it turn in sync with the rest of the generators, even while it is turned-off. So I don't understand how a set of parallel generators would lose sync.
I did read one account that the generators lost sync with the UPSes. Would that imply that the generators were configured to parallel off the UPSes' critical bus? Maybe that would explain it, if the UPS output Hz. was drifting beyond generator tolerance (but why?... was there no bypass input present when running from generator? If bypass input was also from gen, then the UPS should have sync'd to the generator cluster).
Last edited by speedcolo; 07-13-2009 at 06:02 PM.
|
| Posted by speedcolo, 07-13-2009, 06:08 PM | |
Rackspace wrote:
This implies that the generators were not quite paralleled, but yet were still powering load for half an hour. I don't understand how that could work. Once the paralleling bus closes, if they weren't in sync, then they would yank each other into sync... violently if necessary.
|
| Posted by sailor, 07-13-2009, 11:13 PM | |
"Each generator failed on a loss of excitation an inability to maintain the magnetic field"
They probably just need to dump a bottle of viagra in the diesel tanks in case they need them to be able to keep it up next time. Loss of excitation stinks.
but on a serious note
Does anyone know how often they test the gen sets and if they do a full load transfer or if they just start them up and exercise them with no load?
|
| Posted by Softsys Hosting, 07-14-2009, 01:45 AM | |
Most generally, this is done every 15-30 days by doing a full load transfer in most of the datacenters. At very least, I am aware of Handy Networks doing this regularly.
|
| Posted by ZenMonk, 07-14-2009, 02:16 AM | |
"Each generator failed on a loss of excitation an inability to maintain the magnetic field"
How many KWA generator do they have?Overload?
|
| Posted by speedcolo, 07-14-2009, 02:19 PM | |
Overload would do it. It could take a while for heat to build-up due to THD, and that would shut-down the gen-sets in a cascading fashion. That kind of overload would imply that the facility had not been run from generator for longer than a half-hour at today's load. That's not inconceivable, especially if load has increased over time.
|
| Posted by sailor, 07-15-2009, 10:21 AM | |
I guess this is a good reason to ask your data center if they run their gen sets under load at least 2 times a month and if they do extended run tests under load to simulate a real long term outage at least a few times a year.
|
| Posted by bryonhost1, 07-15-2009, 10:33 AM | |
Hi!
I'll be the first to admit I don't know much about gensets. However..
I'm always willing to learn. Look what I found:
http://www.control.com/thread/1026230100
This certainly helps most of us understand more about this.
>>Of course..it helps to be an engineer.
>>>Ohho! A solution!
http://pm.geindustrial.com/FAQ/Docum...9/GER-3183.pdf
Bryon
Last edited by bryonhost1; 07-15-2009 at 10:38 AM.
|
|
 Add to Favourites
 Print this Article
|
