So after not getting to sleep for ages last night I got woken up at 1:30 by a whole stack of pages in a row. Most of my servers were down. At first I thought something had freaked out the UPSes and they'd all shut down in error. But I looked more closely at the logs of the servers that I *could* reach. Sure enough "UPS on battery" on the ones connected with smart signalling cables. So power had gone out in just one section of the building. Checked with Jim what we should do. Decided to just go in early in the morning to turn everything back on. Watched as the upses drained themselves and turned the servers off, then went back to bed.
Woke up at 5:15. Looked at the two machines that aren't connected by serial cables to upses. They were still up. I assumed that the power had been restored before their ups died, so I got ready and went into work.
Total blackness. Everything was still dead. Even the machines that I'd checked before I left - their ups lasted over four hours!!
Sigh.
So called security who got onto the contractors who eventually came out. Watched them as they flicked some very big switches - so big they needed to use a metal bar wedged in them to turn them. Very cool. Seems that some big device in the building took out the earth leakage circuit for the whole section of the building. Yay. So power was out for most of the night. Hope people's freezers didn't warm up too much. Kicking myself I didn't call security at 1:30. But I blame only two hours sleep and not thinking straight on that one.
But the coolest thing is, all of the servers I connected up with their serial cables on Friday shut themselves down nicely. Still a couple of little issues to work out, but it was a good real-life test of the shutdown system.
I'm *very* sleepie now.
JC
Actually-- it may not have been all that big of an arc fault event to blow the main breakers that were reset. Quite frequently I see that the GFCI protection (Ground Fault Circuit Interrupter) settings on the large case breakers have never been moved from the factory settings. (Usually the electrical engineer who works with the architect gives the electricians the settings, factoring in the available kAIC, length of secondary conductors, etc) When this does not happen, and the short time I2T pickup rating is never moved from zero, and a minor fault-- say something on a 20amp circuit faults to ground, the amperage to ground shoots way up, maybe to 150-200amps-- enough to cause a flash, spark, and a Holy crap!, but since the short time I2T rating on the main is set to less than the hypothetical event in discussion, the main trips.
Practical application: The poor electrician who is changing the switch hot because he doesn't wanna hear all the 'suits' bitch about not having lights for all of five minutes in a very localized area, makes a goof, the aforementioned flash, spark and Holy Crap!, but now... the whole building is dark. Oopsie...
kazza
er... pardon..? I didn't understand the half of that .. hehehe :)
But it didn't die again today, which is always a good thing :)
JC
Er... that's the danger of being so specialized in one's respective field-- no one EVER knows wtf I'm talking about.