Sheer Panic

| | Comments (0)

So the file server went live Sunday night.

This afternoon one of the disks in the array decided to disconnect itself from the array. No reason. It just didn't want to play anymore. So I pulled it out, plugged it back in, and it started rebuilding.

Now this is a 7x146gb disk array, and rebuilding a single disk takes hours. 30 of them at the current rate.

If we lose another disk before it's done (which is more likely to happen than not, given that there was no apparent error, it just felt like doing it). And then several hundred people will be pissed off at losing all their days work since the last backup, not to mention not having access to the rest of their files while we figure out how to get everything back for them without a working disk array.

I am absolutely terrified about what could happen between now and tomorrow night. It's the level of stress I could really do without.

Going to get up at 2am after the current backup job is finished and crank up the rebuild rate and hope it can rebuild itself before work hours tomorrow.

Kazza's "Boring Life Of a Geek" aka BLOG

IT geek, originally from Sydney, moved to Canberra in 2007. Married to "the sweetie", aka Stu. Prolific photographer, Lego junkie and tropical fish keeper.

Kazza the Blank One home