How was your day dear?: October 2004 Archives

My finger hurts

| | Comments (6)

Or the nail does, if that were possible. From holding it pressed down for three hours straight. A bunch of us played unreal tournament for several hours, it was a blast.

The central mail server has been down since last night. Power outage of some sort killed the machine, and they're having to recover it from tape. Yay. And it was the old mail admin's last day today too! heh. But since I was expecting some mail from overseas and I couldn't get it, it meant I had a reprieve from doing boring work, and got to spend the day on something somewhat more fun. Always a good thing.

And finished reading "Liner" by James Barlow tonight. Definitely not one of the better books I've read. It's got some great technical detail, even too technical, about the mechanics of running a steam-powered ocean liner, which was quite interesting. But he spent forever developing a bunch of loser characters, almost none of them I liked at all, and it was a full 3/4 of the way through the book before anything actually *happened*. And when it finally did, it was all over very quickly. I almost didn't read it at all, because it was *extremelly* violent at the beginning - so much so that I skipped over whole paragraphs of text within the first few pages. I wouldn't have missed much if I had stopped there. It's kinda weird that I can tolerate violence on tv much better than in books. Violence in books seems somehow much more intense and disturbing. I really don't like it at all.

Anyways, bed time!

*yawn*

| | Comments (0)

'nuff said

Can I go home now?

| | Comments (0)

Well according to the pedometer I did about 10000 steps today. But I don't believe it. Especially when probably a thousand of those "steps" were actually bumps on the road while I was driving. I think I need to paint it black too - then it might just look like a pager.

Was totally wasted at work today. Still haven't recovered from Sunday morning. Went home a bit early, but not nearly as early as I wanted to. Dell are trying to appease us with new hardware though which is kinda funny. I guess they must realise that they should never have sold us what they did knowing how we'd want to configure it. Now they're grovelling. Guess that's a good thing.

Other than that, life is boring.

Weary

| | Comments (1)

I've never been so work-weary in my entire life. I've almost reached the end of my tether with the file server. If one more thing goes badly I honestly don't know what I'll do.

RTA uselessness

Get this. A 22 car accident, one person killed, the entire route north of the city blocked for hours, and the RTA give it an ORANGE ?!?!? I would have been thinking double red. Their site is often as useless as this.

I'm so tired. And trying to make a backup work. cry.

Horizontal

| | Comments (0)

That's how I want to be right now. Dead or asleep, whichever. File server is depressing me. More corrupt disk issues and completely missing directories. I dread to think of how many files have gone missing this week. Hopefully on the weekend we'll be able to copy everything to different disks, then nuke the container and start again.

Have mostly cleaned up the mess on the file server. Still going to have to clean up the corrupt files, either by another chkdsk, or wiping the partition and starting again. Then we have to figure out the raid problems. I'm fairly certain it was the ultra 320 interface. So far no incidents now that we've gone back to the ultra 160 card. Not to mention the backups are now running at a full gig/minute, as opposed to ~650mb/minute with the 320 card. I think we should send Dell an invoice for total lost productivity. My boss had to face a "please explain" from the powers that be. It wasn't too bad, they just wanted to know what went wrong and what we were doing to make sure it didn't happen again. I felt really bad for him though.

But just to keep my disasters in perspective.

weather silliness

Disaster 1. Australia's weather. We've been in drought all year, and today there was so much rain that whole towns have been completely flooded.

staticports.jpg

Disaster 2. An Aeroperu flight crashed in 1996 because a maintenance worker taped up the static ports while cleaning the plane and forgot to remove the tape. The ports are used to monitor flight speed and altitude, and the pilots were getting all sorts of errors, and eventually crashed into the sea because they didn't know how high they were. 70 people died. This was the second episode of Air Crash Investigations I've seen - fascinating show. Last week they had an airbus that ran out of fuel due to a fuel leak, and glided 100 miles over the middle of the Atlantic Ocean and landed safely in the Azores. Freaky stuff!

I think I had just about the worst day of my career today. On Saturday morning during the first of the raid container rebuilds the file system on the file server decided to get corrupted. But of course with very few people accessing the server we didn't know about it until this morning, when one group of users couldn't get to their directory at all. After we checked the event logs and saw what was happening we thought we should do an emergency shutdown and chkdsk so that the affected users might be able to do some work that day. However the server decided it wanted to reset the permissions on every file on the server. All one million of them. After two and a half hours and 150000 files we decided we'd risk losing everything and reboot out of it. The server came back up, but all the file permissions were utterly screwed.

So while everything was so fubar, we decided to go back to the ultra 160 emm card and cable, because we knew that they at least worked. Plus dropping back to 160 from 320 we may eliminate some of the flakyness in the raid containers.

So after replacing the card and rebooting, it was then a matter of resetting all the permissions. Fortunately I'd only recently done a bunch of permissions scripts and consolodation of our documentation, so it wasn't such a difficult thing to reset all the permissions on the box to what they should be. But it did take many hours.

I still have a few major directories to go, and then I have to figure out which files were corrupted and try and restore those from last week's backup tapes.

The irony of all of this is, the backup tonight is running faster than it's ever run before on the new server. And this is on a "slower" emm card. Which may tend to suggest a problem with the 320 emm card or the new cable. Who knows. Dell certainly don't. Before all of this started I talked to them this morning, and they were going to escalate it to their second level support. Well they called back this afternoon and I kinda let them have it somewhat. Well not badly, I just told them what I'd gone through all day and that we'd gone back to the old card. He offered to send out a tech and I said well that would be somewhat difficult on a production server to "fiddle". I want to get the machine settled a bit more, and have some sort of way to recover to another machine if necessary. This of course involves buying another server, and replication software that I haven't had time to research yet :/ Blah.

Urgh. What a horrible day. It's the biggest unplanned outage I've ever had at work. Very distressing.

I was going to go to bed, but I have this urge to watch the backup finish, so watching Peter Pan to recover my mood somewhat.

I blinked and missed it somehow.

Raid array lost disks *twice* over the weekend. Going to be complaining bitterly to dell tomorrow.

The baby kribs are getting bigger. Some of them have fairly distinct horizontal bands on them now. Although they've gone very skittish and won't come up and eat out of my fingers anymore :(

Went to the parents for dinner... mmmmm roast pork and crackling.. food doesn't get any better than that.. :):):):) David came too and we played a game of 500 (Dad and Dave won - doh :) )

I also suggested my idea of getting Christmas presents for ourselves rather than for each other and then bringing those to Christmas day so everyone else could see what we got. Mum completely loved the idea. Dad doesn't care, not sure what David thinks. It seems a little wrong somehow, as the whole idea of Christmas presents is about giving to *others*, but practically speaking, we can all get something that we really want, and say it was from the family, end up spending the same amount of money, and not have to go through the hassle of trying to find something the others would like (which more often than not we have to ask each other what we want anyway). So anyway, we'll see how we go.

Or something

| | Comments (7)

Ended up sorting out lego all day. Whoops.

Also thought I'd watch Peter Pan and Lethal Weapon 2 tonight, but instead watched The Parent Trap and MI2 has just started. Whoops.

Cooked Alan's famous stroganov for dinner tonight. Although really must go and do the dishes.

Rather sleepie. Trying to decide whether I'll stay up til 1am to watch an ebay auction. The set going is #1593, the rarest of all the space lego sets, worth at least $400US. It's got a starting price of 99 pounds and there are *zero* bids. But the counter is well over 500, so I'm guessing there'll be a flurry of last-minute sniper bids.

Also have to make sure the raid array finishes rebuilding. It died *again* last night. That's three times now. Talk about dodgy brothers Dell hardware.

And Hot Again

| | Comments (0)

Hot again today although not as ridiculous as yesterday. Beautiful cool change swept through, although unfortunately I've been out all night so didn't get to open the windows to let the air through :( Still and oven in here :(:(

Oh I forgot to mention I saw Collateral with Striker last night. Was not too bad, especially for a Tom Cruise movie :)

Hotter

| | Comments (0)

Another day another 38C. Another hour long trip in blazing heat, held up by *multiple* breakdowns. Another trip to the post office. Another dead disk.

So the boss says "attach the disk array to the other server, we need the disk space" and I say "but but but what happens if another drive dies" and he says "just do it, it'll be fine" so I do it, and another disk dies. Different disk, but this time instead of saying it'd been removed, it said it had failed. So I pulled it out, stuck it back in, and it's been rebuilding its raid array ever since. 52% and counting. Sent a screen dump of the failure to the boss. At the very next opportunity I'll be removing the old hardware from the equation, as it's almost certainly an old raid card attached to the box that's causing it to freak out.

Hot

| | Comments (0)

I didn't think I was going to make it home tonight. A trip that should have taken twenty minutes took nearly an hour. In blazing 35C heat. With a huntsman spider running around the car freaking me out. And making some traffic lights but not others. Trying to get to the post office by 5pm. Made it with one minute to spare. Nearly died in searing heat as I fought to get the key in the ignition (I'd ripped it out in such a hurry trying to get to the post office it was out of alignment).

Phewf.

Sydney is going to be in all sorts of trouble this summer if today was any indication. Hot and dry all day. That reminds me, must water the plants.

Definitely can't do mod rewrite rules with .htaccess on our server, they need to be done on the server end in the zeus interface. Oh well.

And it seems I wasn't the only one with the idea to watch Superman. Channel 9 is airing it this Friday night.

Well it took just over 24 hours but the raid container rebuilt itself and is online again. Dared to take a breath briefly this afternoon. Backups are still having issues tho :(

Decided to remove the old old server from the equation, perhaps it was having it attached that caused it to freak out. We'll see.

Then a joyous body corporate meeting we're they're basically deciding to run the block ourselves rather than getting a strata manager. Which will mean lots more work for all of us :/

Spent a goodly portion of today tidying up. Luc and Liz and the kids came over to see all the space lego. Aside from that didn't get much done. Watching Harry Potter and the Philosopher's Stone (and ate a bunch of Bertie Bott's Every Flavour Beans at the appropriate moment :) ). It's strange to me that noone wants to eat them when I offer!? I don't know why that would be.. lol!

The Weekend So Far

| | Comments (2)

Friday night:
Crash out with contreau

Saturday morning:
Finish reading the Malloreon

Saturday afternoon:
Finish uploading new Host1 pricing and momentary freedom

Saturday night:
Go to work and play musical servers/hard drives/disk arrays/scsi cables/emm cards

Late Saturday night:
Head out to Malabar and watch fireworks at the bunker near the rifle range

Sunday morning:
Check the file copy job and think it's carked it half way through. Head in to work to kick it

Sunday:
Spend the rest of the day supervising file copying, and playing musical servers

Sunday night:
Crash out in front of Titanic, and catch up on my memes

Kazza's "Boring Life Of a Geek" aka BLOG

IT geek, originally from Sydney, moved to Canberra in 2007. Married to "the sweetie", aka Stu. Prolific photographer, Lego junkie and tropical fish keeper.

Kazza the Blank One home