Thursday, 12 August 2010

There should only be 1 four o clock!

For the last 2 days I have been at work since 0400 hrs.! It sucks!

In a nutshell, one of our servers went pop on Monday and the techies managed to get everything working except SQL replication. Now setting this back up should not usually cause a problem but since the database has grown since the last time I re-initialised replication there is now insufficient space to hold the snapshots that I need. Therefore I needed to replicate the tables one at a time.

Of course I didn't discover this space issue until 0840 on Wednesday when the attempt at replicating the whole job lot failed spectacularly and I discovered that I'd wasted almost 5 hours of my life that I'm never gonna get back again.

So that was it - Back in today at 0400 to do it properly (well one at a time) this time!

What are the odds of 2 raided hard drives failing at the same time....?


  1. Regarding the RAID question, do you mean two drives in different arrays or two drives in the same array? also, what level of RAID and how many drives in the array?

    eg. if the drives are both in a RAID0 array, then the odds of both drives failing is irrelevant, if any drive fails, then the whole array fails.

    Whereas if the drives are each in different RAID1 or RAID5 arrays (other RAID levels are available), then two drives failing will allow the arrays to continue to function (albeit degraded) until the drives are replaced. Of course, in the case of smaller arrays (in terms of numbers of disks), you'll certainly be doing a lot of prayer regarding the other disks in the array not failing before the broken disk(s) have been replaced.

    Well done on the 4am starts anyway.

  2. don't pray for the hdd's.... pray for better server engineers :)