Comedy Of Errors Bringing A Server Back From The Dead

Not too long ago I mentioned that a power surge bricked the Subtext Build Server. What followed was a comedy of errors on my part in trying to get this sucker back to life. Let my sleep deprived misadventures be a cautionary tale for you.

My first assumption was that the hard drive failed, so I ordered a new Hard Drive.

Lesson #1: If you think your hard drive has failed, it might not be a bad idea to actually test it if you can. Don’t just order a new one!

I have my main desktop machine I could have used to test the drive, but due to my sheer and immense laziness, I didn’t just pop the drive in there as a secondary drive to test it out. I just ordered the drive and moved on to other tasks.

Days later, the drive arrived and I popped it in and started to install Ubuntu on the machine. As I got to the disk partitioning part, I noticed that it found a disk and I went ahead and formatted the drive and installed Ubuntu. Sweet! But when I rebooted, the server not find the drive. Huh?

The Scream - Edvard Munch Lesson #2: When installing an Operating System on a machine, make sure to unplug any external USB or Firewire drives.

Yep, I formatted my external hard drive and installed Ubuntu on that. The Ubuntu installation process recognized my firewire drive and offered that as an available drive to partition and install. Ouch!

At this point, I realized that the machine was not detecting my brand new hard drive, though I could hear the hard drive spin up when I powered on the machine. I figure that quite possibly it’s a problem with the SATA cable. So I order a new one.

Lesson #3: In the spirit of lesson 1, why not just temporarily pull a SATA cable from your other machine, if you have it.

I thought the SATA cables were all inaccessible and would be a pain to pull, but didn’t bother to check. It was in fact easy to grab one. To my defense, I figured having extra SATA cables on hand wouldn’t be a bad idea anyways and they are cheap.

So I plugged the SATA cable that I know to be good into the box and still it won’t recognize the hard drive. At this point it seems pretty clear to me that the drive controller on the Motherboard is fried. Any suggestions on how to fix this are welcome, if it is even possible.

In any case, after a good night of sleep, I started doing the right thing. I plugged the old drive into my desktop and sure enough, I can copy all its files onto my main machine.

I installed VMWare server and the build server is now up and running on my main desktop for the time being. Woohoo!

As a side note, I tried to use this VMDK (VMWare) to VHD (Virtual PC) Converter (registration required) so I wouldn’t have to install VMWare Server on my machine, but it didn’t seem to work. Has anyone had good luck converting a VMWare hard disk into a Virtual PC hard disk?

Long story short, do not under any circumstances let me anywhere near your hardware. At least the build server is back up and working fine. It is officially time to subscribe to mozy.com. Im exhausted. Good night.

Technorati tags: , , ,

What others have said

Requesting Gravatar... Mike Minutillo May 09, 2007 2:39 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
This reminds me of a time when I had to fix my mums printer. I plugged into my machine and it didn't work. I figured it might be the cable and so I tried another one. When that didn't work I tried a third.

Eventually I replaced the printer only to find out that all of the printer cables I'd tried were bad.

It had not occurred to me that that might happen.
Requesting Gravatar... Adi May 09, 2007 2:44 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
You don't need the motherboard HD controller - just buy a SATA PCI Controller Card.
Requesting Gravatar... chrisb May 09, 2007 4:25 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
haha, excellent :) gotta love lesson #2
Requesting Gravatar... Krenn May 09, 2007 4:29 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
Another option - depending on your motherboard, you may actually have two SATA controllers on the board. Gigabyte does this commonly, with one set (the south bridge) yellow and one set (their controller) purple.

If you do have two controllers, it's unlikely that both of them are dead. But if it's the south bridge controller that's bad, then you may have more problems than just the SATA failing.
Requesting Gravatar... Ray Booysen May 09, 2007 5:26 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
Don't try Mozy. ;) I've had numerous issues with them.. Trying Amazon's S3 at the moment.
Requesting Gravatar... David Mohundro May 09, 2007 5:32 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
I can relate to the OS installation wiping your external drive.

In fact, something similar happened TWICE. The first time, I lost backups on my external drive.

The second time, I sort of forgot that the install displays your drives in the order they were plugged in (i.e. SATA0 is the first, SATA1 is the second, etc). Well, my C: drive had been SATA1 so I wiped my data drive. Needless to say, I actually did lose data in that one.

If you give Mozy (or some other service) a try, I'd be interested in hearing your experiences with it.
Requesting Gravatar... Sergio May 09, 2007 5:34 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
Krenn has a good point. I had a bit of a hard time understanding which devices I could have connected at the same time and on which channels. My mobo has connectors for 4 PATA drives and 4 SATA drives but I cannot have all of those at the same time.
But, of course, it can still be fried anyway.
Requesting Gravatar... Steve Harman May 09, 2007 6:23 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
Assuming that the Shuttle mobo didn't have two (2) SATA controllers, I think the next best option is to spring for a new SATA controller card. I bought one for my old P4 workstation when I turned it into my home server and I think it ran me about $30 from MicroCenter.
Requesting Gravatar... The Other Steve May 09, 2007 6:49 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
Nothing beats the time i was on a job, and the client had purchased 4 DEC Alpha servers to run their NT domain off of. Three of them were in pieces, because we had to use the parts to get the last one running long enough to setup a domain controller while waiting for DEC to provide warranty service. :-)
Requesting Gravatar... Aaron May 09, 2007 6:53 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
Lesson 1
I now have usb adaptors so i can run hard drives on any other system. it was cheap and cool ( only have open one PC )
Lesson 2
My AlienWare computer did an xp sp2 upgrade and tanked when i reformatted I some how picked the wrong drive(usb external) and there went over 100 gigs of my music collection. ( That hurt )
Requesting Gravatar... Darren Kopp May 09, 2007 8:48 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
I fell your pain. I remember when i spent 3 hours backing all my stuff to another hard drive in my computer, started the xp install, formatted the partition, went and got a drink, sat down, sighed...... OMG. what hard drive did i pick??????

in the end i just went back to my trusty ontrack backup. got about 90% of the data :S.

The lesson i learned? don't just format the drive you think you should format based on the drive letter that it has in the setup process...
Requesting Gravatar... Jon Limjap May 09, 2007 8:06 PM
# re: Comedy Of Errors Bringing A Server Back From The Dead
Hah, whenever I have to reformat a harddisk for an OS reinstall I always *pull* the SATA cables AND the power from my backup disk (I use a 2nd HDD as backup), just to make sure I'm reformatting the right one! :p
Requesting Gravatar... Denis Basaric May 10, 2007 4:46 AM
# re: Comedy Of Errors Bringing A Server Back From The Dead
As it may just yesterday I posted the article on How to prepare for hard-disk catastrophic failures. The crashes happen more often than you think and after seeing number of emails from people writing to request backup of their serial numbers and software from us, I decided to write down my processes and procedures for backup.

It is a lengthy piece but I try to outline what worked for me and hope that other people find it useful.

BTW, I use Mozy and works great for me. However, it is online backup system and I recommend using any online backup system for critical data that are not too large. I do not think that backing up 1-2 Gigs is really practical over Internet.

Hope this helps...
Requesting Gravatar... hellrazer1976 Oct 03, 2007 7:19 PM
# re: Comedy Of Errors Bringing A Server Back From The Dead
I did one of those os reloads just the other day on a costumers laptop that he had four boot choices of xp pro that all were dead. so I popped in my trusty os reload disk I had ordered from Microsoft and rebooted when i got to the hdd setup screen. I saw about ten partitions on his drive so I started deleting them. I FORGOT to pull my 2gig thumb drive out of the laptop, well I lost about $2500 worth of system tools on it! you ask about the backups, well the owner dropped the 250gig ext. drive off our 3 foot high work bench while it was running. it made a loud screech and only clicked ofter that. double ouch!!! Don't ask what that cost him.

What do you have to say?

(will show your gravatar)
Please add 6 and 4 and type the answer here: