Comedy Of Errors Bringing A Server Back From The Dead

May 8, 2007 personal suggest edit

Not too long ago I mentioned that a power surge bricked the Subtext Build Server. What followed was a comedy of errors on my part in trying to get this sucker back to life. Let my sleep deprived misadventures be a cautionary tale for you.

My first assumption was that the hard drive failed, so I ordered a new Hard Drive.

Lesson #1: If you think your hard drive has failed, it might not be a bad idea to actually test it if you can. Don’t just order a new one!

I have my main desktop machine I could have used to test the drive, but due to my sheer and immense laziness, I didn’t just pop the drive in there as a secondary drive to test it out. I just ordered the drive and moved on to other tasks.

Days later, the drive arrived and I popped it in and started to install Ubuntu on the machine. As I got to the disk partitioning part, I noticed that it found a disk and I went ahead and formatted the drive and installed Ubuntu. Sweet! But when I rebooted, the server not find the drive. Huh?

Lesson #2: When installing an Operating System on a machine, make sure to unplug any external USB or Firewire drives.

Yep, I formatted my external hard drive and installed Ubuntu on that. The Ubuntu installation process recognized my firewire drive and offered that as an available drive to partition and install. Ouch!

At this point, I realized that the machine was not detecting my brand new hard drive, though I could hear the hard drive spin up when I powered on the machine. I figure that quite possibly it’s a problem with the SATA cable. So I order a new one.

Lesson #3: In the spirit of lesson 1, why not just temporarily pull a SATA cable from your other machine, if you have it.

I thought the SATA cables were all inaccessible and would be a pain to pull, but didn’t bother to check. It was in fact easy to grab one. To my defense, I figured having extra SATA cables on hand wouldn’t be a bad idea anyways and they are cheap.

So I plugged the SATA cable that I know to be good into the box and still it won’t recognize the hard drive. At this point it seems pretty clear to me that the drive controller on the Motherboard is fried. Any suggestions on how to fix this are welcome, if it is even possible.

In any case, after a good night of sleep, I started doing the right thing. I plugged the old drive into my desktop and sure enough, I can copy all its files onto my main machine.

I installed VMWare server and the build server is now up and running on my main desktop for the time being. Woohoo!

As a side note, I tried to use this VMDK (VMWare) to VHD (Virtual PC) Converter (registration required) so I wouldn’t have to install VMWare Server on my machine, but it didn’t seem to work. Has anyone had good luck converting a VMWare hard disk into a Virtual PC hard disk?

Long story short, do not under any circumstances let me anywhere near your hardware. At least the build server is back up and working fine. It is officially time to subscribe to mozy.com. I’m exhausted. Good night.

Found a typo or mistake in the post? suggest edit

Comments

14 responses

Mike Minutillo • May 8th, 2007
This reminds me of a time when I had to fix my mums printer. I plugged into my machine and it didn't work. I figured it might be the cable and so I tried another one. When that didn't work I tried a third.
Eventually I replaced the printer only to find out that all of the printer cables I'd tried were bad.
It had not occurred to me that that might happen.
Adi • May 8th, 2007
You don't need the motherboard HD controller - just buy a SATA PCI Controller Card.
chrisb • May 8th, 2007
haha, excellent :) gotta love lesson #2
Krenn • May 8th, 2007
Another option - depending on your motherboard, you may actually have two SATA controllers on the board. Gigabyte does this commonly, with one set (the south bridge) yellow and one set (their controller) purple.
If you do have two controllers, it's unlikely that both of them are dead. But if it's the south bridge controller that's bad, then you may have more problems than just the SATA failing.
Ray Booysen • May 8th, 2007
Don't try Mozy. ;) I've had numerous issues with them.. Trying Amazon's S3 at the moment.
David Mohundro • May 8th, 2007
I can relate to the OS installation wiping your external drive.
In fact, something similar happened TWICE. The first time, I lost backups on my external drive.
The second time, I sort of forgot that the install displays your drives in the order they were plugged in (i.e. SATA0 is the first, SATA1 is the second, etc). Well, my C: drive had been SATA1 so I wiped my data drive. Needless to say, I actually did lose data in that one.
If you give Mozy (or some other service) a try, I'd be interested in hearing your experiences with it.
Sergio • May 8th, 2007
Krenn has a good point. I had a bit of a hard time understanding which devices I could have connected at the same time and on which channels. My mobo has connectors for 4 PATA drives and 4 SATA drives but I cannot have all of those at the same time.
But, of course, it can still be fried anyway.
Steven Harman • May 8th, 2007
Assuming that the Shuttle mobo didn't have two (2) SATA controllers, I think the next best option is to spring for a new SATA controller card. I bought one for my old P4 workstation when I turned it into my home server and I think it ran me about $30 from MicroCenter.
The Other Steve • May 8th, 2007
Nothing beats the time i was on a job, and the client had purchased 4 DEC Alpha servers to run their NT domain off of. Three of them were in pieces, because we had to use the parts to get the last one running long enough to setup a domain controller while waiting for DEC to provide warranty service. :-)
Aaron • May 8th, 2007
Lesson 1
I now have usb adaptors so i can run hard drives on any other system. it was cheap and cool ( only have open one PC )
Lesson 2
My AlienWare computer did an xp sp2 upgrade and tanked when i reformatted I some how picked the wrong drive(usb external) and there went over 100 gigs of my music collection. ( That hurt )
Darren Kopp • May 8th, 2007
I fell your pain. I remember when i spent 3 hours backing all my stuff to another hard drive in my computer, started the xp install, formatted the partition, went and got a drink, sat down, sighed...... OMG. what hard drive did i pick??????
in the end i just went back to my trusty ontrack backup. got about 90% of the data :S.
The lesson i learned? don't just format the drive you think you should format based on the drive letter that it has in the setup process...
Jon Limjap • May 9th, 2007
Hah, whenever I have to reformat a harddisk for an OS reinstall I always *pull* the SATA cables AND the power from my backup disk (I use a 2nd HDD as backup), just to make sure I'm reformatting the right one! :p
Denis Basaric • May 9th, 2007
As it may just yesterday I posted the article on How to prepare for hard-disk catastrophic failures. The crashes happen more often than you think and after seeing number of emails from people writing to request backup of their serial numbers and software from us, I decided to write down my processes and procedures for backup.
It is a lengthy piece but I try to outline what worked for me and hope that other people find it useful.
BTW, I use Mozy and works great for me. However, it is online backup system and I recommend using any online backup system for critical data that are not too large. I do not think that backing up 1-2 Gigs is really practical over Internet.
Hope this helps...
hellrazer1976 • October 3rd, 2007
I did one of those os reloads just the other day on a costumers laptop that he had four boot choices of xp pro that all were dead. so I popped in my trusty os reload disk I had ordered from Microsoft and rebooted when i got to the hdd setup screen. I saw about ten partitions on his drive so I started deleting them. I FORGOT to pull my 2gig thumb drive out of the laptop, well I lost about $2500 worth of system tools on it! you ask about the backups, well the owner dropped the 250gig ext. drive off our 3 foot high work bench while it was running. it made a loud screech and only clicked ofter that. double ouch!!! Don't ask what that cost him.