Vehicles, Hard Drives & Chocolate
So after arriving from Victoria I needed a vehicle. I left my car there because I didn't feel like driving 14 hours for 2 weeks to turn around and go back. AJ said I could use one of their extra vehicles. Turns out he couldn't get the mini van back soon enough so he let me borrow the Expedition. Totally cool! I feel like a monster driving that thing ... it's so _HUGE_. But a pig on gas. I got it on E, and I put $20 in. That only gave me a quarter tank! Today I went with AJ to pick of the van, and now I'm driving that instead. Much better.
It's a good thing I've had a vehicle over the weekend because man did I need it. The hard drive went on the southland server while I was in Victoria. I rebooted and it froze. AJ went up Friday morning and was telling me what's on the screen ... not good. I couldn't walk someone through it over the phone. I needed to be there. So Saturday I went up and booted into maintenance mode. I had to mount a bunch of partions to figure out what they were before fixing them ... I can't remember how big and what partions were on a server I built 2.5 years ago. Anyways I had to manually run fsck on a RH 7.3 box with ext3 filesystems. Man were there a lot of inode unbalances. I cleaned everything, and the system did reboot cleanly. I did not want to leave that disk in there to have it happen again. So I dumped all of the data to my home machine and burned it to a disc.
I went around to Computer Rack to see Brian. He wasn't working but I needed supplies. I picked up 2 80G Seagate disks, an LG CD-writer, IDE cable, and some discs. I was going to rebuild that system properly. I just got some SuSE 9.2 discs and decided it was time. I grabbed the configs and away I went. The Install went fairly easily. I did end up redoing my partitions table a couple of times to get it right. It's not overly intuitive when you are trying to set up LVM and RAID at the same time. I got it done right, which makes me happy.
Now Murphy's law when you re-install a server. I've set everything up and plugged the box back in. No internet. I change the IP based on the config file I saved. Doot Doot Doot ... hrm still no internet. I can't get by host name so I realized I forgot the DNS servers. I checked on another box up there to get the IPs. Still nothing ... not even by IP can I get anywhere. Oh crap ... I forgot about the gateway. I mean I forgot to get the value of the gateway from the old configs. So I ended up plugging in the hard drive to another machine to log in and check the IP value.
So I got it up and running. At this point in time I realized the server next to the one I just fixed started to flake out. Man ... how do two separate servers have their hard drives fail at the same time. Ever had a segmentation fault when you run ls? I did over SSH, which led me to believe it might be a RAM problem. I ran the x86Memtest on it with no errors. And the system would never boot up again.
This box was the first server install I did for this client over 3 years ago. It's running RH7.3 with an ext2 filesystem. I'm not sure how many of you remember non journalled filesystems, but I'll remind you why they suck. Becuase whenever I had a problem and had to power off the box because it hung during init, the filesystems were not cleanyly unmounted, running fsck on /, /home, /usr ... on and on. When trouble shooting a freezing computer with a flaky hard drive ... that's not fun. It's very time consuming and annoying as all get out.
Amidst my computer troubles at the north office, I realized something. It's much nicer to do those sorts of things during the day up there becuase the office tends to have lots of chocolate up there during the holiday season. I had to leave the office while waiting for one fsck just to stop stuffing my face with chocolate. Man those truffles are addicting.
I finally got the system to boot fully into single user mode. I tested out the internet connection. I could get in, and out. YIPPEE. This means I can at least get the needed data off of that sick computer before it dies. I had tar die a few times creating archives to move around easily. I ended up just moving the directories, and planned to transfer through KDE. I could drag and drop, because I couldn't tar them up and sftp them back. By hook or crook I'll get my data back! So I had the file transfers die a few times while transfering files through the kio_slave fish:// That was kind of alarming to me. I decided it would be a good idea to try moving stuff in smaller chunks. So one directory at a time with hundreds of small files in them. Eventually I got all of the files ... somewhere. I have to re-organize them still, but it's there.
After all was said and done I left and decided to install software, and configure things remotely from home. I didn't realize that SuSE 9.2 does not have binaries on the mirrors yet. Not until mid January I read. Damn! So I spent an hour attempting to build some things from src rpms. _EVERY_ time I did one I ended up missing -devel packages or other dependencies. That's a minimal system for you. I finally got fed up and spent 7 minutes compiling PostgeSQL from source. I already had apache, and PHP installed from the CDs. postgresql-server was not available though. :-( But it's all good, healthy, and everyone seems happy now. Lets see what tomorrow brings for the next hard drive I'll reformat.
Oh ... and happy holidays!