Saturday, December 10, 2005
Parachutes, New Hardware & Parking
So it's been a crazy couple of days. I was plugging away on rebuilding the server for Fortress internally, and Gregg phoned me up and asked me about doing a quick Atrium job. He said "It's kind of a parachute mission". The clients had hit disaster, with complete loss of email, web, DNS and who knows what else. Their computer tech is a nice guy, but he is in _way_ over his head. It's clear when several of us worked with him that he just really does not know enough about computer systems. Gregg mentioned when he first got there that they had 2 DHCP servers running on the same network, and that was causing IP conflict problems. They had moved DNS to network solutions, and I found on the old server named files. I was not informed the new systems were to be public name servers. I just left handling the routing, DHCP services, and everything else for the time being. I wanted to get the mail servers running, and ASAP.
Let me backtrack for a bit. Gregg confirmed we were going ahead on Tuesday afternoon. I was delivered the new servers and hardware Wednesday morning, and I began the process. The first problem I encountered was the pain of installing a second hard drive into the cage. The cables (with some re-arranging) were just barely long enough to provide access and power. After I had the boxes assembled I attempted to install Kubuntu 5.10. The first installation was underway and I started on the second server. When the reboot happened it booted directly to console. I was surprised and figured something was incorrect. It happened on the second computer as well. I could login without problems, yet starting X would crash horribly. To cut the story short, after a couple of hours of investigation, the kubuntu installation does not properly detect the on board ATI RADEON EXPRESS 200 IGP cards. It configures the xdriver "ati". The x driver "radeon" does not work either. It turns out I needed to install the xorg-driver-fglrx package and manually configure xorg.conf to load the module, and use that display driver. After the hardware issues were resolved, the Kolab installations were a breeze, and the servers were ready to go. I also imported the users with some scripts I have written.
I installed and setup the mail servers on site. We waited until after hours to turn on the DNS MX records, and switch everything over. Everything was working properly from my side. I know when I left we had switched accounts and tested them. They were having internally network issues though. Ray helped them out the next morning. There was a small issue on the new servers where the system clock was running 2-3 faster than normal. THis meant the email times would be all out of whack. I tried a fix which didn't work, and then I hacked a cron job to restart ntpdate every minute until I figured things out. It turns out to be a bug with some motherboards and the ubunutu 2.6.10 kernel. I hade to add some kernel options (noapic nolapic noapictimer) to the kernel boot line in /boot/grub/menu.lst. I did this remotely, and rebooted a few times. The CPU was no longer double clocked, and top registered 90% and over idle, instead of the 50% auto use as before. That was a tricky bug, and has apparently bitten others.
During all of this mass rush to get things working, I have really come to dislike the parking situation in downtown Calgary. When I went to do the initial on site setup, I went to 5 different lots and they were all full. Signs up, attendants turn me away. I drove past the client building 6 times, and finally parked by the band room, and jumped the C-Train. It's funny how there was a write up in the paper about parking that same day. Everyone seems to agree it's a mess. It won't get any better I don't think. That's why I don't go out in the day much. ¶ 3:27:00 PM
Let me backtrack for a bit. Gregg confirmed we were going ahead on Tuesday afternoon. I was delivered the new servers and hardware Wednesday morning, and I began the process. The first problem I encountered was the pain of installing a second hard drive into the cage. The cables (with some re-arranging) were just barely long enough to provide access and power. After I had the boxes assembled I attempted to install Kubuntu 5.10. The first installation was underway and I started on the second server. When the reboot happened it booted directly to console. I was surprised and figured something was incorrect. It happened on the second computer as well. I could login without problems, yet starting X would crash horribly. To cut the story short, after a couple of hours of investigation, the kubuntu installation does not properly detect the on board ATI RADEON EXPRESS 200 IGP cards. It configures the xdriver "ati". The x driver "radeon" does not work either. It turns out I needed to install the xorg-driver-fglrx package and manually configure xorg.conf to load the module, and use that display driver. After the hardware issues were resolved, the Kolab installations were a breeze, and the servers were ready to go. I also imported the users with some scripts I have written.
I installed and setup the mail servers on site. We waited until after hours to turn on the DNS MX records, and switch everything over. Everything was working properly from my side. I know when I left we had switched accounts and tested them. They were having internally network issues though. Ray helped them out the next morning. There was a small issue on the new servers where the system clock was running 2-3 faster than normal. THis meant the email times would be all out of whack. I tried a fix which didn't work, and then I hacked a cron job to restart ntpdate every minute until I figured things out. It turns out to be a bug with some motherboards and the ubunutu 2.6.10 kernel. I hade to add some kernel options (noapic nolapic noapictimer) to the kernel boot line in /boot/grub/menu.lst. I did this remotely, and rebooted a few times. The CPU was no longer double clocked, and top registered 90% and over idle, instead of the 50% auto use as before. That was a tricky bug, and has apparently bitten others.
During all of this mass rush to get things working, I have really come to dislike the parking situation in downtown Calgary. When I went to do the initial on site setup, I went to 5 different lots and they were all full. Signs up, attendants turn me away. I drove past the client building 6 times, and finally parked by the band room, and jumped the C-Train. It's funny how there was a write up in the paper about parking that same day. Everyone seems to agree it's a mess. It won't get any better I don't think. That's why I don't go out in the day much. ¶ 3:27:00 PM
