Saturday, November 20, 2004
VPN & Cigarettes
I know I'm a little bit late writing about this latest episode, but I've just been too drained to think about rehashing this experience for a few days. Let's just say I wanted to bask in the glory of completely disintegrating, and re-implementing a 6 node VPN that spans over 500 KM from furthest point to point. :-D
As any good system administrator knows that when dealing with _anything_ that involves upgrading, changing, installing etc. for a corporate environment, it automatically moves your working hours to outside of the regular business hours. That doesn't mean that you get the day off, it just means you work late. Sometimes you work really late.
I started last Friday afternoon at 4:00 P.M. Sent a staff wide email letting them know about network downtime starting after 5:00. I spent some time reading the manual, and researching the Linksys BEFVP41 VPN routers. I had 7 routers sitting on my desk, and in order to re-work the company's VPN setup, it was an all or nothing change over.
The fun thing about VPN networking is you are generally connecting to different physical locations. The two locations I was working from were about a 30 minute city drive apart. So I went slow and got the internet (NAT) working at the main VPN hub, and packed up a router and hit the road. Got internet working at location #2, but for all of my efforts I could not get the VPN connection to properly work.
Back to the main hub location for some more research and testing. Make some changes, try some different things, and back on the road up north to try it again. This cycle repeats for a while, and after 8 hours of research, informed trial and error, I realized that all of the routers require a firmware upgrade in order for the IPSEC-Passthrough to work properly. So at 1:00 A.M I decide to not tackle this now, and I back out my hardware changes and then plug the OpenBSD systems back in and head home. I had to re-apply the old setup as the offices were open on Saturday.
Saturday afternoon at 4:00 I decide to tackle this project for a second round. I updated each router individually, and then I started my configurations again. By this point in time I had created an installation instructions document for other technically savvy people at the company. This served useful as a checklist for me too. Back and forth between points, but not quite as much as the night before. I figured out the niceness of "remote administration".
To make this story short, 13 hours later, lots of driving, phone calls to Linksys support, phone calls to ISP support, and a large amount of cigarettes later, it all boiled down to the fact that the main VPN hub was connected to an older modem which needed to be reset for the modem to allow the public internet to reach the routers IP address.
The saga continued on Sunday as I set up another location which went smooth as silk. Monday morning I hit another location at 9:00 A.M. on 3 hours sleep before I hit the highway to get to another city to install their router. I sent another technical guy from the company to another city with a pre-configured router. I was crossing my fingers that everything would work properly for him. I t was a simple unplug the OpenBSD computer, plug in the router, count to 20 and everything should work. Luckily it did, as I didn't really feel like trouble shooting new router from 500 clicks away.
So my weekend efforts of over 30 hours. A sleep total of about 15 hours in 4 days. And the network has been removed and completely replaced. I am uber happy with this new solution. The routers are $170, and now anyone can configure them, or administer to them. Linksys provides a very very nice web interface for modifications. They offer several cool features (Full VPN, Configurable logs, Email notifications, port forwarding, MAC address cloning, Firewall configurations). They are a very viable solution as opposed to an OenBSD style situation where each firewall runs ~ $1000 for the computer, and I am the only one around who can handle the technically aspect of them. besides the Linksys routers run embedded Linux. That's super cool!
I'm happy, I'm tired, and I want to smoke some more.
¶ 1:14:00 AM
As any good system administrator knows that when dealing with _anything_ that involves upgrading, changing, installing etc. for a corporate environment, it automatically moves your working hours to outside of the regular business hours. That doesn't mean that you get the day off, it just means you work late. Sometimes you work really late.
I started last Friday afternoon at 4:00 P.M. Sent a staff wide email letting them know about network downtime starting after 5:00. I spent some time reading the manual, and researching the Linksys BEFVP41 VPN routers. I had 7 routers sitting on my desk, and in order to re-work the company's VPN setup, it was an all or nothing change over.
The fun thing about VPN networking is you are generally connecting to different physical locations. The two locations I was working from were about a 30 minute city drive apart. So I went slow and got the internet (NAT) working at the main VPN hub, and packed up a router and hit the road. Got internet working at location #2, but for all of my efforts I could not get the VPN connection to properly work.
Back to the main hub location for some more research and testing. Make some changes, try some different things, and back on the road up north to try it again. This cycle repeats for a while, and after 8 hours of research, informed trial and error, I realized that all of the routers require a firmware upgrade in order for the IPSEC-Passthrough to work properly. So at 1:00 A.M I decide to not tackle this now, and I back out my hardware changes and then plug the OpenBSD systems back in and head home. I had to re-apply the old setup as the offices were open on Saturday.
Saturday afternoon at 4:00 I decide to tackle this project for a second round. I updated each router individually, and then I started my configurations again. By this point in time I had created an installation instructions document for other technically savvy people at the company. This served useful as a checklist for me too. Back and forth between points, but not quite as much as the night before. I figured out the niceness of "remote administration".
To make this story short, 13 hours later, lots of driving, phone calls to Linksys support, phone calls to ISP support, and a large amount of cigarettes later, it all boiled down to the fact that the main VPN hub was connected to an older modem which needed to be reset for the modem to allow the public internet to reach the routers IP address.
The saga continued on Sunday as I set up another location which went smooth as silk. Monday morning I hit another location at 9:00 A.M. on 3 hours sleep before I hit the highway to get to another city to install their router. I sent another technical guy from the company to another city with a pre-configured router. I was crossing my fingers that everything would work properly for him. I t was a simple unplug the OpenBSD computer, plug in the router, count to 20 and everything should work. Luckily it did, as I didn't really feel like trouble shooting new router from 500 clicks away.
So my weekend efforts of over 30 hours. A sleep total of about 15 hours in 4 days. And the network has been removed and completely replaced. I am uber happy with this new solution. The routers are $170, and now anyone can configure them, or administer to them. Linksys provides a very very nice web interface for modifications. They offer several cool features (Full VPN, Configurable logs, Email notifications, port forwarding, MAC address cloning, Firewall configurations). They are a very viable solution as opposed to an OenBSD style situation where each firewall runs ~ $1000 for the computer, and I am the only one around who can handle the technically aspect of them. besides the Linksys routers run embedded Linux. That's super cool!
I'm happy, I'm tired, and I want to smoke some more.
¶ 1:14:00 AM
