What a trip this last few weeks have been. I have mentioned previously that I have been busy with some cool implementation projects at work. Specifically, I have been tasked to configure our PIX appliance to accept remote VPN client requests. This is a very interesting and fun project for me because I have never done any of these before. I have never even been inside a pix OS nor even seen one in my IT career. I have mentioned before that aside from the few good years where I maintained and implemented a Windows Active Directory infrastructure at my old job, most of my career was relegated to doing menial help desk support - something I’ve made a decision to change. And nine months after a made that decision, I’m finally seeing that change.
Last week I was able to finally see my work bear some fruits - in a matter of saying. I now have remote users from our company hitting our pix and able to access local resources in our corporate office (thanks to Barry of bitbucketblog.com, in part). There’s still a lot of work I need to do to clean up my configurations but seeing my implementation actually working is a big boost on my confidence.
Some of the things I need to clean up for sure is the routing. Everything so far is static (which is fine for our purposes since we don’t have a lot of routers or sites that need dynamic routing.) But it would be nice to have OSPF running later. Also, right now, the users authenticate against a local username/password on the pix appliance. Ideally, we would like them to authenticate on a Windows RADIUS server.
Despite all that, though, I already learned a ton of things. Some things I’ve never used before but now understand a little better:
- What IPsec is all about
- Configure ISAKMP parameters
- Configure IPsec parameters
- Crypto maps
- Dynamic crypto maps
- NAT
- NAT-T
- Split-tunnels
- Better understanding of IP access-lists
- Reverse Route Injection
- a few more that I probably am not remembering
Now I still can’t say that I understand them well. But at least I have a better idea of what these things are all about. And with time and experience, I can develop a more solid understanding of them. In fact, learning how to do the step by step configuration was pretty easy. The real challenge is to really understand everything behind all the commands I was typing in. And for the most part, I took particular attention to what I was asked to type in by the Cisco documentations. I’ve downloaded and printed out thousands of pages of Cisco docs to peruse to better understand what I was doing. I’ve spent late nights and weekends reading for hours on every configuration command that was asked of me. Needless to say, my brain is packed with information that I’m sure I will forget 75% of. But that’s ok. There’s absolutely no doubt in my mind that I’ve learned something valuable. In fact, I printed out the running configs on my routers and pixs and I can honestly say that I can read them in a whole new light because of my new understanding. And that my friends is pretty exciting. 
Oh by the way, last Thursday, prompted by a renewed confidence in me, my boss asked me if I was up to tearing down our old router-to-router gre tunnels to our remote sites and configure a multiple router-to-pix ipsec vpn tunnels to replace the old one. Not wanting to miss out on the opportunity I immediately said, “hells yeah!” Much of the initial configuration was very similar to the client configurations so I thought I can fumble my way around it. It turns out that my boss’s confidence in me was a little bit pre-mature because I failed miserably. In fact, I think he might have gotten a little annoyed in me for being so confident that I could do it. He told me at first that if I wasn’t comfortable, that I should tell him right then and there when he asked me. I wanted to do it so bad, partly to get the “hands-on” and partly to show him initiative and that I can do it. But it proved to be a little bit over-whelming as I worked on it from 8am to 9pm almost non-stop that day only to end up breaking things. In the end my boss told me to go home and no to touch the routers any further. A little bit dejected and hit with a little dose of you-are-way-in-over-your-head reality I went home and cracked open a thick binder of documentations I printed from work and dug in through the steps and looked for what I was doing wrong.
The next day the boss ( a former CCIE, but years separated from hard core IOS hands on) was in his office with his room door shut working away at fixing some of the configs I broke. That whole day sure felt very long and uncomfortable and I knew my boss was not particularly happy because he was short with me when I ask him questions. So I just sat in my corner and used every opportunity to continue researching on what I did wrong. I was just resigned to let things be with an almost nonchalant “oh well” attitude. By the end of the day, my boss has not succeeded in getting the configuration running and the deadline to get the tunnel up was at the end of that day because the primary Internet circuit that the current tunnel is running on is about to get turned down at the end of business day. To make things worse he had to leave early that day. So, faced with frustration of the whole day, my boss turned to me again and told me to look through his configuration because he has been looking at it all day and tunnel vision (pun) has impaired his brains that he is having a hard time spotting little mistakes that he might have made but otherwise could not spot. He told me what to look for and I started looking at the configurations line by line. Much to my surprise, or non-surprise, most of the configuration he put in there were very similar to what I had initially configured. In fact they were pretty much the same ones minus a few changes (e.g. where I configured a 3des, he put in a des or where i put in an md5 hash, he substituted a sha). I even spotted an acl that he configured that I thought was not right.
And so thinking that nothing else could possibly go more insane, I cleared all his configurations - with his approval, of course (or so I interpreted something he said as approval ;)). And with the notes that I jotted down from the night before and all through the day, I rebuilt the configuration… And what do you know! A few hours of careful and meticulous reconfiguration, I finally got one tunnel up and endpoints talking to each other. In the process of him fixing my mess, he also broke the client vpn configurations I made earlier that week. But I was also able to reconfigure it back to its proper working order. I tested all the routing and ping and traceroute outputs were flying back and forth. I felt vindicated. Actually, I wanted to say out loud in a sinister tone, “vengeance is mine!” but that didn’t feel quite right. After that, configuring all the other routers were cake.
Now I can’t claim that I’m smarter than my boss or anything. Because 999 times out of 1000, he will out-configure me. He is also 100 times smarter than me. But I can’t say that I got lucky either, because this has nothing to do with luck. It’s either configured correctly or not. Maybe it was more of him being unlucky just for that day that allowed me to out-do his work
Looking back, I don’t know what it is that I did wrong the first time around that I didn’t do this time that made it work or vice versa. The irony is that, I found what was wrong with his configuration which I worked to resolve. I guess if he hadn’t changed the configs that allowed me to see something that didn’t look right, I wouldn’t have had the werewithal to change it again for fear of breaking anything further. Change is good.
In the end, I have a pix authenticating remote vpn clients and three remote sites configured with router-to-pix tunnel up and running. And all that was done on a production network by an (almost)engineer with nearly no experience or business being on a router. In any other environment, I might not have had this opportunity. But one thing is for sure, whether the opportunity is there or not, I learned that you must always be prepared and constantly train yourself by reading, asking, testing, tinkering, labbing, etc. Because when real opportunity comes, you’d have already armed yourself with the ability to say “yes” to that opportunity, even though you might not feel entirely ready.
