Profile cover photo
Profile photo
Phillip Gervasi
About
Phillip's posts

Post has attachment

Post has shared content
Network engineers like redundancy. It’s not that we just want double of everything – we want the networks we design and manage to be super fast, super smart, and super resilient. In the LAN and in the data center we’ve been logically joining network…

I STILL GO TO SCHOOL EVERYDAY, BUT NOW I CALL IT PROFESSIONAL DEVELOPMENT

Few fields require the continual professional development that IT does, but few fields offer the incredible rewards that a commitment to developing the skills of our trade can provide. Many factors come together to shape if, why, and how we advance in our field, and though I can speak only of my own experience, I believe the lessons I’ve learned from my journey so far may be of some value to others also on a similar path. In this post I’m going to explore what I believe professional development is for the network engineer.

In college I focused on literature and history. I developed that in grad school and turned it into a teaching career. If you’ve read my About Me page you’ll know that a teaching career wasn’t really for me. However, I'm convinced that the skills I developed as a teacher and passion for lifelong learning that grew out of those few years have been vital to becoming a well-rounded network engineer. I still go to school every day, but now I call it professional development.

I use the term “professional development” only because it’s ubiquitous in the IT field. We all have a sense of what it means, but I think it’s somewhat of a misnomer. One of the main reasons I got into IT was for the chance to own my skillset apart from having to play company politics or climb the corporate ladder. I did not (and do not) want to be beholden to an organization to grow in my career and my ability to provide for my family. Though I'm grateful for my job and believe we ought to be respectful toward our employer, I don't rely on my company to develop my depth of knowledge, income potential and to an extent my job satisfaction.

Professional development, then, is not really the formal plan your boss gives you during your yearly review. Yes, you can definitely progress that way, but it'll be according to the desires of your employer and at their pace. That may align with your plan, and it may give you a good framework to work with, so I don’t want to dismiss that situation entirely. But in my experience, I’ve never been handed a formal plan. My professional development has been much more personal. My employers knew I was already deep in my own studies. Of course it was convenient that my goals were in line with the direction of the company, but so far it’s worked for me.

I do remember once when an employer gave me a specific directive. A Cisco VAR I worked for wanted me to develop my wireless skills, so they assigned me the task of getting a CCNA and CCNP wireless in one year. They incentivized it, and I earned both within about five months. But understand that this didn’t take away from my plan because I already wanted to round off my CCNP level knowledge in other areas.

I believe that it’s because I already had a personal plan that their directive worked out so well. I reject the idea that I have to wait for my boss to create my career for me. I even reject waiting for my employer to pay for resources before I can make progress.

For a network engineer, a professional development plan is the ongoing endeavor to be a better network engineer. By the very nature of that goal the results will be advancement, more interesting work and more lucrative employment.

I concluded early in my career that I would be a network engineer by trade and not because it was the job I happened to have. That means I would be a network engineer regardless of who I worked for or my precise title. Think about your own career that way. Don’t make it the basis for your identity, of course, but I think it’s right to look at network engineering as our trade: a craft we love, take pride in, and at which we continually improve.

You need to develop your own plan. Where do you want to be in your career in five years? What sort of income do you want? What technology do you want to work with? You need to first take an honest look at the industry and determine what areas of the IT field can accommodate those goals. If you want to earn six figures within five years, then you’re in the right business. If you thrive in an environment with constant technical challenges, the opportunities are there. But if you have no plan and float aimlessly through five years of L1 and L2 helpdesk, you won't get to the next level.

For me, after one year on helpdesk it was time to get serious, so I set a specific goal with a timetable. Then I became relentless in my pursuit to fulfill it. This is when my career actually began. I decided that I wanted to be a network engineer. My goal was the job title and a certain salary. I tackled the CCNA first which took me almost a year. I started with reading a CompTIA Network+ textbook and watching the DVDs that came with it. The content wasn’t difficult - it was basic network concepts which I picked up quickly. When looking at the CCNA content itself I realized a textbook wouldn’t cut it. I hung around with some of the network engineers at my company and learned how to put together a lab. They recommended Pluralsight and CBT Nuggets, so I immediately purchased subscriptions with my own money.

Now I had a goal and resources, but I didn’t have decent study habits yet.  Remember this was a career change for me and I desired to catch up as fast as I could. How would I get a CCNA in one year when all of this was so new to me? The answer was simple in theory but difficult in practice. I was going to have to work hard.

This is something you have to settle within yourself if you want to advance in any serious manner as a network engineer. You must be willing to work hard.

I set up a lab and got up at 4:30am every weekday to study before work. I studied at night after my wife and I put the baby to bed. I studied every moment I could scrounge. Lunch was studying. Early morning was a CBT Nuggets or Pluralsight video. Nighttime was a lab. I still failed both ICND 1 and 2 the first time before earning the CCNA, but when I got it I was able to get a much higher paying helpdesk job almost immediately.

My second job was also at a Cisco VAR, though it was a dramatically smaller company. I was given all the networking tasks because I was the guy with the CCNA and working on his CCNP. This is important to note. This company was so small that I had the opportunity to work on everything. We didn’t work on huge projects, so it was a perfect environment to work on basic networking skills every day. The concepts became so ingrained in me that advancing through CCNP content wasn't too difficult. It required a lot of work to learn and lab, but it was familiar. I still got up early to get to the office by 6:30am every day to study, but now the ball was really rolling.

This is something to consider. A smaller value added reseller is a great place to go from helpdesk work to entry-level network engineering because a small company can't afford experienced high-end network engineers and usually doesn't have the work for them anyway.  By nature of the small business environment you will learn, learn, learn. 

In one year I earned a CCNP, and the company grew enough that I worked on CCNP level projects regularly. After two years my title changed to Senior Network Engineer and I got a nice raise. This was the fulfillment of my first plan, and it took getting up very early, spending some of my own money, turning off Netflix (the hardest part sometimes), and staying up late reading.

Since then I’ve earned other Cisco certifications, and today I have a new goal, a new plan and new resources, but the same strategy. The best part, though, is that it’s mine. I wasn’t handed my professional development plan. I created it for myself, personally.

Follow me on Twitter  @network_phil

SHORT AND SWEET CISCO VSS

Cisco's Virtual Switching System (VSS) is a pretty common technology that joins two physical Cisco Catalyst switches into one logical switch thereby joining the two data planes. It's different from stacking switches because you can use regular ethernet cables rather than stackwise cables and modules. The two technologies are otherwise kind of similar in the benefits they offer.

VSS limits you to only two switches in a cluster (switching system), and is used for the 4500, 6500 and 6800 platforms. This is then a great technology to build chassis redundancy at a large distribution or core level. In my mind one of the biggest benefits is multi-chassis etherchannel which provides both link and hardware fault tolerance for your switch uplinks or servers.

Below is an easy copy/paste config you can grab quick rather than dig through a white paper. VSL stands for "virtual switch link" which is the link for VSS traffic between the two switches. This example is from a pair of 4500X switches I put in not long ago and uses 4 port port-channel for the virtual switch link. There will be a bunch of QoS config that gets auto-generated as part of this brief process, and I'd personally tell anyone who asks that you did that part yourself manually and from memory.



Switch 1

conf t
switch virtual domain 10
switch 1
exit

int port-channel 5
description VSL for VSS
switchport
switch virtual link 1
no shut
exit

int range ten1/1 - 4
description VSL port for VSS
switchport mode trunk
channel-group 5 mode on
end

switch convert mode virtual



Switch 2

conf t
switch virtual domain 10
switch 2
exit

int port-channel 10
description VSL for VSS
switchport
switch virtual link 2
no shut
exit

int range ten1/1 - 4
description VSL port for VSS
switchport mode trunk
channel-group 10 mode on
end

switch convert mode virtual



Verification

sh switch virtual
sh switch virtual role
sh switch virtual link
sh switch virtual link port-channel



Follow me on Twitter  @network_phil

I STILL GO TO SCHOOL EVERY DAY, BUT NOW I CALL IT PROFESSIONAL DEVELOPMENT

Few fields require the continual professional development that IT does, but few fields offer the incredible rewards that a commitment to developing the skills of our trade can provide. Many factors come together to shape if, why, and how we advance in our field, and though I can speak only of my own experience, I believe the lessons I’ve learned from my journey so far may be of some value to others also on a similar path. In this post I’m going to explore what I believe professional development is for the network engineer, and I’ll share some of my experiences from my first few years in the field.

In college I focused on literature and history. I developed that in grad school and turned it into a teaching career. If you’ve read my About Me page you’ll know that a teaching career wasn’t really for me. However, I’m convinced that the skills I developed as a teacher and passion for lifelong learning that grew out of those few years have been vital to becoming a well-rounded network engineer. I still go to school every day, but now I call it professional development.

I use the term “professional development” only because it’s ubiquitous in the IT field. We all have a sense of what it means, but I think it’s somewhat of a misnomer. One of the main reasons I got into IT was for the chance to own my skillset apart from having to play company politics or climb the corporate ladder. I did not (and do not) want to be beholden to an organization to grow in my career and my ability to provide for my family. Though I’m grateful for my job and believe we ought to be respectful toward our employer, I don’t rely on my company to develop my depth of knowledge, income potential and to an extent my job satisfaction.

Professional development, then, is not really the formal plan your boss gives you during your yearly review. Yes, you can definitely progress that way, but it’ll be according to the desires of your employer and at their pace. That may align with your plan, and it may give you a good framework to work with, so I don’t want to dismiss that situation entirely. But in my experience, I’ve never been handed a formal plan. My professional development has been much more personal. My employers knew I was already deep in my own studies. Of course it was convenient that my goals were in line with the direction of the company, but so far it’s worked for me.

I do remember once when an employer gave me a specific directive. A Cisco VAR I worked for wanted me to develop my wireless skills, so they assigned me the task of getting a CCNA and CCNP wireless in one year. They incentivized it, and I earned both within about five months. But understand that this didn’t take away from my plan because I already wanted to round off my CCNP level knowledge in other areas.

I believe that it’s because I already had a personal plan that their directive worked out so well. I reject the idea that I have to wait for my boss to create my career for me. I even reject waiting for my employer to pay for resources before I can make progress.

For a network engineer, a professional development plan is the ongoing endeavor to be a better network engineer. By the very nature of that goal the results will be advancement, more interesting work and more lucrative employment.

I concluded early in my career that I would be a network engineer by trade and not because it was the job I happened to have. That means I would be a network engineer regardless of who I worked for or my precise title. Think about your own career that way. Don’t make it the basis for your identity, of course, but I think it’s right to look at network engineering as our trade: a craft we love, take pride in, and at which we continually improve.

You need to develop your own plan. Where do you want to be in your career in five years? What sort of income do you want? What technology do you want to work with? You need to first take an honest look at the industry and determine what areas of the IT field can accommodate those goals. If you want to earn six figures within five years, then you’re in the right business. If you thrive in an environment with constant technical challenges, the opportunities are there. But if you have no plan and float aimlessly through five years of L1 and L2 helpdesk, you won’t get to the next level.

For me, after one year on helpdesk it was time to get serious, so I set a specific goal with a timetable. Then I became relentless in my pursuit to fulfill it. This is when my career actually began. I decided that I wanted to be a network engineer. My goal was the job title and a certain salary. I tackled the CCNA first which took me almost a year. I started with reading a CompTIA Network+ textbook and watching the DVDs that came with it. The content wasn’t difficult – it was basic network concepts which I picked up quickly. When looking at the CCNA content itself I realized a textbook wouldn’t cut it. I hung around with some of the network engineers at my company and learned how to put together a lab. They recommended Pluralsight and CBT Nuggets, so I immediately purchased subscriptions with my own money.

Now I had a goal and resources, but I didn’t have decent study habits yet.  Remember this was a career change for me and I desired to catch up as fast as I could. How would I get a CCNA in one year when all of this was so new to me? The answer was simple in theory but difficult in practice. I was going to have to work hard.

This is something you have to settle within yourself if you want to advance in any serious manner as a network engineer. You must be willing to work hard.

I set up a lab and got up at 4:30am every weekday to study before work. I studied at night after my wife and I put the baby to bed. I studied every moment I could scrounge. Lunch was studying. Early morning was a CBT Nuggets or Pluralsight video. Nighttime was a lab. I still failed both ICND 1 and 2 the first time before earning the CCNA, but when I got it I was able to get a much higher paying helpdesk job almost immediately.

My second job was also at a Cisco VAR, though it was a dramatically smaller company. I was given all the networking tasks because I was the guy with the CCNA and working on his CCNP. This is important to note. This company was so small that I had the opportunity to work on everything. We didn’t work on huge projects, so it was a perfect environment to work on basic networking skills every day. The concepts became so ingrained in me that advancing through CCNP content wasn’t too difficult. It required a lot of work to learn and lab, but it was familiar. I still got up early to get to the office by 6:30am every day to study, but now the ball was really rolling.

This is something to consider. A smaller value added reseller is a great place to go from helpdesk work to entry-level network engineering because a small company can’t afford experienced high-end network engineers and usually doesn’t have the work for them anyway.  By nature of the small business environment you will learn, learn, learn. 

In one year I earned a CCNP, and the company grew enough that I worked on CCNP level projects regularly. After two years my title changed to Senior Network Engineer and I got a nice raise. This was the fulfillment of my first plan, and it took getting up very early, spending some of my own money, turning off Netflix (the hardest part sometimes), and staying up late reading.

Since then I’ve earned other Cisco certifications, and today I have a new goal, a new plan and new resources, but the same strategy. The best part, though, is that it’s mine. I wasn’t handed my professional development plan. I created it for myself, personally.

Follow me on Twitter  @network_phil

DON’T FORGET THE PLUGS
 
How many times has one of your network projects come to a screeching halt (probably at 2am) because you didn’t have the right power plug or patch connector? Seems like such a trivial thing, but millions of dollars of equipment won’t do much more than look pretty in the racks until it’s all powered up and connected together.

Almost two years ago I was the lead network engineer on a rip and replace project swapping out some Catalyst 6509s for some 6880s, adding in some 4500Xs, replacing old PIXes with a pair of new ASAs, and setting up some Nexus 5K pairs with a line of top-of-rack 2Ks.  After the core cutover there would be a big LAN switch refresh, a Prime/ACS upgrade, and a migration to a new UCS B series. It was a  several million dollar project.

I worked with the storage engineer to get some things straightened out with migrating VMs. I worked out the schedule with my project manager and the customer. I even ran the plan by the most senior network engineer at my company.  Because of some configuration requirements there was no good way to run the old and new core/edge in parallel, so it was going to be a hot cutover. We discussed spanning-tree. We discussed EIGRP convergence. We discussed VMotion (well the storage guy did), and we discussed rollback strategies. Everyone liked the plan just fine, and we moved forward.

The cutover was scheduled for 11pm Friday night (naturally). I started the pre-work at 7pm and touched up my scripts. I made sure the boxes labeled “power cables” and “fiber patch cables” were nearby. I didn’t open them up though.

I think you know where I’m going with this.

I got the initial configs on the gear before it was delivered, and now it was all racked and about half of it booted up. It was maybe 7:45 when I started opening boxes looking for the Twinax cables. Found ’em. Grabbed some aqua patches with LC connectors and started getting the 5548s cabled up.

The customer had the campus core in a room across the building, so now it was time to home these 5548s from this room back to the core using the fiber patch panel….but wait….are those ST connectors? I thought? Weren’t they?  …..oh no……

I had all LC-LC. The patch panel was ST.

Yes, it’s initially the solutions architect’s and PM’s responsibility to have that figured out ahead of time, but the project was handed off to me long before the cutover. I had no one to blame. Hundreds of thousands of dollars of equipment sat disconnected just prior to an overnight cutover because I had the wrong connectors. Lovely. The customer wasn’t thrilled but at least we could get the core switches and firewalls cutover and leave the Nexuses (Nexi?) out until the next change window.  I grabbed the power cables box and headed across the building to the network MDF. The core switches were racked and the firewalls in place. They were still unplugged so I tore open the box I was holding.

My heart sank. All C13 plugs. PDUs were all C14, of course.

That night we got little accomplished. Thankfully getting the plugs and connectors was no big deal, and the actual cutover (rescheduled for a week later) went very well.  After taking the customer out to lunch, all was just fine. I learned a hard lesson. I planned out a clean cutover and expertly configured some advanced networking technologies, but in my haste to be the hero of the day I left out some of the most basic considerations like power and layer 1 connectivity. Today I typically start with those areas and take as many pictures as I can.

I’ve heard it said that an expert is someone who’s made all the possible mistakes that can be made in a particular field. Well, scratch that mistake off the to-do list.

Follow me on Twitter  @network_phil

WHY BUY ONE WHEN YOU CAN BUY TWO FOR TWICE THE PRICE

I get pretty excited when new network gear shows up at the loading dock. I get psyched when I get to configure an interesting technology that I rarely get to use. But considering our responsibility to our customer or employer, sometimes we need to put that aside in favor of the simpler (or cheaper) but more appropriate solution. Let me give you one example.


Not long ago I worked with a nonprofit in the healthcare industry to replace their network core. It was an organization of a little less than 2,000 end users on a single campus. Many people were remote part-time workers and not really information users, though they would log in occasionally to access an online timesheet or email. There was one network egress through an unreliable ISP, so part of this project would be adding a second internet connection to the mix. The main goal was to gain better network fault tolerance.

We looked at hardware and configuration options – they were stretching their budget to the max. Our initial thoughts were pretty standard: probably Catalyst 4500Xs at the core running VSS, a couple 3945 edge routers running HSRP, a failover pair of Cisco ASA 5545Xs and some switch stacks. We even discussed a building generator and moving their email to Office 365.

I was the delivery engineer for this project, not the solutions architect, so I fell out of the loop during the rest of the design process and was brought in just before kickoff. Looking over the final design I saw the 4500X switches, one 3945 router, and a pair of ASA 5585Xs in multicontext mode. They decided to skip the second ISP, increase the bandwidth with their current provider and go with larger ASAs. I had a mild disagreement with the ASA sizing and dropping the ISP, but that wasn’t my decision. Then I realized that the customer, who relied heavily on Anyconnect as their remote access solution, would be up the creek since multicontext mode didn’t support remote access VPN.

I faced a conflict here because there was a disparity between the business requirement and the technical solution.

First let’s consider the elimination of the second ISP from the plan. It was clear that the incredible cost of the bigger ASAs precluded the customer from entertaining a second provider. That made no sense to me because of how unreliable their current provider was and how heavily they relied on RA VPN. I looked at the purchase order and almost fell out of my chair when I saw how much each ASA plus licensing cost. Cisco ASA 5585Xs aren’t cheap, and each security context costs thousands. The business requirement called for local resources to be available by remote workers at all times, so eliminating the second ISP seemed strange to me.

Now let’s consider multicontext mode. (No, I won’t be explaining how to configure it because, well, Google.) Multicontext mode on a pair of Cisco ASAs allows for an Active/Active state so you can load-balance traffic as well as have subsecond failover. You also get multiple security contexts, or in other words, multiple virtual firewalls living in the same box. This is useful for multi-tenancy configurations such as with service providers and large multiple business-unit companies. This customer had no need for that whatsoever, and since multicontext mode does not support remote access VPNs and is extremely expensive I was again confused why this layer of cost and complexity was added.

Also consider the actual hardware.  Failover designs should plan for each individual firewall to be able to accommodate 100% of network traffic in case of a failover scenario. It’s important to choose a model that can alone support all network traffic. In this case each new 5585X was such an incredible overkill that I felt it was a waste of a lot of money. I know sometimes we choose hardware to meet future growth, but this was a very static organization funded mainly by government grants. There was no need to be able to accommodate huge growth within the next 7 – 10 years before the next hardware refresh. Even the lower end 5500X firewalls like the 5515X or 5525X would easily fit the bill. 

An alternative would be running the firewall pair in Active/Standby on smaller (cheaper) firewalls. This configuration also provides subsecond failover, plus it offers all the features of the ASA including remote access VPN. No, there’s no load balancing, but I really don’t think that was relevant in this case. This would have freed up all the money needed for a second router and a second ISP. Even having a second ISP but just a single router would have made sense to me.

I know that multicontext mode on Cisco ASAs has its place –  I’ve used it for several projects. I’ve also installed 5585Xs in data centers and very large networks where the traffic baselines required the larger model. But for this much smaller customer, I don’t believe the huge cost increase was worth it at all. Sticking with an Active/Standby solution would have freed up the money to do the other things necessary to meet the customer’s business requirement.

Lastly, there was no mention of replacing any of their old, daisy-chained access switches. My thought was to put in a couple basic switch stacks with 2960Xs in order to run portchannels to mission critical servers and portchannels back to the core. The 2960Xs are among the cheapest Catalyst switches you can get other than the SMB models, and in my mind they were necessary to add the access layer redundancy the customer asked for.

Though the final design cost much more in hardware than my preliminary design, it didn’t meet the customer’s business requirements. It’s my opinion that though a design governed by margin on hardware can be more lucrative, it certainly isn’t always the best solution for the customer. In the long run, that may even hurt the relationship and therefore the bottom line. I’ve seen enough one AP per classroom designs that I already had a tendency to look at final purchase orders with a bit of cynicism.

I love cool new network equipment. I really do. But consider why you choose a technology. If it isn’t truly the best solution to meet the customer’s need, then it probably isn’t the right solution.

Follow me on Twitter @network_phil
Wait while more posts are being loaded