Toward the bottom are several books that I’ve read much more recently, including books about network automation and machine learning in networking. Personally, for anyone seeking to get into network engineering or to become a more well-rounded networking professional, I recommend topics around automation and ML as well.

I’m sure you can think of many books that have been foundational for you that aren’t on this list, so please feel free to drop a book title in the comments or on social media.

1. Routing TCP/IP, Volume 1, 2nd Edition, by Jeff Doyle and Jennifer Carroll

2. Routing TCP/IP, Volume 2, 2nd Edition, by Jeff Doyle

3. Optimal Routing Design (Networking Technology) 1st Edition, by Russ White, Alvaro Retana, Don Slice

4. Practical Packet Analysis, Third Edition, by Chris Sanders

5. Machine Learning for Network and Cloud Engineers: Get ready for the next Era of Network Automation, by Javier Antich

6. Network Warrior: Everything You Need to Know That Wasn’t on the CCNA Exam 2nd Edition, by Gary Donahue

7. Computer Networking: A Top-Down Approach, by James Kurose and Keith Ross

8. Computer Networks, by Andrew S. Tanenbaum and David J. Wetherall

9. Network Programmability and Automation: Skills for the Next-Generation Network Engineer 2nd Edition, by Matt Oswalt, Christian Adell, Scott S. Lowe, Jason Edelman

10. Mastering Python Networking 4th Edition, by Eric Chou

However, it never ceases to amaze me that some the most important skills and technical knowledge I’ve gained over the years is understanding how to calculate a power budget for an IDF, the difference between an L620P and 620P cord, the difference between various types of fiber, and remembering to ask my customer about the direction of airflow in their data center.

When designing real-world networks, it really is the overall picture we have to keep in mind. Speeds and feeds, bits and bytes, and all the syntax in the world isn’t enough to properly design a real-work network that you can actually power on and plug into.

Thanks,

Phil

That’s the question I posed on Twitter recently, and it’s a question I ask myself now and then when I consider my own career trajectory. Pre-sales engineering is the sales part of engineering in which a technology professional works with customers, account managers, and project managers to discover customers’ needs, design solutions, build quotes, determine scopes of work, and create bills of materials. Sometimes these folks are called Sales Engineers, sometimes Solutions Architects.

Every company I’ve interacted with uses these titles somewhat differently, and I don’t think there’s a true consensus on  the nature of the role. Sure, you can look up the actual definition if you like, but survey a bunch of companies and individual engineers and you’ll get a variety of responses.

In my estimation, someone meeting with a customer to learn their business needs and how they relate to technology doesn’t necessarily have to have a deep knowledge of every routing protocol and every piece of hardware out there let alone how to implement them. Discovering an organization’s business needs is a skill in and of itself, and in that sense doesn’t need someone to be able to configure a router or firewall.

But those business requirements eventually translate into technology requirements. When a pre-sales engineer determines that a particular technology will fulfill that requirement, I have to assume they know something about it.

Account Managers spearhead the initiative to discover the customer’s needs and begin the process of building trust. But the real trust is built on the [successful] delivery of the solution. The pre-sales folks start this process by instilling some semblance of confidence by explaining and recommending some technology, but it’s the actual delivery activities that make that real.

If the design doesn’t reflect an intimate knowledge of how this stuff works in real life, the project is set up for failure. The trust which was just starting to develop will be lost, the relationship strained, the business opportunity squandered.

Deep familiarity with the technology, including the gotchas, caveats, and intricacies deployment engineers live and breath, is what really produces a solid design and implementation plan. What sets good pre-sales engineers apart isn’t their charming smiles and ability to squeeze just a little more margin on every deal, but the ability to meet the customer’s business need with technology in the context of how it works in real life.

A pre-sales engineer with little to no deployment experience is lacking in their ability to deliver a top quality design. I don’t mean that a pre-sales person with a few certifications and no delivery experience can’t churn out Visio diagrams based on the last 154 projects the company did. In fact, that may work out just fine in certain market segments. But that’s not solutions architecture, and it’s a disservice to most customers.

I know a lot of deployment engineers who see pre-sales as the next step in their careers after tiring of 2am cutovers and what seems like an eternity of windshield time. I don’t disagree with this, because the experienced deployment engineer who also has the ability to see the big picture and interact effectively with customers is the linchpin in a successful project.

This doesn’t mean that a solid pre-sales person needs to be straddling the fence between pre and post sales. The really good pre-sales folks I interact with are frequently in the lab experimenting with a design or keeping up with the latest technology. Even if they’re unable to actually implement the solution, they’re staying close to how the technology works in the real world, and that gives them the edge to go from Senior Visio Maker to Senior Solutions Architect, in the truest sense of the title.

Does a pre-sales engineer need to have strong implementation experience to be good at their job? I think so. Even if it’s from labbing potential designs, the best pre-sales engineers I know actually know how this stuff works in the real world.

First, the underlying hardware utilizes more generic components which, in theory, reduce the cost of the switch. This is as opposed to vendors that use proprietary chipsets sometimes claiming they are able to offer unique features as a result of their technology.

Second, decoupling hardware from the network operating system provides greater ability to centralize management and use programmatic tools such as Chef and Ansible. It also opens the door for a vendor-neutral approach to networking utilizing OpenStack which can provide vendor interoperability and manage network hardware as pools of resources.

Third, because these normally Linux-based network operating systems don’t rely on (or are minimally reliant on) the underlying hardware, they can be completely customized.

In recent years, whitebox switching has begun to trickle down from webscale-sized companies to large enterprise data centers that need some of the same features normally needed by only the biggest infrastructures in the world. Requirements such as network functions virtualization and a compostable infrastructure were common only in those huge organizations but are now being implemented in much smaller environments.

I’m interested in how this is relevant to the non-webscale enterprise. It’s still debatable whether or not there is much of a cost savings with whitebox switches at all, and even pretty good sized enterprise organizations do just fine with their favorite vendor switches. The campus access layer, for example, requires very few advanced Layer 3 features and tends to be somewhat static with regard to configuration changes. Rarely do changes to the access layer require any sort of orchestration or the deployment of end-to-end services.

If the compelling reasons to move to a whitebox model are cost savings, increased programmability, and customizable operating systems, I wonder what the actual benefit is changing an access layer from a single or only several models from the one vendor to a whitebox solution.

In my experience, access switches are among the cheapest network devices even from the biggest network vendors such as Cisco. Also, an access layer is typically comprised of many devices all doing the same thing which means it’s easy to get a deep discount when buying in bulk. Without comparing a Cisco and whitebox BoM, it’s hard for me to definitively say one option is cheaper than the other, but I suspect buying a cheap Dell switch and a separate NOS license will not give me huge cost savings.

Network programmability is certainly an advantage for busy network administrators, but the access layer is normally static enough to make programmability interesting but not necessarily a reason to overhaul an entire switching environment.

Typically, access switches aren’t replaced until they get very old and not when new features are released. I understand that there are exceptions like supporting 802.1x at the port level, for example, but normally these kinds of feature additions are made through iterative code upgrades. The access layer just doesn’t do that much to require significant customization and the frequent addition of new features. I know we can always think of exceptions, but keep in mind that I’m speaking of a typical mid-sized enterprise access layer, not a webscale company’s data center.

To me, the access layer is already pretty much a compostable part of the infrastructure. How many models do you have in your access layer? I can think of maybe three, and they’re all simple, cheap, and easy to swap out.

Even though I don’t see whitebox switching as extremely relevant to an enterprise access layer, I still really like the idea of centralized management and increased programmability so long as the cost is a wash. What I’m doing is taking steps to deploy just a few switches in the corner of the network such as a network closet servicing only a few endpoints. This way I can see first hand the real benefits or drawbacks of bothering with this technology in this often forgotten part of the network.

IP Infusion presented at Networking Field Day 15 introducing themselves as the whitebox NOS we’ve all used but never heard of. ZebOS, in particular, has been used by by OEM manufacturers for almost 20 years on hardware from vendors such as NEC, Brocade, F5, Riverbed, and quite a few others. They started in data centers but also provide solutions for service providers with the goal to provide the very best network OS for whitebox and NFV.

Their expertise and experience is in carrier grade networks, but in the last few years they’ve seen the shift to software at every level of the network. They developed a complete finished product specifically for end-users in the enterprise, OcNOS, and it’s NFV companion, VirNOS, both based on ZebOS.

Their NOS supports most layer 2 and Layer 3 features and is based on a modular design meaning customers can purchase and use only the features they need such as simple switching or advanced routing functions. ZebOS also provides data plane integration by supporting a variety of chipsets in order to create a hardware abstraction layer.

Hardware abstraction means that protocols have no dependency on the silicon itself. The hardware abstraction layer has dependencies on SDKs such as the Broadcom SDKs, but the protocol modules that sit on op of the hardware abstraction layer don’t have dependencies on the underlying hardware.

This opens up to us a new world of centralized network programmability for the access layer using REST APIs and NETCONF while also supporting traditional management methods such as the CLI and SNMP.

If the cost of licensing OcNOS and a small pile of Dell or Edge-Core switches is similar to what you would spend on a closet switch refresh, this is a great way to PoC whitebox switching at the access layer and provide the hands-on experience to inform us whether or it’s relevant to this part of the network just yet.

I’m not convinced that it’s time to schedule a hardware refresh of an enterprise’s access layer and move to whitebox switching, but I really like the idea of better centralized management and network programmability. I’m interested in testing this out in one small part of the access layer while keeping in mind that there’s nothing wrong with changing my opinion later on.

Disclaimer: Gelstalt IT, the organizers of Networking Field Day, provides travel and expenses for me to attend Networking Field Day. I do not receive cash compensation as a delegate. Also, I do not receive compensation for writing about or promoting Networking Field Day.

Recently I had a conversation with my friend, Jordan Martin, about how we’d love to see these technical conversations brought into the public square. This is something we’ve both been thinking about lately, so the timing was right to start something new.

It started a few weeks ago as I was following a group chat when one of these great discussions unfolded before my very eyes like some bit of high theater. These were some of the best engineers I know, and they spoke their lines with wit and technical eloquence.

As I followed the group chat, I felt like I was sitting in a networking bootcamp, and I made the comment that I wished others could experience it as well. Jordan mentioned he wanted to start a project to bring these discussions to the public square, so this was music to my ears.

A few minutes later Eyvonne Sharp added her voice to the growing triumvirate of nerdness. She had some deep technical topics she wanted to explore as well as some serious thoughts on the networking profession as a whole.

Within five minutes of Jordan mentioning he wanted to start something new, we were putting the pieces together. Later that day, we had Network Collective.

Since we were already on the same page, the idea developed quickly. We wanted something organic and community-driven. I don’t mean organic in the sense that we wanted to wing it every time we hit the record button, but in the sense that we wanted to have engineers discuss technical topics without a net. Jordan suggested we go with a video-first format, and the roundtable idea was floated to maximize community involvement.

We believe a video roundtable format will, by virtue of having a group of participants, facilitate a dynamic discussion and the opportunity for ideas to be explored with a variety of perspectives.

If you haven’t seen the introductory video Jordan published a few days ago, please view it here. Our goal is to get all levels of network engineers working in the trenches to come together and share war stories, discuss design ideas, and debate technical solutions.

We’re not really interested in discussing new vendor products or industry news because there are already some amazing podcasts that do a fantastic job with that. Instead, we want to focus on small group technical discussions.

I’m excited. It’s a group of five or six people which means the conversation is less predictable. It’s video which means everyone will see me fumbling with my notes as I pretend I know what I’m talking about. It’s live which means there’s no safety net.

Who knows what’s coming down the road. For now, though, we’re full speed ahead with bringing to the public forum a community-driven, video roundtable podcast with a technical focus by engineers and for engineers.

Please visit thenetworkcollective.com if you’re interested in participating, and please check out our first live stream on Tuesday, April 11 at 7pm EDT. We’re going to try to publish something every couple weeks or so starting with next Tuesday when we’ll be discussing our best networking blunders and the lessons we learned from them.

Thanks,

Phil

TELoIP is a little different from most other SD-WAN vendors in that it offers a managed WAN solution and has its own cloud infrastructure into which customers connect. This isn’t a common approach among SD-WAN vendors, but it also isn’t exclusive to TELoIP. That’s why I’m really interested in knowing how they differ specifically from vendors using a similar approach but also how they’re better than the broader scope of SD-WAN providers.

I’m not going to explain what SD-WAN is other that to say that it’s basically an overlay on top of your WAN to allow for intelligent path selection based on link quality and other traffic engineering such as per packet flow aggregation. This is powerful technology that’s been running in production environments for several years at least at this point.

That means that my questions aren’t really about how SD-WAN works or what their secret sauce does to make the magic happen. Instead, I want to know how TELoIP treats SD-WAN as a separate function from WAN optimization and how that plays into security concerns.

It also means that I want to learn about their service delivery and how TELoIP works with providers of their solution to deliver the last mile between the TELoIP PoPs and the customer edge.

The SD-WAN component isn’t groundbreaking anymore. To me, the technology lost it’s sheen already. That doesn’t mean I’m not interested in SD-WAN or don’t recognize the benefits of the technology, it just means that I’m more interested now in service delivery and operational aspects.

I still have some technical questions in mind, but I’m looking forward to learning how TELoIP is positioning itself and why. Check out their website and their presentation at Networking Field Day 15 by tuning into Tech Field Day’s YouTube channel. Also make sure to follow the discussion on Twitter using #NFD15.

In this post I’m going to look at how we can solve a WAN failover scenario when we have a default route learned from both of our service providers and a reachability problem via our primary ISP.

In this scenario, ISP-A has been having internal problems several hops into their cloud from our edge. This means that though the BGP session with our primary ISP never drops and therefore no failover action occurs, routing to the internet still breaks.

We need to configure ISP failover using only what we have. We aren’t able to take a full feed from our ISPs, and our solution can’t require manual intervention.

For this scenario, we have the following limitations:

1. We are using a single router connected to two ISPs for internet connectivity
2. ISP-A must be primary
3. ISP-B must take over if ISP-A fails even if the failure is several hops upstream
4. Traffic flow must revert back to ISP-A with no manual intervention
5. Both ISPs are advertising a default route which cannot be changed
6. Failover to ISP-B and fail-back to ISP-A should each take as little time as possible

For our topology, we have the following considerations:

1. INTERNET simulates the public internet beyond our ISP’s PE devices. It advertises several loopback addresses via BGP to ISP-A and ISP-B
2. Both ISPs peer with COMPANY_ROUTER, and both advertise only a default route to us
3. LAN represents our inside network and has a static route to the internet via COMPANY_ROUTER which has a static route for the inside networks behind LAN

This is our topology:

Since we’re learning the same default route from both providers, the first thing we need to do on COMPANY_ROUTER is make sure we prefer ISP-A. The default local preference in BGP is 100, and since higher values are preferred, all we need to do is make sure the local preference for routes learned from ISP-B is lower.

route-map CHANGE_LOCAL_PREF permit 10
 set local-preference 80

Apply this route-map inbound on the neighbor statement for ISP-B.

router bgp 100
 neighbor 10.10.13.2 route-map CHANGE_LOCAL_PREF in

Now we need to prepend additional autonomous systems to our advertisement to ISP-B so that return traffic from the internet prefers the path to us through ISP-A.

route-map PREPEND_AS permit 10
 set as-path prepend 100 100 100

Make sure to apply this route-map outbound on the neighbor statement for ISP-B.

router bgp 100
 neighbor 10.10.13.2 route-map PREPEND_AS out

This is enough configuration to prefer ISP-A and to have failover to ISP-B if the BGP session with ISP-A fails. But this kind of failover is triggered by the loss of the BGP session which would be caused by our router and the ISP router not being able to communicate with each other for any reason.

Our situation is a little different, though. When ISP-A has problems several hops away from us into their cloud, we don’t lose our session with ISP-A’s router. COMPANY_ROUTER and ISP-A  are both up and communicating, so despite there being a routing problem just beyond ISP-A’s PE router, we keep it as the next hop for our default route. As far as COMPANY_ROUTER is concerned, nothing is wrong.

There are a few ways to accommodate this type of failure, but remember that there can be no manual intervention and we can’t ask our ISPs for a full feed.

One method that worked in my lab setup was to tie a track object to an EEM script in order to shutdown the neighbor and force the session to drop. My first thought was to shut down the interface itself (which would work), but I couldn’t think of the logic to bring the interface back up automatically. Shutting down the neighbor keeps the interface up and allows me to bring back the neighbor automatically.

We start by configuring the IP SLA and track object. For this lab I’m pinging one of the loopbacks on INTERNET, but in production you can ping the last hop in your provider’s cloud before it goes to the next ISP or something reliable on the internet that you trust. Notice that my source is the interface on COMPANY_ROUTER connected to ISP-A.

ip sla 1
 icmp-echo 2.2.2.2 source-ip 10.10.12.1
 frequency 5
ip sla schedule 1 life forever start-time now
ip sla 10
 icmp-echo 2.2.2.2 source-ip 10.10.12.1
 frequency 5
ip sla schedule 10 start-time now

Now we configure the script to shut down the peer.

event manager applet SHUT_ISP-A_PEER
 event track 1 state down
 action 0 cli command "enable"
 action 1 cli command "conf t"
 action 2 cli command "router bgp 100"
 action 3 cli command "neighbor 10.10.12.2 remote-as 200 shutdown"
 action 4 cli command "end"

This EEM script tracks reachability to 2.2.2.2 through ISP-A. If reachability is lost because of an upstream ISP issue, the script runs through the commands above to shut down the neighbor forcing a failover to ISP-B.

To revert back to ISP-A, the inverse action is required in a second EEM script.

event manager applet RE_ENABLE_ISP-A_PEER
 event track 1 state up
 action 0 cli command "enable"
 action 1 cli command "conf t"
 action 2 cli command "router bgp 100"
 action 3 cli command "no neighbor 10.10.12.2 shutdown"
 action 4 cli command "end"

You can also add a log message as a fifth action, but for this example I’ve kept things very simple.

When we simulate the upstream outage by shutting down interface gig 0/0 on ISP-A (the far end from us), the track object goes to a down state triggering the script and failing over our default route to ISP-B.

But we’re not done just yet. After we fail to ISP-B, the pings to our tracked IP address will succeed again causing us to revert back to ISP-A. To make sure we’re checking for reachability through ISP-A’s local cloud, we can use a simple static route to our tracked IP address with the next hop of ISP-A.

ip route 2.2.2.2 255.255.255.255 10.10.12.2

Now, even when we’ve failed over to ISP-B, COMPANY_ROUTER will still be checking for reachability through ISP-A.

When the issues with upstream routing are resolved, the pings to the track object through ISP-A will succeed triggering the second EEM script to revert to ISP-A.

In order to speed up slow convergence after the failover you can adjust your BGP timers on a per-neighbor basis or globally on the BGP process. This is based on your needs and your overall topology.

Since this is a very simple topology using only a default from each ISP, I reduced the timers global timers a lot. How much is too much is debatable. It’s something to experiment with and possibly talk to your ISPs about.  Just remember that applying new timers requires a hard reset of the sessions causing a brief outage.

A better way to do this would be to take a full feed from our ISPs and configure BGP accordingly, but in this case that wasn’t an option. If you have other ideas of how we could have solved this scenario, please let me know in the comments or tweet them out. One thing I’ve learned over the years is that there’s often more than one right way to do something!

Sure, that’s kind of funny for a quick tweet or in the greater context of a blog post on change control, but for me, I’m not sitting at my desk shaking my head right now. Instead, I’m reminded about the times I did the exact same thing (on a much smaller scale) and will probably do it again.

Here are a few of my most memorable:

• Remotely shutting down the only WAN interface on an edge router in another country.
• Accidentally flipping the second and third octects in a script for modifying production site-to-site VPNs.
• Applying a service policy to the wrong interface and killing all phone calls.
• Forgetting to disable the reload in command after a successful change.
• I even destroyed several physical ports in a core switch by accidentally ripping out cables while trying to make the rack, which wasn’t bolted to the floor, line up better with the floor tiles.

I’m not really thinking about the S3 incident itself, and I’m not too concerned with how a manager should handle this type of situation with someone on their team. I don’t know that much about either of those things. What does come to mind is that I’ve been there to one extent or another more times than I care to admit. In the context in which I work, that could just as easily have happened to me.

I’ve never taken down huge swaths of the internet or services for thousands of customers, but I have had those moments in which I went from confidently hacking away at the CLI to staring wide-eyed at my screen in a cold sweat and with a sinking feeling in my stomach.

I guess this is just part and parcel of the human interaction we still have with our infrastructures and why we have change control. In any case, I’m not judging the engineer who took down S3 accidentally. I’ve been there in my own small way, and I’m sure we all have to one extent or another.

Feel free to leave some of your most memorable blunders in the comments or tweet them out.

Click on the network-y activity to listen, and make sure to have your sound at a decent volume. Most of these tasks take longer than the length of one typical song, so usually I’m listening to the entire album.

Keep in mind that these links open a YouTube window. Feel free to refute, agree, or suggest alternatives in the comments.

1. Hacking out ACLs for an ASA cutover

2. Racking a pair of massive data center core switches

3. Mission Critical Site-to-Site VPN Troubleshooting

4. The moment you’re in the middle of CLI drudgery and you begin to sense the entire 2am cutover starting to go south

5. Rebuiling a Call Manager after hours from an old backup

6. Mounting a couple hundred access points

7. Bouncing data center core switches back and forth during a code upgrade

8. Configuring routing and seeing adjacencies coming up perfectly on every router

9. Spending the afternoon placing hundreds of access points in Prime

10. Troubleshooting some gear while suffering from the flu and sitting in the cold part of a data center

11. Driving home in the middle of the night in the summer with the windows down and moonroof open after a gloriously perfect cutover 

12. Putting the finishing touches on an AMAZING IDF re-cable

13. Working on documentation

14. Driving home in the middle of the night tired and hungry after only a meh cutover

15. Cranking out emails