Several of these tweets made light of the fact that network automation is merely deploying some scripts to push configuration to a small (or large) number of network devices. Therefore, a misplaced comma or some other minor typo in the code would devastate the entire production network. This would be my conclusion as well if the premise was completely true, but I don’t believe it is.

I’m sure there are some network engineers out there that are quick to write a sloppy script and push config into production without proper review, testing and validation, but that’s not at all part of the network automation movement as I see it. NetDevOps applies the DevOps paradigm to network management which means there is great value placed on continuous improvement, testing and validation, collaboration and community. Yes, there is also the no-blame concept which may decrease the level of accountability for any single network engineer, but that’s more than matched by the emphasis on peer-review and built-in client-side mechanisms to prevent the network from blowing up. Therefore, the original argument that network automation is too dangerous battles against a straw man that has little to do with what’s really going on in the network automation trend.

Next, to say that network automation is clever but too risky to use in production presupposes that managing network devices via SNMP or directly on the command line one device at a time poses much less risk. Trends in network automation seek to decrease human error by eliminating human interaction with individual boxes. Assuming that proper measures were taken to test and validate configuration, this means that proper use of network automation techniques would actually decrease the likelihood of error, not increase it. For example, copying and pasting 10 lines of config into a router has no built in validation or rollback mechanism if the configuration doesn’t work right. Some lines of configuration would apply, though, resulting in an ugly effort to figure out what commands did take and then go back to delete them one by one. This is in contrast to pushing configuration objects to an open NX-OS device. If there was some problem with any part of the configuration, the rollback-on-error option would automatically rollback the entire configuration if a mistake was found.

I also understand that part of the concern is the awkward balance between efficiency, i.e. managing many network devices at once, and keeping the failure domain as small as possible, i.e. managing network devices one box at time. This tension makes sense to me because I’ve been in the dreaded network down scenario enough times that I know to avoid it at all costs. However, the cause of most network outages I’ve witnessed was human error at the command line. Therefore, decreasing the potential for this is a step in the right direction, not a move toward nuclear annihilation of the network.

I know that I’m making a small pile of assumptions as well. I assume that the network engineer utilizing Python, Ansible, open APIs and the variety of other means opening up to us is using them skillfully. A sloppy script will take down a network, but a careless application of the shutdown command or VTP revision number will cause significant devastation as well.

Therefore, it’s not the methods and techniques that are flawed, necessarily, but the network engineer. We make mistakes, and we need to find ways to mitigate those mistakes. Network automation absolutely does this, but just like any other professional endeavor, it requires skill to utilize effectively.

Also, there is a sentiment that network engineers’ jobs will disappear or change so much that their skills will not be very relevant. I see this assumption all over the place, and maybe addressing it should be its own post, but for now I’d like to address it simply by saying that a network engineer will always need to know  what a VLAN is and what the K values are in EIGRP. Network engineers will always need to have a deep understanding of networking concepts because of the simple fact that they are managing networks. Yes, some elements of configuration may be abstracted; nevertheless, the serious and professional network engineer will still need to know the deep aspects of networking in order to make the magic happen.

Network automation is another tool in the network engineer’s toolbox. 

Picture a veteran astronomer working late one evening, PhD diploma on the wall, coffee at the ready, and dual monitors flickering with incoming data from an exciting new object he’s discovered in the brilliant night sky. Quickly he and his graduate student assistants put together a Python script to analyze the data in order to deepen their understanding of this exciting new phenomenon in the sea of space.

When the astronomer turned aside to write this script, was he no longer an astronomer? Did the usefulness of using Python to analyze and manipulate awesome space data immediately negate any need for a serious understanding of astronomy? Of course not!

In this case, Python is just a tool in the scientist’s toolbox for being a more effective astronomer. In the same way, using the network automation techniques developed and promulgated by the open source community and vendors alike does not presuppose the network engineer is no longer relevant. In fact, network automation is simply another tool in the network engineer’s toolbox allowing him or her to be just that much better of an engineer.

I’m pretty new to this brave new world, and my passion is still certainly in traditional networking, but I can say confidently that no one is trying to re-invent the networking wheel. The industry is just trying to make the ride smoother.

Ambitious network engineers thrive on facing challenges and learning new technology. They love to learn, they get excited at the latest and greatest technology, and they get bored easily. And in my experience, the most ambitious network engineers are constantly looking for opportunities to be part of bigger, more complex and more interesting projects.

Recently I had a discussion with some friends about our early days when we were in junior positions racking and stacking hundreds of simple access switches for hardware refresh projects. To many veteran network engineers, this type of project is something we hand off to the junior folks or at least manage from afar while we work on fancier things. A switch refresh is definitely not at the top of the list of cool projects hungry network engineers aspire to work on.

The reality is there’s nothing trivial about a large-scale access switch hardware refresh. Some of the technical components may be simpler to configure on a switch by switch basis, and being successful in an access switch refresh project generally doesn’t require an expert level understanding of networking. However, planning to swap out the entire point of entry to a network for end-users certainly requires incredible planning, deliberate patience, stubborn dedication, an iron stomach and a strong understanding of layer 2 troubleshooting.

Network automation methods can make this much less painful, but the actual design and implementation still requires a certain level of understanding both to deploy and to troubleshoot. It’s the latter case, troubleshooting, where I think junior and senior engineers alike prove their worth, and a switch refresh at a reasonably sized enterprise is no exception.

Coordinating with end-users, understanding the paths IDFs take to distribution and core switches, and how spanning-tree can blow everything up is all serious business. Though no one is configuring global DMVPN topologies with 13 different underlay and overlay networks, royally messing up a switch refresh project takes all users offline completely.

I’ve had the opportunity to work on all sorts of networks from the SMB space to large school districts to large enterprises to one of the largest global enterprise networks in the entire world. To be honest, though, the absolute most fun and fondest memories I’ve ever had as a network engineer was working on large-scale access switch refreshes with like-minded coworkers who took great pride in their cable management and switch/patch panel layout design. Cutting over IDFs at 11pm may not sound as glamorous as a fancy data center project for SuperCoolCompany, but man alive was it satisfying to finish off a closet and compare before and after pictures. And wow was it fulfilling to figure out how to fix a loop we accidentally created ourselves plugging in one stupid switch in the wrong place.

I will likely never do those types of projects again, but I will never for even a second believe they are trivial, menial or beneath me. Getting the job done, whether blog-worthy or not, always gave me a deep sense of accomplishment in my work. Besides, it’s always the junior engineers cutting over IDFs in the middle of the night that get to expense all the pizza they want.

Amidst discussion of SDN ambiguity at Networking Field Day 11, SD-WAN stood apart as something we understood, and I think that’s because we understood its use case very well.

Most of us in the room during Silver Peak’s presentation were network engineers building high quality, fault tolerant networks, and we were familiar with the pain SD-WAN seeks to address.

As a brief overview, Software Defined WAN reduces the complexity and improves the performance of the WAN edge by adding optimization, automation and measurement-based exterior routing.  This approach offers fault tolerance for all types of failures from the link itself through the public cloud. This means that the complexity associated with current technologies such as DMVPN, IP SLA, and PfR is largely eliminated.  This also means that a type of QoS can be delivered even on public WAN links.  And this means significant cost savings by eliminating expensive private MPLS links. Several vendors are already doing this, so the technology isn’t brand new. I wasn’t necessarily skeptical walking into Silver Peak’s offices, but I wasn’t expecting anything groundbreaking.

Their presentation was very compelling, though, and I was impressed with David Hughes’ ability to jump in, as Founder and CEO, and dig into the technology. He explained that the impetus to move to SD-WAN is the shift to cloud applications, and that their product delivers applications with high quality over whatever connectivity the user chooses. He also explained that their solution meets a specific use case: “to connect users to applications much more easily, efficiently and cost-effectively.” In David’s words, Silver Peak’s technology is more about “providing an overlay over whatever underlying technology you want to use for your connectivity…and making that overlay really easy to manage and deploy.” Flexibility and ease of use seem to be key to Silver Peak, which is, of course, a good thing.

Damon Enniss, Senior VP of Products, presented their Unity EdgeConnect  solution to easily harden the WAN edge and deliver path conditioning, latency mitigation and increased reliability. I’ve struggled with providing this level of cost-effective optimization and fault tolerance to the WAN edge. Especially in the small to medium sized business space, organizations can’t always afford the hardware or skillset to implement the design they need. Though their requirement for uptime and a quality connection is no less important, the cost can be prohibitive.

Silver Peak did a great job, but then again I’m already excited about this technology.  Zero-touch provisioning for highly resilient WAN edge devices? Sign me up! It makes sense, especially in the SMB space, because there’s an actual business use case for this.

The most compelling value proposition to me is that using this technology means I can get rid of all my private connections. I want to ditch my expensive private MPLS link and have a pair (or three) of public internet connections. My MPLS links can cost thousands of dollars per month each. Silver Peak says they can deliver performance, quality and resiliency and cut costs up to 90%, so the real value proposition is going with all public connections. Damon mentioned that only in passing, but I think it’s the most compelling reason to go SD-WAN with Silver Peak.

Over the last few years I’ve noticed ISPs improving their service delivery, and even though I still tease about Time Warner Cable going out when it rains, business internet connectivity has increased in quality and bandwidth and decreased in cost. There are parts of the country and world where internet connectivity is still very poor, but in the northeast where I live, I can run voice over the internet today when I wouldn’t dare 5 years ago.

I want to drop the hybrid solution and go SD-WAN with all public internet links. Cheaper, easier to deploy, and easier to switch ISPs if I want. To me that’s the real value of SD-WAN. Silver Peak didn’t make that specific case very strongly, and I think they should have. An organization may require a hybrid model for some particular reason, and I understand that. But by and large, Silver Peak’s technology can provide to even the smallest customer private line performance and reliability cheaper with public internet connections only. Win.

[Follow up edit: I believe my opinion that Silver Peak didn’t focus enough on an enterprise going all public internet links was based on a misguided  first impression. Over the last five weeks I’ve looked into Silver Peak’s public literature and kept an eye on their tweets. They are absolutely positioning Unity EdgeConnect to replace private MPLS links much more than I thought.]   

Disclaimer

I attended a Silver Peak presentation as an invited delegate to Network Field Day 11. The sponsors pay to present to the NFD delegates, and in turn that money funds my transport and accommodation while I am there. That said, I don’t get paid anything to be there, and I’m under no obligation to write anything, positive or negative, about any of the presenters. 

I’m not really a security expert, but I’ve worked on enough network security projects and am ramping up for one now to have some opinions. Networking Field Day 11 wasn’t really security-focused, but there was discussion from one vendor and then during a roundtable that got the wheels turning.

At the NFD 11 roundtable on security, Greg Ferro posited that the ROI in information security isn’t worth it. It was certainly a provocative statement that only Mr. Ferro could get away with in his own inimable way, and that began a thought process for me on whether or not we actually want good information security in the first place. The beginnings of that was a comment in the PacketPushers Show 272 right around 10:17.

Information security used to be an afterthought, but it’s becoming top of mind for technical and non-technical people alike. However, though I think folks are more aware of a need for good security, I don’t think it’s mainly for the reason of actually securing networks. I’ve seen IPS appliances in racks, powered on, and doing nothing whatsoever but sitting on the management subnet in order to meet a compliance requirement. And in many cases it has little or nothing to do with compliance. These are expensive devices, and they have a predictable lifecycle: they’re purchased, configured by a specialist, start causing trouble, then taken out of prod but left on the network. If security is such a priority these days, why is this ok? Buying this gear isn’t a trivial matter, so something doesn’t add up.

My initial thought is that these reactive actions are mainly to provide peace of mind while knowing full well not much is being done to secure anything. Seeing the gear in the racks and software icons in the toolbar feels good, and it feels good to the end-users. It’s good PR.

How often is a persistent end-user given a back door to their work desktop without going through change management?

If you get to know the security guards, maybe bring them a bottle of cheap wine at Christmas, would you be able to get through most badge access doors at your facility?

How many times have you seen endpoint security software installed everywhere but disabled on the most critical servers because of the performance hit or because something’s interfering with an application?

Don’t get me wrong. I’ve seen some of these technologies and methods used effectively as part of a real security strategy, especially in the government space, but the number of times I’ve seen security done poorly and then forgotten suggests that I’m not off base here. I kind of feel like we don’t really care about security even though we say we do.

One of the recurring themes at Network Field Day 11 was that we need to get back to the heart of the matter: the business use case. A webserver exists to host a website that people can access easily to buy stuff, read stuff, or whatever. But the narrative in the news tells us the bad guys are coming, so orgs pile on the widgets, security protocols and HR consequences to mitigate today’s popular attacks. But lock it or the back-end process down too much, and it undermines its business use case.

So what’s good security then?

Matt Oswalt made a good point in one of the roundtables that one reason for server insecurity is people writing bad code. The guys at Skyport Systems, Inc discussed micro-segmentation of workloads to secure applications from lateral attacks, and they cast their argument in terms of protecting against misconfigured firewalls. To me, both Matt and the folks at Skyport gave good examples of people not doing their jobs very well. Could it be that good security is just a matter of everyone actually sticking to the best practices in their area?

A webserver is vulnerable on port 80 and/or 443 because it’s a webserver. You can’t lock it down completely unless you just shut the thing off. But is the best way to secure it a big expensive application firewall looking at GET requests and sending TCP-resets? Maybe, but as soon as it negatively impacts user experience it’s going to get pulled out. Maybe someone just needs to stop writing bad PHP code. Or maybe someone needs to stop adding a permit ip any any in their firewall (hyperbole, I hope).

Skyport posed the question – how do you build the most secure server possible? Their answer was technology: never trust the VM. If someone owns the underlying operating system they own the applications. They proposed a technology to protect actual workloads in anticipation of the trend to disaggregate application workloads among many systems. This is really cool and makes sense to me. Big Switch discussed visibility into network traffic with their pervasive monitoring and security solution. This is awesome, and according to Greg, something we should have been doing for years. So the technology is there. Maybe it’s not absolutely perfect, but the technology is there.

But I think much of this technology addresses symptoms of a deeper problem. The desire for security seems to be there, but there is little effort to QC what’s already in place and certainly not at the expense of user experience. We don’t really want information security. We just want peace of mind even if it’s unfounded.

So what does good security look like? I’m wondering if it’s just a simple matter of everyone in the stack doing their jobs correctly. Sounds simple, but really that’s a difficult strategy. No more untidy firewall rule sets. No more random IP tables. No more 300 year old versions of CentOS. Extra effort getting code peer reviewed. Actually sticking to corporate security policy even when no one’s looking.

The reality is that if we want to maintain an acceptable user experience, we really can’t have perfect security. So maybe good security is just doing what we’re already doing – but better. But until we do, with a wink a nod, we’ll probably just add more widgets to the network to feel more secure.