What strikes me as odd is that this didn’t catch on in the network side of the house. Of course, there are always exceptions. A look at networking forums from years ago will showcase a few ambitious engineers arguing over using Expect or Perl scripts to manage their switches, but this was the exception in the networking industry, not the norm.

At Networking Field Day 17, several vendors that have long embraced network programmability turned their gaze from the data center to enterprise networking, and it seems as if networking may be finally ready to catch up to what we’ve been doing with servers for decades.

In particular, Extreme Networks presented their philosophy of network programmability and what it should look like in action. Patrick LaPorte, Director of Product marketing for Network Automation, started with painting a picture of the “new” Extreme, which he described as focused on software-driven networking for the enterprise. Until recently, the idea of an automated network infrastructure has been in the context of the data center and WAN and concerned with device provisioning. Very few in the vendor community have expressed much interest in the campus and with ongoing network operations.

There’s great value using network automation for device provisioning. For example, for a switch refresh project, I’d start with reviewing with a customer their current switch configuration and developing a new base configuration for their new gear. Then I’d spend days copying and pasting that config onto switches while also recording switch information such as MAC addresses, hostnames, and management IP addresses into a spreadsheet.

This can take days or even weeks, and my spreadsheets become less reliable after so much copying and pasting. More than once I’ve skipped a device but not a line and completely botched a spreadsheet simply because of an administrative error.

Tools such as Ansible and Python make this much faster and more reliable, and Patrick alluded to this when he explained that, among other things, Extreme’s goal for their technology is to provide engineers a means for greater agility and efficiency in programming their networks.

But the specific problem isn’t the initial device provisioning. What Extreme is concerned with is the chokepoint between device provisioning and Day 2 operations. And herein lies the problem for many enterprise organizations. No one is disputing the benefit of network automation, but the skillset required to build entire automated workflows for ongoing network operations is difficult to come by. A typical enterprise like a regional hospital or local manufacturing facility often can’t afford or attract that kind of talent let alone realize there’s a better way to do networking.

Recently I’ve begun to see vendors package up the most common open source tools into a neat and tidy platform that abstracts away the code and presents network engineers a GUI to automate the network. As great as that seems, I don’t believe this is a new concept in networking. We’ve managed physical resources with this sort of interface for years using platforms such as Call Manager and our various wireless LAN controllers.

What Extreme is doing, however, is embracing the automation of an entire workflow including machine-to-machine automation and an open perspective able to accommodate multiple vendors. Ram Galla, a Senior Product Manager at Extreme, explained that he sees cross-domain automation as “not just provisioning per se, but how you integrate provisioning into the customer’s processes and tools.”

This is the right way to think about automation. It’s more than Day 0 provisioning, and it has to integrate completely into an existing network operations workflow presupposing some semblance of openness. Patrick and Ram were quick to explain that their Workflow Composer platform isn’t a closed system working only with their own devices, though it was designed primarily for their SLX, VDX, and MLX series switches and routers.

Instead, Extreme works with the open-source community and integrates with NAPALM in order to provide methods to communicate with non-Extreme CLI devices, execute actions in a multi-vendor environment, and integrate with a network team’s other tools to serve as the foundation of a completely automated network operations workflow. As one can see in the right side of the image above, even the code itself is made available for an engineer to examine and adjust according to their particular operational processes.

I believe they’re doing this right because they understand that many enterprise organizations will likely start with the automation platform alone and replace only the switches in one part of their network. This lowers the barrier to entry for a programmable infrastructure, makes buy-in easier from reluctant decision-makers, and decreases the potential failure domain if things go poorly.

I’m not interested in a closed automation system. I don’t much care for a controller-based pile of switches that requires I use only specific hardware, locks me into only the built-in automation templates, and doesn’t integrate with my existing tools. Something like that will end up being yet another platform that everyone is excited about for a few days and then never logs into again.

Network programmability is something the industry is embracing, and Extreme’s approach is to work with the community to provide an open platform that can fully integrate with a network operations workflow. Though it’s going to take time, networking is starting to catch up to what we’ve been doing with servers for decades.

Disclaimer: Gelstalt IT, the organizers of Networking Field Day, provides travel and expenses for me to attend Networking Field Day. I do not receive cash compensation as a delegate. Also, I do not receive compensation for writing about or promoting Networking Field Day.

After that, I kept going back to tweak it. Then, instead of moving on to the next task, I looked for things I could do that required I write another script. I became kind of hooked on getting better writing Python. I’m still pretty bad, but I’m spending quite a bit of time every day trying to improve.

At the same time, I’ve been studying for the CCIE on and off for almost three years, in earnest for the last year. There’s very little more satisfying to a network engineer than configuring a solution, testing connectivity, and getting back !!!!!

But lately, putzing around with lines of Python code until I finally get it to run without errors has been almost equally satisfying.

The thing is, when I go to work every day I’m seeing practical examples of how knowing how to write a simple script is as helpful to me as deep networking knowledge. My experience has been that once BGP neighbors relationships are formed, OSPF adjacencies are in the Full state, and traffic takes the correct path across my WAN, I’m not wiggling those proverbial wires much any more. Instead, I find myself incorporating sysadmin skills and simple programming to get day 2 and beyond tasks done.

I don’t work for a VAR anymore, so I don’t have the opportunity to finish one big WAN project then jump to the next for a different customer. Instead I have to find a way to add value with what I have in front of me, or in other words, make the network (and network security in particular) more robust, easier to manage, and automated.

About 18 months ago I chatted with Jon Langemak about this very thing. He explained how it was difficult for him to shift from a CCIE focus to his current focus in the automation and DevOps world, but how it was needful because it was how he could add value in the context in which he worked. Basically, he looked around and identified where he could be most useful, and then he went for it. That conversation has stuck with me ever since.

I’ve reaped tremendous rewards and seen incredible opportunities as a result of earning a variety of technical certifications, most of them from Cisco. But for some reason, the opportunities that present themselves to me nowadays don’t lend themselves to going from project to project configuring ASRs or replacing switch stacks in IDFs. I’ve done that before, and a big part of me really wants to do that again. Instead, the opportunities that come my way have less to do with configuring boxes and more to do with automation, creating better workflows, and aligning technology with business.

I’m not sure what to do with all this right now. In my mind, the natural progression for a network engineer is to get the next level certification, which for me is the CCIE, and then work on bigger and more complex technical projects. I know that’s still the natural progression for many engineers out there, but it’s not what’s happening in my life right now, even though I want it to be.

I’m not commenting on the industry here, and this isn’t another blog post about how network engineering is changing. These are the observations of my personal journey with the CCIE and with my current dayjob.

So where can I add value, and what opportunities are right in front of me that will help me grow? Well, for me, that means finding ways to automate tasks, collect large amounts of information from the network, find meaning in it, and use whatever methods I can to make network devices and the security posture of my organization more stable and consistent.

It also means creating professional writing, casting long-term vision for the area I’m responsible for, and being an available resource that can speak to the overall picture of the network rather than focus on one router at a time.

Though I’m still going through CCIE labs regularly and taking breaks to re-watch an INE video on various topics, I’m also spending just as much time improving my ability to write in Python using resources like Code Academy and Learn Python the Hard Way.

This doesn’t mean that my networking knowledge is now useless. In fact, it forms the foundation for everything I’m working on. I believe that the CCIE is still extremely relevant, and many of my friends in the networking community are working on higher level networking projects daily that require that sort of knowledge. But that’s simply not what’s been happening in my life lately, no matter how much I’d like it to be or how hard I try to make it happen.

So I feel I’m at a crossroads – probably how Jon felt a few years ago. Maybe it’s more of a transition or career evolution, I really don’t know. At first, it was absolutely terrifying to confront. But now, rather than continue to kick against the goads, I’m open to what seems to be developing naturally.

Thanks,

Phil

The Problem: Network-Blindness

Derick Winkworth (@CloudToad) and Jeremy Schulman (@nwkautomaniac) gave an interesting presentation exploring the nature of a network as a single logical construct rather than individual pieces of hardware. Derick started off by describing a condition called “face-blindness.” Those suffering from face-blindness have difficulty organizing the images of facial features into an entire face. They can see the individual components, but they can’t see the logical construct the arrangement of these images creates – a face.

He explained that some see the network in the same way, calling the phenomenon “network-blindness.” In other words, the network doesn’t exist in and of itself; it’s actually an ideal state that exists in our minds. A network is a logical construction of various components such as cables, hardware, and configurations, and we expect it to behave a certain way. Unfortunately, those suffering from network-blindness aren’t always aware of the network as a logical whole and instead focus on its individual components.

This may sound esoteric, but it makes sense to me, and I feel it’s important to flesh this out because it forms the foundation of what Apstra is doing. What Derick focused on was context, and I really enjoyed hearing a fellow networking geek think about networks this way. Think of it like this: alone, each individual router, switch, cable, and line of code is not a network. However, each component is part and parcel of the greater context of the network in which it participates.

He elaborated by saying that many large networks are so complex that it’s just too difficult for an engineer to have an accurate big picture. I know from experience that Derick is right about that. Networking folks really do get lost in the weeds configuring devices and features in one small part of the network and [unintentionally] forget the big picture. We want to maintain the vision of the big picture, but it’s hard to do that when troubleshooting why DSCP markings aren’t treated properly on some switch somewhere.

Jeremy explained that Day 2 troubleshooting is probably 70% of the problem with networks. Almost immediately after an initial install, the network is going to be different than the original design. How does an engineer know the true state of the network at any given time? And probably more importantly, how does the current state differ from the expected state?

The Cure: Situational Awareness

From a technological perspective, the Apstra Operating System maintains what Jeremy Schulman calls “situational awareness.” This means AOS knows the big picture. It knows what the engineer expects of the network, so it can therefore detect when something isn’t right based on how the current state differs from expected state. What we expect from the network is the intent, and this is how Apstra cures network-blindness.

Under The Hood

Under the hood, AOS breaks up a particular service into its components with regard to configuration and expected state. AOS continually collects network state information, validates configuration, and ultimately performs device configuration.

The impression I got was that Apstra is taking the methods to create a programmable infrastructure that we’re already familiar with and packaging them into an extremely easy-to-use graphical user interface. Then AOS takes abstraction to the next level by providing all the built-in logic for the intent-engine. This is an overlay approach, not device-level programmability, and is what I feel SDN has been promising for years.

Picture a single piece of software that has all your Python scripts, Ansible playbooks and Chef cookbooks all rolled into one. In this case Chef is a good analogy because each network device needs an AOS agent installed in order to be managed.

Rather than use SNMP, AOS uses this agent-based method to collect network data, perform those often-forgotten validation checks, and configure devices. This is different than other vendors’ solutions because not only does AOS not use SNMP, but it’s also vendor agnostic.

During the first demo, Jeremy pointed out that the interface was so straightforward that a facilities person would be able to identify and fix a cabling problem in the data center. In the screenshot below, notice the two red cables and the alarms.

This means that AOS knows what the network is supposed to be like, and this is the context that Derick referred to. This implies there is either a large  number of build templates that comes with AOS, or I’ll be spending a lot of time configuring my own. Maybe I need to stop configuring snowflakes, but as the platform matures, I wonder how the AOS intent-engine will reconcile and validate built-in templates with custom configurations.

AOS also validates running configuration in even a multi-vendor network. That means the folks at Apstra built so much logic into their software that you can swap out switches from different vendors and AOS would find the diffs automatically applying the necessary changes to produce the intended state. That’s just amazing. Even if you’re a one-vendor-shop, this is still utterly amazing.

Take a look at the screenshot below. The validation process alone is huge and something I would require before even attempting to integrate AOS into my network.

AOS utilizes pools of built-in resources to build configurations based on the parameters a network engineer sets for a service. The system uses a series of config files using extendable Quagga and Jinja templates, for example. When a device enters inventory, information about the device is gathered in order to make it available for use as a resource. Currently, version 1.0 supports 26 platforms including hardware running Cisco NXOS, Arista EOS/vEOS, Cumulus Linux, and CVX.  Version 1.1 should be coming out very soon and support more platforms.

Minor Concerns

Something important to note is that the AOS configs are automatically generated using what Apstra considers best practice for a particular service. They determined these by speaking with the various networking vendors, but this does affect your ability to use certain proprietary features. Jeremy addressed this by explaining that AOS doesn’t really abstract device level mechanisms and instead abstracts entire network services comprised of smaller network services.

Therefore the abstraction AOS provides utilizes whatever features are available to compose the overall service regardless of the proprietary technologies involved. This means you get the base configurations that get the job done (provides a particular network service) and allows you to pursue DevOps goals around the edges.

I can’t imagine how much programming has to be built into the system in order to accommodate the many services and vendors used in today’s networks, which is why I’m not surprised to hear that the platform technology is currently focused mainly on layer 3 data center Clos designs.

To that end, Apstra embraces the open network-developer community and welcomes community development of software to interface with their APIs. This should expedite the development of the platform and its relevance in actual production networks. I assume we’ll see a lot of third-party integration tools show up in time, so keep an eye on their GitHub page for more details.

I do wonder if relying on AOS build templates will be OK with higher level network engineers designing and building large data centers. I want to feel comfortable with that, but I admit that I’d probably feel compelled to scroll through all the configs myself after AOS did its magic.

So far, AOS is geared mainly for the data center, which I completely understand, but it’s also a sign of the platform’s immaturity. I don’t mean that to be negative; in fact, Jeremy explained that “the platform technology of AOS is adaptable for just about any kind of network application you want to use.” I have to assume that as AOS evolves beyond version 1.0, its relevance to the campus, WAN, etc. will also be developed and evolve. Apstra’s goal is to follow the example of the server world from years ago when hardware and software was decoupled giving engineers incredible choice and control over the infrastructure.

—

There are networking vendors that have a slick GUI to manage piles of their switches, but AOS is vendor agnostic, community-focused, and intent-driven. These are major differentiators and not trivial whatsoever.

I don’t know too much about the business side of things, and I know that companies are bought and sold regularly in Silicon Valley. But from a technical perspective, the logical conclusion of what Apstra is doing is incredibly compelling. In the near term, this will change data center networking. In the long-term, this will revolutionize the way we look at networking altogether.

Pay attention for their next release coming soon.

I’m particularly interested in Apstra’s presentation on Thursday afternoon. I recently wrote an article about intent-driven networking, something of particular interest to me, so I’m really interested to hear what they have to say about their platform, the Apstra Operating System, or AOS.

Intent-driven networking is concerned with telling the network what you want it to do rather than how to configure services device by device. This is another level of network programmability but with more intelligence than simply device provisioning. Apstra boasts the ability to eliminate many of the repetitive configuration tasks common among various networking operating systems by providing programming to automate intent expressions made by a network engineer. AOS is then vendor agnostic and focused on the end-to-end network service rather than micro-configurations.

Apstra explains that AOS is not opinionated, or in other words, network engineers can still insert their intelligence into design and configuration. I’m very interested to hear more about their distributed intelligence model using agents on network devices interacting with a logically unified data repository. That sounds pretty cool, and I’d like to know more about how specifically the network can adjust itself during production without human intervention. They also seem to be completely data center focused, which is fine, but I am curious to hear about their plans for campus, WAN, and firewall implementations.

I agree with Ethan Banks who in a Packet Pushers podcast with Apstra said that intent-driven networking is sort of what we all thought SDN was going to be. This is why I also think we should all pay attention to what Apstra and other similar companies are doing in this space.

I’m a fan of Star Trek, and I love how Lt. Commander Geordi LaForge tells the U.S.S. Enterprise computers what to do with infrastructure pathways, and the computer just does it. I’m really interested to see where intent-driven networking takes us in the coming years.

Several of these tweets made light of the fact that network automation is merely deploying some scripts to push configuration to a small (or large) number of network devices. Therefore, a misplaced comma or some other minor typo in the code would devastate the entire production network. This would be my conclusion as well if the premise was completely true, but I don’t believe it is.

I’m sure there are some network engineers out there that are quick to write a sloppy script and push config into production without proper review, testing and validation, but that’s not at all part of the network automation movement as I see it. NetDevOps applies the DevOps paradigm to network management which means there is great value placed on continuous improvement, testing and validation, collaboration and community. Yes, there is also the no-blame concept which may decrease the level of accountability for any single network engineer, but that’s more than matched by the emphasis on peer-review and built-in client-side mechanisms to prevent the network from blowing up. Therefore, the original argument that network automation is too dangerous battles against a straw man that has little to do with what’s really going on in the network automation trend.

Next, to say that network automation is clever but too risky to use in production presupposes that managing network devices via SNMP or directly on the command line one device at a time poses much less risk. Trends in network automation seek to decrease human error by eliminating human interaction with individual boxes. Assuming that proper measures were taken to test and validate configuration, this means that proper use of network automation techniques would actually decrease the likelihood of error, not increase it. For example, copying and pasting 10 lines of config into a router has no built in validation or rollback mechanism if the configuration doesn’t work right. Some lines of configuration would apply, though, resulting in an ugly effort to figure out what commands did take and then go back to delete them one by one. This is in contrast to pushing configuration objects to an open NX-OS device. If there was some problem with any part of the configuration, the rollback-on-error option would automatically rollback the entire configuration if a mistake was found.

I also understand that part of the concern is the awkward balance between efficiency, i.e. managing many network devices at once, and keeping the failure domain as small as possible, i.e. managing network devices one box at time. This tension makes sense to me because I’ve been in the dreaded network down scenario enough times that I know to avoid it at all costs. However, the cause of most network outages I’ve witnessed was human error at the command line. Therefore, decreasing the potential for this is a step in the right direction, not a move toward nuclear annihilation of the network.

I know that I’m making a small pile of assumptions as well. I assume that the network engineer utilizing Python, Ansible, open APIs and the variety of other means opening up to us is using them skillfully. A sloppy script will take down a network, but a careless application of the shutdown command or VTP revision number will cause significant devastation as well.

Therefore, it’s not the methods and techniques that are flawed, necessarily, but the network engineer. We make mistakes, and we need to find ways to mitigate those mistakes. Network automation absolutely does this, but just like any other professional endeavor, it requires skill to utilize effectively.

Also, there is a sentiment that network engineers’ jobs will disappear or change so much that their skills will not be very relevant. I see this assumption all over the place, and maybe addressing it should be its own post, but for now I’d like to address it simply by saying that a network engineer will always need to know  what a VLAN is and what the K values are in EIGRP. Network engineers will always need to have a deep understanding of networking concepts because of the simple fact that they are managing networks. Yes, some elements of configuration may be abstracted; nevertheless, the serious and professional network engineer will still need to know the deep aspects of networking in order to make the magic happen.

Network automation is another tool in the network engineer’s toolbox. 

Picture a veteran astronomer working late one evening, PhD diploma on the wall, coffee at the ready, and dual monitors flickering with incoming data from an exciting new object he’s discovered in the brilliant night sky. Quickly he and his graduate student assistants put together a Python script to analyze the data in order to deepen their understanding of this exciting new phenomenon in the sea of space.

When the astronomer turned aside to write this script, was he no longer an astronomer? Did the usefulness of using Python to analyze and manipulate awesome space data immediately negate any need for a serious understanding of astronomy? Of course not!

In this case, Python is just a tool in the scientist’s toolbox for being a more effective astronomer. In the same way, using the network automation techniques developed and promulgated by the open source community and vendors alike does not presuppose the network engineer is no longer relevant. In fact, network automation is simply another tool in the network engineer’s toolbox allowing him or her to be just that much better of an engineer.

I’m pretty new to this brave new world, and my passion is still certainly in traditional networking, but I can say confidently that no one is trying to re-invent the networking wheel. The industry is just trying to make the ride smoother.