Most of the traffic in our data centers is east-west, with only a small fraction actually being northbound out to the rest of the world. The cost of massive firewall appliance clusters operating at line-rate is astronomical, and it doesn’t make sense to punt traffic all over the place if there’s a better way.

For years we’ve written ACLs on core switches and deployed massive data center firewalls to segment our servers, but VMware has been taking security much closer to the endpoints themselves with their brand of distributed firewall. Last January at Networking Field Day 17, Geoff Wilmington, Senior Technical Product Manager, NSBU, took us on a deep dive presentation into the nuts and bolts of VMware’s context-aware micro-segmentation solution.

Micro-segmentation is about isolation among parts of your network even if the devices exist in the same IP address space. VMware’s micro-segmentation strategy is to segment the actual tiers of an application, such as the front end, database, etc, in order to provide stateful firewall services at the virtual machine level itself. For example, a network admin would not only be able to lock down a VM to one network port such as HTTPS but actually  lock down an application or database tier from another further decreasing its attack surface.

Built right into VMware NSX, this distributed context-based firewall collects flow data from vNICs on VMs and feeds it into flow tables forming the basis of the distributed firewall rules. This is important because it means each VM has its own firewall perimeter solving the problem of having to shift traffic around the data center to inspect it.

VMware NSX is simplifying network security by bringing security right down to the application itself.

The distributed firewall module is in the actual kernel of every hypervisor allowing firewall services to easily scale horizontally. Each individual VM running its own firewall at first glance might seem more complex, not less, but because firewall policies are created for logical groups rather than at the network edge, configuring the distributed firewall is actually more simple.

All firewall rules are managed centrally as logical groups in NSX. Because NSX integrates fully with vCenter, a network administrator can configure rules for applications based on workflows rather than IP addresses and MAC addresses. And since rules are applied to workflows at the application tier, rules actually follow VMs as they move from host to host.

Putting the firewall at the kernel level means that firewall throughput is near line-rate – a very expensive proposition in hardware-based traditional data center firewalls. And in case you were worried running a firewall this way would chew up all your CPU cycles, the good news is that typically you’ll see consumption of about only 3% – 5% CPU utilization related to firewall services.

VMware’s context-aware micro-segmentation capabilities are largely based on a 5-tuple identification with growing capabilities for more identification factors. Geoff explained that the 5-tuple method is still the primary way to create firewall policy, and this was a little disappointing to hear. The 5-tuple method looks at 5 aspects of network traffic including the source and destination IP address as well as port numbers and protocol.

At the time of VMware’s presentation they did support about 50 server side applications for east-west traffic, which, I hope, will increase significantly. That’s at the heart of the problem VMware is trying to solve, namely, the complexity of effectively securing east-west workflow traffic.

Geoff concluded by stating very frankly that micro-segmentation isn’t easy – it’s a process. VMware’s goal is to take the complexity out of micro-segmentation so engineers can benefit from the increased security posture without having to also increase their workload.

The Application Rule Manager was introduced in NSX 6.3 as the toolset to automate this process and simplify identifying workflows and creating rules. With ARM we can take advantage of the distributed firewall platform without having to figure out and manage innumerable rules.

I really like the concept of pushing security closer and closer to the endpoint and focusing on application flows rather than only network parameters. It provides more granular firewall control and a better security posture. The only issue is the daunting task of managing hundreds if not thousands of potential firewalls individually. VMware solves this problem with their context-aware distributed firewall built in to NSX and with the automation of the Application Rule Manager.

My short time focused on network security was mostly about wondering how I could secure traffic without an entire team of engineers. VMware’s developing approach gives us the ability to micro-segment our data centers in one single platform.

Disclaimer: Gelstalt IT, the organizers of Networking Field Day, provides travel and expenses for me to attend Networking Field Day. I do not receive cash compensation as a delegate. Also, I do not receive compensation for writing about or promoting Networking Field Day.

I’m not really a security expert, but I’ve worked on enough network security projects and am ramping up for one now to have some opinions. Networking Field Day 11 wasn’t really security-focused, but there was discussion from one vendor and then during a roundtable that got the wheels turning.

At the NFD 11 roundtable on security, Greg Ferro posited that the ROI in information security isn’t worth it. It was certainly a provocative statement that only Mr. Ferro could get away with in his own inimable way, and that began a thought process for me on whether or not we actually want good information security in the first place. The beginnings of that was a comment in the PacketPushers Show 272 right around 10:17.

Information security used to be an afterthought, but it’s becoming top of mind for technical and non-technical people alike. However, though I think folks are more aware of a need for good security, I don’t think it’s mainly for the reason of actually securing networks. I’ve seen IPS appliances in racks, powered on, and doing nothing whatsoever but sitting on the management subnet in order to meet a compliance requirement. And in many cases it has little or nothing to do with compliance. These are expensive devices, and they have a predictable lifecycle: they’re purchased, configured by a specialist, start causing trouble, then taken out of prod but left on the network. If security is such a priority these days, why is this ok? Buying this gear isn’t a trivial matter, so something doesn’t add up.

My initial thought is that these reactive actions are mainly to provide peace of mind while knowing full well not much is being done to secure anything. Seeing the gear in the racks and software icons in the toolbar feels good, and it feels good to the end-users. It’s good PR.

How often is a persistent end-user given a back door to their work desktop without going through change management?

If you get to know the security guards, maybe bring them a bottle of cheap wine at Christmas, would you be able to get through most badge access doors at your facility?

How many times have you seen endpoint security software installed everywhere but disabled on the most critical servers because of the performance hit or because something’s interfering with an application?

Don’t get me wrong. I’ve seen some of these technologies and methods used effectively as part of a real security strategy, especially in the government space, but the number of times I’ve seen security done poorly and then forgotten suggests that I’m not off base here. I kind of feel like we don’t really care about security even though we say we do.

One of the recurring themes at Network Field Day 11 was that we need to get back to the heart of the matter: the business use case. A webserver exists to host a website that people can access easily to buy stuff, read stuff, or whatever. But the narrative in the news tells us the bad guys are coming, so orgs pile on the widgets, security protocols and HR consequences to mitigate today’s popular attacks. But lock it or the back-end process down too much, and it undermines its business use case.

So what’s good security then?

Matt Oswalt made a good point in one of the roundtables that one reason for server insecurity is people writing bad code. The guys at Skyport Systems, Inc discussed micro-segmentation of workloads to secure applications from lateral attacks, and they cast their argument in terms of protecting against misconfigured firewalls. To me, both Matt and the folks at Skyport gave good examples of people not doing their jobs very well. Could it be that good security is just a matter of everyone actually sticking to the best practices in their area?

A webserver is vulnerable on port 80 and/or 443 because it’s a webserver. You can’t lock it down completely unless you just shut the thing off. But is the best way to secure it a big expensive application firewall looking at GET requests and sending TCP-resets? Maybe, but as soon as it negatively impacts user experience it’s going to get pulled out. Maybe someone just needs to stop writing bad PHP code. Or maybe someone needs to stop adding a permit ip any any in their firewall (hyperbole, I hope).

Skyport posed the question – how do you build the most secure server possible? Their answer was technology: never trust the VM. If someone owns the underlying operating system they own the applications. They proposed a technology to protect actual workloads in anticipation of the trend to disaggregate application workloads among many systems. This is really cool and makes sense to me. Big Switch discussed visibility into network traffic with their pervasive monitoring and security solution. This is awesome, and according to Greg, something we should have been doing for years. So the technology is there. Maybe it’s not absolutely perfect, but the technology is there.

But I think much of this technology addresses symptoms of a deeper problem. The desire for security seems to be there, but there is little effort to QC what’s already in place and certainly not at the expense of user experience. We don’t really want information security. We just want peace of mind even if it’s unfounded.

So what does good security look like? I’m wondering if it’s just a simple matter of everyone in the stack doing their jobs correctly. Sounds simple, but really that’s a difficult strategy. No more untidy firewall rule sets. No more random IP tables. No more 300 year old versions of CentOS. Extra effort getting code peer reviewed. Actually sticking to corporate security policy even when no one’s looking.

The reality is that if we want to maintain an acceptable user experience, we really can’t have perfect security. So maybe good security is just doing what we’re already doing – but better. But until we do, with a wink a nod, we’ll probably just add more widgets to the network to feel more secure.

Not long ago I worked with a nonprofit in the healthcare industry to replace their network core. It was an organization of a little less than 2,000 end users on a single campus. Many people were remote part-time workers and not really information users, though they would log in occasionally to access an online timesheet or email. There was one network egress through an unreliable ISP, so part of this project would be adding a second internet connection to the mix. The main goal was to gain better network fault tolerance.

We looked at hardware and configuration options – they were stretching their budget to the max. Our initial thoughts were pretty standard: probably Catalyst 4500Xs at the core running VSS, a couple 3945 edge routers running HSRP, a failover pair of Cisco ASA 5545Xs and some switch stacks. We even discussed a building generator and moving their email to Office 365.

I was the delivery engineer for this project, not the solutions architect, so I fell out of the loop during the rest of the design process and was brought in just before kickoff. Looking over the final design I saw the 4500X switches, one 3945 router, and a pair of ASA 5585Xs in multicontext mode. They decided to skip the second ISP, increase the bandwidth with their current provider and go with larger ASAs. I had a mild disagreement with the ASA sizing and dropping the ISP, but that wasn’t my decision. Then I realized that the customer, who relied heavily on Anyconnect as their remote access solution, would be up the creek since multicontext mode didn’t support remote access VPN.

I faced a conflict here because there was a disparity between the business requirement and the technical solution.

First let’s consider the elimination of the second ISP from the plan. It was clear that the incredible cost of the bigger ASAs precluded the customer from entertaining a second provider. That made no sense to me because of how unreliable their current provider was and how heavily they relied on RA VPN. I looked at the purchase order and almost fell out of my chair when I saw how much each ASA plus licensing cost. Cisco ASA 5585Xs aren’t cheap, and each security context costs thousands. The business requirement called for local resources to be available by remote workers at all times, so eliminating the second ISP seemed strange to me.

Now let’s consider multicontext mode. (No, I won’t be explaining how to configure it because, well, Google.) Multicontext mode on a pair of Cisco ASAs allows for an Active/Active state so you can load-balance traffic as well as have subsecond failover. You also get multiple security contexts, or in other words, multiple virtual firewalls living in the same box. This is useful for multi-tenancy configurations such as with service providers and large multiple business-unit companies. This customer had no need for that whatsoever, and since multicontext mode does not support remote access VPNs and is extremely expensive I was again confused why this layer of cost and complexity was added.

Also consider the actual hardware.  Failover designs should plan for each individual firewall to be able to accommodate 100% of network traffic in case of a failover scenario. It’s important to choose a model that can alone support all network traffic. In this case each new 5585X was such an incredible overkill that I felt it was a waste of a lot of money. I know sometimes we choose hardware to meet future growth, but this was a very static organization funded mainly by government grants. There was no need to be able to accommodate huge growth within the next 7 – 10 years before the next hardware refresh. Even the lower end 5500X firewalls like the 5515X or 5525X would easily fit the bill. 

An alternative would be running the firewall pair in Active/Standby on smaller (cheaper) firewalls. This configuration also provides subsecond failover, plus it offers all the features of the ASA including remote access VPN. No, there’s no load balancing, but I really don’t think that was relevant in this case. This would have freed up all the money needed for a second router and a second ISP. Even having a second ISP but just a single router would have made sense to me.

I know that multicontext mode on Cisco ASAs has its place –  I’ve used it for several projects. I’ve also installed 5585Xs in data centers and very large networks where the traffic baselines required the larger model. But for this much smaller customer, I don’t believe the huge cost increase was worth it at all. Sticking with an Active/Standby solution would have freed up the money to do the other things necessary to meet the customer’s business requirement.

Lastly, there was no mention of replacing any of their old, daisy-chained access switches. My thought was to put in a couple basic switch stacks with 2960Xs in order to run portchannels to mission critical servers and portchannels back to the core. The 2960Xs are among the cheapest Catalyst switches you can get other than the SMB models, and in my mind they were necessary to add the access layer redundancy the customer asked for.

Though the final design cost much more in hardware than my preliminary design, it didn’t meet the customer’s business requirements. It’s my opinion that though a design governed by margin on hardware can be more lucrative, it certainly isn’t always the best solution for the customer. In the long run, that may even hurt the relationship and therefore the bottom line. I’ve seen enough one AP per classroom designs that I already had a tendency to look at final purchase orders with a bit of cynicism.

I love cool new network equipment. I really do. But consider why you choose a technology. If it isn’t truly the best solution to meet the customer’s need, then it probably isn’t the right solution.