Why is Everyone Upset with RadioShack?

The following is a position paper that I wrote in April of 2015 To set the timeline, this was merely weeks if not days after RadioShack announced that it was selling it's customer information database, which came shortly after it's bankruptcy. You know, that database that was assembled with the information demanded of you at the register every time you stopped in to grab a pack of batteries. This is in spite of their policy that they would never sell that information without your consent (emphasis mine).

Symmetric Traffic and IPS

A well known problem for network and security professionals in the enterprise is asymmetric routing.  At it's simplest, this is where traffic flows outbound through Router A, while the return traffic returns through Router B, or through both Routers A and B.   If you're using a reflexive ACL, for example, this will lead to some, if not all of the return traffic being blocked as it attempts to return through Router B.  This is due to Router A having a record of the outbound traffic while Router B does not.  Riverbed breaks this down into several sub-categories such as complete asymmetry, server-side asymmetry, client-side asymmetry, and multi-SYN retransmit.  But for our purposes here, it's all asymmetric, and it's all a bad thing.  While some firewalls are able to share state to avoid this situation, not all do.  And Cisco Routers running IOS do not.

While asymmetric routing is known to be a problem at the network edge, it can be a problem for security professionals internally as well.  And the larger the network is, the more likely asymmetric traffic is to occur at some level.  When you deploy an IPS sensor in the network, it must be able to see all traffic in both directions for maximum effectiveness.  When an IPS sensor is able to see all the traffic involved in a particular session, you get better threat detection, reduced susceptibility to IPS evasion techniques, and less susceptibility to false-positives and false-negatives. 

While it cannot be completely avoided at the enterprise edge, the good news is that internally, steps can be taken to reduce if not eliminate the effects of asymmetric routing.  So good network design is a must to get the maximum effectiveness of an IPS deployment, particularly if there are going to be multiple sensors along a given traffic flow.

There's a few options to ensure symmetric traffic flows, or to mitigate the effect of asymmetric traffic flows including:

• Duplicate traffic across multiple IPS sensors to ensure each sensor can see all applicable data.  In addition to the challenges presented in getting all the relevant data to each IPS, we also have a greater likelihood of overloading IPS sensors with traffic, which will result in packets being dropped.
• Integration of an IPS switch.  This is reducing traffic down to a single switch.  While it is better from an IPS standpoint, it's introducing a single point of failure into the network.
• Correctly configuring spanning tree parameters to ensure symmetrical paths across Layer 2 areas.
• Routing manipulation with techniques such as PBR. This is a cost effective solution as it involves only configuration changes rather than additional hardware.  But it adds complexity to the network in addition to requiring cooperation between security and networking. 
• Sticky load-balancing utilizing technology such Cisco's ACE module or Riverbed's Asymmetric Routing Detection to better reduce the chances of asymmetric routing.
• In cases of HSRP induced asymmetry, utilize EEM and EOT in order to change the paths of HSRP related routes dynamically.
• Configuring firewalls as active/standby pairs rather than active/active pairs.

But as you can see, many of these techniques involve taking redundant data paths out of the equation, and therefore reducing the amount of overall usable bandwidth across the network.  Others involve sending more data to or through each IPS unit, increasing the burden on each unit and increasing the likelihood of dropped packets.  So there is obviously a balancing act between performance and visibility.

The Accuracy of Sampled Netflow

To alleviate the fear of overburdening the CPU due to the collection of NetFlow statistics, Cisco gives us the option of using Sampled NetFlow. Sampled NetFlow allows you to sample 1 out of 10 packets, 1 out of 100 packets, or however much of a subset of the total number of packets. The theory is that with a good sample, the traffic will still be indicative of what is flowing through the router. If 10% of the total amount of packets following through the router is DNS queries, for example, then approximately 10% of the total amount of packets in the sample will also be DNS queries, and so on.

The reason that this is necessary is because of the way that a router handles traffic when collecting NetFlow statistics. In order to process a packet in order to collect NetFlow statistics, that packet has to be processed by the CPU. When sampling is enabled, the packets that are not part of the sample are switched faster because they will not require the additional processing required.

NetFlow sampling is enabled on supported IOS platforms with just a few commands.

ip route-cache flow sampled

ip flow-sampling-mode packet-interval 100

NetFlow sampling can be monitored with the show ip flow sampling command.

So as you can see, NetFlow sampling is simple to configure and monitor. It only takes a couple commands. But the question now needs to be asked, how indicative of the total network traffic is the sample? In other words, if I’m seeing 10% of all traffic being DNS queries in my sample, is 10% of the total traffic flowing through this router really DNS queries? Or is there some significant level of error in the sampling? In Cisco documentation and Certification Exam Guides, it is admitted that the sample will never be 100% accurate, but that it should usually be pretty close. They’ll also mention that you should obviously check the accuracy periodically.

Recently, I came across an academic article talking about the accuracy of NetFlow sampling. In the article, they collect data over time with a 1 in 250 packet NetFlow sample and compared it to a raw traffic sniff utilizing tcpdump. Shown below is Figure 8 from the article, which summarizes their findings. The red dotted line shows real time data of traffic flowing through the router, while the solid blue line shows real time data of their 1 in 250 NetFlow sample.

The article states that "In Figure 8 the cumulative empirical probability is plotted with its relative error. It indicates that the performance of systematic and static random sampling is not distinguishable in practice. We believe it is true in most of backbone links where the degree of multiplexing of flows is high."  In other words, the sample is really indistinguishable from the full data set.  Equally of importance, they found that the processing overhead of NetFlow sampling to be insignificant.  Further accuracy of their collection methodologies is demonstrated by SNMP byte count data strongly correlating with NetFlow byte count data.  There's a lot of statistics and graphs in the article if you're into that sort of thing.

Conversely, in another academic article, the researchers found their sampling to be significantly less accurate.  They stated that "Our experimental results allow us to come to the conclusion that: (i) our traffic classification method can achieve similar accuracy than previous packet-based techniques, but using only the limited set of features provided by NetFlow, and (ii) the impact of packet sampling on the classification accuracy of supervised learning methods is severe."  They discuss a training process which gets their accuracy to 85% for a 1/100 sampling.  Good enough for most use cases, but still too manual and still still a far cry from the results of the first study.

So where do we stand with Sampled NetFlow accuracy?  One study says it's pretty accurate, and the other says not so much.  So the jury is still out, and we're back to Cisco's recommendation that you should be testing the accuracy to determine if it is good enough for your use case.  Like the team in the first article, you can easily use a network tap or SPAN port to compare what is actually coming out of a router interface with the NetFlow sample estimating what is coming out of that router interface.  Don't just assume.

IOS Zone Based Firewall

One of the most commonly covered security features when it comes to Cisco security is the ZBF.  It wouldn't be much of a network security blog without at least one post on this topic, so here's my take.

With IOS version 12.4(6)T, Cisco introduced the Zone Based Firewall (ZBF), sometimes referred to as the Zone Policy Based Firewall.  With this, the Classic IOS Firewall or Context-Based Access Control (CBAC), available since IOS version 11.2, is now deprecated. Nearly all of the features of the Classic IOS Firewall are implemented in ZBF as well as wide range of new features. In addition to the new features available in ZBF, it is also said to improve firewall performance over CBAC for most inspection activities.  I've seen it stated in some places that if you attempt to inter-mingle CBAC configuration commands with your ZBF, it MIGHT work, however most documentation states that it wont.  So I wouldn't risk it.  Choose one or the other.

Server 2003 IAS RADIUS Server

Since I'm sure many home labbers are still rocking Server 2003, I'll put it up in hopes that someone will still find it useful. This post was originally done a number of years ago when Server 2008R2 was still new and memory was still at a premium on my virtual machine host. I was hoping to save a few MB by sticking with 2003. I'm sure 2000 Server is pretty similar (and even smaller), though I have never set up IAS on that platform.

The first step is to install Internet Authentication Service (referred to as IAS from hereon out). Ensure that you have your Server 2003 installation CD handy. Go to Start, Control Panel, and launch the Add or Remove Programs applet. Along the side of the applet, there will be a button called Add/Remove Windows Components. Launch that. In the Components box, highlight Networking Services and then click on Details. Scroll down until you find Internet Authentication Service and select it. Choose OK, then click Next. That’s it, IAS is now installed and ready to be configured.

Now let’s launch the IAS Control Panel. Depending on the configuration of your server and your preferences, you can go to Start > Administrative Tools > IAS. Once it’s started, you’ll see a window such as the below screenshot. This is where you'll be doing all your RADIUS server configuration.

Next we want to add the clients that will be allowed to authenticate. Right click on RADIUS clients and then select New RADIUS Client. You will get a dialog box that pops up with allows you to enter the information for the client. For Friendly Name, enter a string to identify the device. It will probably be a good idea to enter the hostname of the device, especially if you are going to enter dozens of routers and switches. In IP Address, enter the IP address of the device. You want to enter the IP Address that will be seen in the source address of the packet being received by Windows Server. In the Client-Vendor drop down list, select Cisco. In Shared Secret, enter the RADIUS password to be used with this device. Enter the same password again in Confirm Shared Secret, and you're done. Click OK to complete the configuration. Repeat these steps for each additional device you wish to authenticate to this server.

Next, you’ll want to choose users who will be allowed to authenticate via RADIUS. You can go with existing users, or you can create new users here. It doesn’t matter if you want to use local users or Active Directory users, the process really isn’t that different. You just need to add the users to a group which you'll be using later.

Right click on Remote Access Policies and select New Remote Access Policy. Click next through the welcome screen. You'll now be at the Policy Configuration Method screen. Select Set up a custom policy, give it an appropriate name and click next. You're now at the Policy Conditions window. Click Add. In the Select Attribute window, scroll down to "Windows-Groups" and select Add. You'll now get a window called Select Groups. From this location indicates where you'll be selecting the group from, the local machine or a domain. If you want to use a group on the local machine, this should be the computer name, otherwise it should be the name of the domain. In the large white box below that, enter the name of the group and hit Check Names. If all is well, you will see the group listed in the form "Computer\GroupName." Hit OK. You'll be back at the policy conditions box and your policy conditions will say something to the effect of Windows-Group matches "Computer\Group." Hit next, Grant remote access permission, hit next again and you'll be at the profile window.

Hit Edit Profile. You'll be at the Edit Dial In Profile window seen here. Uncheck all authentication methods except for unencrypted authentication and click apply. Now select the advanced tab. In the box, select Service-Type, and change the value to Login. Click OK, and now remove the Framed-Protocol option. Click Add to add a new option. Scroll down and find Vendor-Specific and click add. Click add and select Cisco. Select Yes, It conforms. Complete the window as follows: Vendor assigned attribute number - 1. Attribute Format - string. Attribute value - shell:priv-lvl=15. This string will be used by IOS to determine a privilege level for the user once authenticated to the device. OK your way back out to the Edit dial-in profile box, which should now appear as follows:

Click OK and then a couple Next's to finish up.

Now back at the IAS window, select Remote Access Policies,right click on your policy, and select Move Up until it is the first policy in the list. You have now completed setting up IAS to serve as a RADIUS server for all of your devices.