Why is Everyone Upset with RadioShack?

The following is a position paper that I wrote in April of 2015 To set the timeline, this was merely weeks if not days after RadioShack announced that it was selling it's customer information database, which came shortly after it's bankruptcy. You know, that database that was assembled with the information demanded of you at the register every time you stopped in to grab a pack of batteries. This is in spite of their policy that they would never sell that information without your consent (emphasis mine).

Symmetric Traffic and IPS

A well known problem for network and security professionals in the enterprise is asymmetric routing.  At it's simplest, this is where traffic flows outbound through Router A, while the return traffic returns through Router B, or through both Routers A and B.   If you're using a reflexive ACL, for example, this will lead to some, if not all of the return traffic being blocked as it attempts to return through Router B.  This is due to Router A having a record of the outbound traffic while Router B does not.  Riverbed breaks this down into several sub-categories such as complete asymmetry, server-side asymmetry, client-side asymmetry, and multi-SYN retransmit.  But for our purposes here, it's all asymmetric, and it's all a bad thing.  While some firewalls are able to share state to avoid this situation, not all do.  And Cisco Routers running IOS do not.

While asymmetric routing is known to be a problem at the network edge, it can be a problem for security professionals internally as well.  And the larger the network is, the more likely asymmetric traffic is to occur at some level.  When you deploy an IPS sensor in the network, it must be able to see all traffic in both directions for maximum effectiveness.  When an IPS sensor is able to see all the traffic involved in a particular session, you get better threat detection, reduced susceptibility to IPS evasion techniques, and less susceptibility to false-positives and false-negatives. 

While it cannot be completely avoided at the enterprise edge, the good news is that internally, steps can be taken to reduce if not eliminate the effects of asymmetric routing.  So good network design is a must to get the maximum effectiveness of an IPS deployment, particularly if there are going to be multiple sensors along a given traffic flow.

There's a few options to ensure symmetric traffic flows, or to mitigate the effect of asymmetric traffic flows including:

• Duplicate traffic across multiple IPS sensors to ensure each sensor can see all applicable data.  In addition to the challenges presented in getting all the relevant data to each IPS, we also have a greater likelihood of overloading IPS sensors with traffic, which will result in packets being dropped.
• Integration of an IPS switch.  This is reducing traffic down to a single switch.  While it is better from an IPS standpoint, it's introducing a single point of failure into the network.
• Correctly configuring spanning tree parameters to ensure symmetrical paths across Layer 2 areas.
• Routing manipulation with techniques such as PBR. This is a cost effective solution as it involves only configuration changes rather than additional hardware.  But it adds complexity to the network in addition to requiring cooperation between security and networking. 
• Sticky load-balancing utilizing technology such Cisco's ACE module or Riverbed's Asymmetric Routing Detection to better reduce the chances of asymmetric routing.
• In cases of HSRP induced asymmetry, utilize EEM and EOT in order to change the paths of HSRP related routes dynamically.
• Configuring firewalls as active/standby pairs rather than active/active pairs.

But as you can see, many of these techniques involve taking redundant data paths out of the equation, and therefore reducing the amount of overall usable bandwidth across the network.  Others involve sending more data to or through each IPS unit, increasing the burden on each unit and increasing the likelihood of dropped packets.  So there is obviously a balancing act between performance and visibility.

The Accuracy of Sampled Netflow

To alleviate the fear of overburdening the CPU due to the collection of NetFlow statistics, Cisco gives us the option of using Sampled NetFlow. Sampled NetFlow allows you to sample 1 out of 10 packets, 1 out of 100 packets, or however much of a subset of the total number of packets. The theory is that with a good sample, the traffic will still be indicative of what is flowing through the router. If 10% of the total amount of packets following through the router is DNS queries, for example, then approximately 10% of the total amount of packets in the sample will also be DNS queries, and so on.

The reason that this is necessary is because of the way that a router handles traffic when collecting NetFlow statistics. In order to process a packet in order to collect NetFlow statistics, that packet has to be processed by the CPU. When sampling is enabled, the packets that are not part of the sample are switched faster because they will not require the additional processing required.

NetFlow sampling is enabled on supported IOS platforms with just a few commands.

ip route-cache flow sampled

ip flow-sampling-mode packet-interval 100

NetFlow sampling can be monitored with the show ip flow sampling command.

So as you can see, NetFlow sampling is simple to configure and monitor. It only takes a couple commands. But the question now needs to be asked, how indicative of the total network traffic is the sample? In other words, if I’m seeing 10% of all traffic being DNS queries in my sample, is 10% of the total traffic flowing through this router really DNS queries? Or is there some significant level of error in the sampling? In Cisco documentation and Certification Exam Guides, it is admitted that the sample will never be 100% accurate, but that it should usually be pretty close. They’ll also mention that you should obviously check the accuracy periodically.

Recently, I came across an academic article talking about the accuracy of NetFlow sampling. In the article, they collect data over time with a 1 in 250 packet NetFlow sample and compared it to a raw traffic sniff utilizing tcpdump. Shown below is Figure 8 from the article, which summarizes their findings. The red dotted line shows real time data of traffic flowing through the router, while the solid blue line shows real time data of their 1 in 250 NetFlow sample.

The article states that "In Figure 8 the cumulative empirical probability is plotted with its relative error. It indicates that the performance of systematic and static random sampling is not distinguishable in practice. We believe it is true in most of backbone links where the degree of multiplexing of flows is high."  In other words, the sample is really indistinguishable from the full data set.  Equally of importance, they found that the processing overhead of NetFlow sampling to be insignificant.  Further accuracy of their collection methodologies is demonstrated by SNMP byte count data strongly correlating with NetFlow byte count data.  There's a lot of statistics and graphs in the article if you're into that sort of thing.

Conversely, in another academic article, the researchers found their sampling to be significantly less accurate.  They stated that "Our experimental results allow us to come to the conclusion that: (i) our traffic classification method can achieve similar accuracy than previous packet-based techniques, but using only the limited set of features provided by NetFlow, and (ii) the impact of packet sampling on the classification accuracy of supervised learning methods is severe."  They discuss a training process which gets their accuracy to 85% for a 1/100 sampling.  Good enough for most use cases, but still too manual and still still a far cry from the results of the first study.

So where do we stand with Sampled NetFlow accuracy?  One study says it's pretty accurate, and the other says not so much.  So the jury is still out, and we're back to Cisco's recommendation that you should be testing the accuracy to determine if it is good enough for your use case.  Like the team in the first article, you can easily use a network tap or SPAN port to compare what is actually coming out of a router interface with the NetFlow sample estimating what is coming out of that router interface.  Don't just assume.

IOS Zone Based Firewall

One of the most commonly covered security features when it comes to Cisco security is the ZBF.  It wouldn't be much of a network security blog without at least one post on this topic, so here's my take.

With IOS version 12.4(6)T, Cisco introduced the Zone Based Firewall (ZBF), sometimes referred to as the Zone Policy Based Firewall.  With this, the Classic IOS Firewall or Context-Based Access Control (CBAC), available since IOS version 11.2, is now deprecated. Nearly all of the features of the Classic IOS Firewall are implemented in ZBF as well as wide range of new features. In addition to the new features available in ZBF, it is also said to improve firewall performance over CBAC for most inspection activities.  I've seen it stated in some places that if you attempt to inter-mingle CBAC configuration commands with your ZBF, it MIGHT work, however most documentation states that it wont.  So I wouldn't risk it.  Choose one or the other.

Server 2003 IAS RADIUS Server

Since I'm sure many home labbers are still rocking Server 2003, I'll put it up in hopes that someone will still find it useful. This post was originally done a number of years ago when Server 2008R2 was still new and memory was still at a premium on my virtual machine host. I was hoping to save a few MB by sticking with 2003. I'm sure 2000 Server is pretty similar (and even smaller), though I have never set up IAS on that platform.

The first step is to install Internet Authentication Service (referred to as IAS from hereon out). Ensure that you have your Server 2003 installation CD handy. Go to Start, Control Panel, and launch the Add or Remove Programs applet. Along the side of the applet, there will be a button called Add/Remove Windows Components. Launch that. In the Components box, highlight Networking Services and then click on Details. Scroll down until you find Internet Authentication Service and select it. Choose OK, then click Next. That’s it, IAS is now installed and ready to be configured.

Now let’s launch the IAS Control Panel. Depending on the configuration of your server and your preferences, you can go to Start > Administrative Tools > IAS. Once it’s started, you’ll see a window such as the below screenshot. This is where you'll be doing all your RADIUS server configuration.

Next we want to add the clients that will be allowed to authenticate. Right click on RADIUS clients and then select New RADIUS Client. You will get a dialog box that pops up with allows you to enter the information for the client. For Friendly Name, enter a string to identify the device. It will probably be a good idea to enter the hostname of the device, especially if you are going to enter dozens of routers and switches. In IP Address, enter the IP address of the device. You want to enter the IP Address that will be seen in the source address of the packet being received by Windows Server. In the Client-Vendor drop down list, select Cisco. In Shared Secret, enter the RADIUS password to be used with this device. Enter the same password again in Confirm Shared Secret, and you're done. Click OK to complete the configuration. Repeat these steps for each additional device you wish to authenticate to this server.

Next, you’ll want to choose users who will be allowed to authenticate via RADIUS. You can go with existing users, or you can create new users here. It doesn’t matter if you want to use local users or Active Directory users, the process really isn’t that different. You just need to add the users to a group which you'll be using later.

Right click on Remote Access Policies and select New Remote Access Policy. Click next through the welcome screen. You'll now be at the Policy Configuration Method screen. Select Set up a custom policy, give it an appropriate name and click next. You're now at the Policy Conditions window. Click Add. In the Select Attribute window, scroll down to "Windows-Groups" and select Add. You'll now get a window called Select Groups. From this location indicates where you'll be selecting the group from, the local machine or a domain. If you want to use a group on the local machine, this should be the computer name, otherwise it should be the name of the domain. In the large white box below that, enter the name of the group and hit Check Names. If all is well, you will see the group listed in the form "Computer\GroupName." Hit OK. You'll be back at the policy conditions box and your policy conditions will say something to the effect of Windows-Group matches "Computer\Group." Hit next, Grant remote access permission, hit next again and you'll be at the profile window.

Hit Edit Profile. You'll be at the Edit Dial In Profile window seen here. Uncheck all authentication methods except for unencrypted authentication and click apply. Now select the advanced tab. In the box, select Service-Type, and change the value to Login. Click OK, and now remove the Framed-Protocol option. Click Add to add a new option. Scroll down and find Vendor-Specific and click add. Click add and select Cisco. Select Yes, It conforms. Complete the window as follows: Vendor assigned attribute number - 1. Attribute Format - string. Attribute value - shell:priv-lvl=15. This string will be used by IOS to determine a privilege level for the user once authenticated to the device. OK your way back out to the Edit dial-in profile box, which should now appear as follows:

Click OK and then a couple Next's to finish up.

Now back at the IAS window, select Remote Access Policies,right click on your policy, and select Move Up until it is the first policy in the list. You have now completed setting up IAS to serve as a RADIUS server for all of your devices.

ACLs by Country

Have you ever wanted to create an ACL by country?  There's a number of different ways you can go about it.  Certain models of firewalls have this functionality built in.  IOS based routers and ASA firewalls have no such capability, so we'll have to do this a bit more manually.  I'll present two methods.

The first method is detailed here.  In this post, wget goes out to the Internet and grabs the necessary data from the applicable RIR.  Some custom Perl code pulls out the subnets associated with that country and then builds the ACL.  This one is probably not for the faint of heart nor someone not very fluent in Unix command lines.

A simpler way is through a website called Country IP Blocks.  Navigate to The Create Country ACL page on their site and you can select one or more countries to build an ACL for.  Then pick which format you want the results in.  Cisco ACL is just one of 12 options here, and then click "Create ACL" and you're done.  Other sites such as ipdeny.com and countryacl.tools.uebi.net provide similar functionality.

These lists get pretty long. Want one that that will permit or deny United States based addresses?  That'll be 55,348 lines.  Want to create an ACL that will block Russia and China?  That's 27,386 lines.  Hope your router is maxed out on RAM.

TCL Scripting

According to it's man page, "tclsh is a shell-like application that reads TCL commands from its standard input or from a file and evaluates them. If invoked with no arguments then it runs interactively, reading TCL commands from standard input and printing command results and error messages to standard output. It runs until the exit command is invoked or until it reaches end-of-file on its standard input."  The TCL Developer Xchange describes the TCL language as  "a very powerful but easy to learn dynamic programming language, suitable for a very wide range of uses, including web and desktop applications, networking, administration, testing and many more. Open source and business-friendly, TCL is a mature yet evolving language that is truly cross platform, easily deployed and highly extensible. "

The language was created John Ousterhout at the University of California, Berkley.  It's either installed by default or available through the package repositories in nearly every Linux distribution and flavor of BSD.  ActiveState maintains an edition called ActiveTcl.  The community edition has precompiled binaries for Windows, Mac and Linux.  The Enterprise Edition adds binaries for HP-UX, Solaris and AIX.  ActiveState also the home of ActivePerl and ActivePython, which are solid editions of Perl and Python for the same platforms.

Tclsh was added to Cisco IOS in version 12.3(2)T and 12.2(25)S, and to Cisco NX-OS in Release 5.1(1) to provide scripting capability.  With it, you are able to run TCL commands directly from the Cisco IOS prompt, or to create and execute scripts written in the TCL language.  Just about anything you can do in tclsh on a Linux or BSD system can be done in tclsh on a Cisco router.  This of course assumes you're using straight TCL and not any add-on packages.

To use tclsh, simply type the command tclsh at the exec prompt.  To exit tclsh, type tclquit.  While within tclsh you can create scripts with the proc command, by typing proc script_name {, and then ending your script with a closing }.  A great example of the power of this scripting environment can be found in this post at INE, where Brian McGahan, along with an assist from reader Jason Cook, demonstrates a TCL script to generate a number of random IP addresses and subnet masks tied to Loopback interfaces.  I've used this script several times in the lab to quickly add routes into a routing protocol.  If you're feeling really adventurous, you can even get your router to Tweet.  And that's the beauty of scripting, you are able to quickly and easily automate the mundane tasks that aren't what you are working on and interested in, but still need accomplished.

And though it is not suggested, you can change your login shell on a Linux, BSD or Unix system to tclsh and do your day to day work in it as a means of learning the language and environment.  However, as noted, its not suggested as it's not really suited as being used a login script wasn't in the design goals.  See this article for more details.

I'll be getting more into TCL on IOS in the near future.

Some Good References to Get Started:

The official reference from Cisco:
Cisco IOS Scripting with TCL Command

The Cisco book:
Tcl Scripting for Cisco IOS

Some books that come recommended by the TCL Developer Exchange:

Practical Programming in Tcl and Tk, 4th ed.
Tcl/Tk, Second Edition: A Developer's Guide
Tcl and the Tk Toolkit, 2nd ed.
Tcl/Tk 8.5 Programming Cookbook

Hard Code DNS Servers with PowerShell

The following is a PowerShell script to quickly hard code DNS servers for every network interface present on a computer. It will overwrite the existing DNS servers configured on that machines interfaces.  In this example, we'll be using the IP addresses for OpenDNS servers.

# The servers that we want to use
$newDNSServers = "208.67.220.220","208.67.222.222"

# Get all network adapters that already have DNS servers set
$adapters = Get-WmiObject Win32_NetworkAdapterConfiguration | Where-Object {$_.DNSServerSearchOrder -ne $null}

# Set the DNS server search order for all of the previously-found adapters
$adapters | ForEach-Object {$_.SetDNSServerSearchOrder($newDNSServers)}

Do the Google

It still surprises me that in 2016, there are still people out there who cannot, or will not, use Google to find the answer to their question.  I'm the admin of a large group on Facebook that exists primarily for people pursuing the CCNA certification, though most technical discussion that stays on the right side of the law is permitted.  Since we get at least one question a day that involves something that could be solved in 10 seconds with Google such as "what is the CCNA?" or "what is Spanning Tree?" I've decided to put together a page explaining how to use Google to find what you're looking for.  And for the Microsoft slappies out there, you are more than welcome to try that other "search engine" as long as you're enjoying it's crap results.  I for one very rarely got good results when I tried it a couple times in the past.

In case you were wondering, the title of this post, "Do the Google" came from a poster on Reddit by the name of /u/sajaschi, in the Tales from Tech Support sub-reddit, that he heard from a family member who's computer he supported.   I liked the term and immediately told him that I'm stealing it. The thread is here for completeness.

Resequencing an ACL

Here's a quick post on a very useful command when working with ACLs.  I first heard about it while watching a CBT Nugget video, and I can say that it was definitely not covered in the NetAcad curriculum when I went through the classes, because I remember bringing it up to the instructor and it was news to him.

So let's begin by setting the scenerio.  You have the following ACL:

show ip access-list EXAMPLE

Extended IP access list EXAMPLE
    1 permit ip host 10.10.10.28 any
    2 permit ip host 10.10.10.10 any
    3 permit ip host 10.10.10.11 any
    4 permit ip host 10.10.10.45 any
    5 deny ip 10.10.10.0 0.0.0.255 any
    10 permit tcp any host 192.168.10.4 eq smtp

And let's say that we now need to allow one additional host out.  We could rewrite the ACL, but that could be a lot of work if its a long ACL.  Any other options?

Yes, the resequence command can help.  This command was introduced in IOS 12.2(14)S, and allows you to easily resequence an entire ACL.

ip access-list resequence EXAMPLE 10 10

This will renumber every line in the ACL starting with 10, and with an increment of 10 between each line.  This is the default sequencing for an access-list where no sequence numbers are entered.  The end result would be:

Extended IP access list EXAMPLE
    10 permit ip host 10.10.10.28 any
    20 permit ip host 10.10.10.10 any
    30 permit ip host 10.10.10.11 any
    40 permit ip host 10.10.10.45 any
    50 deny ip 10.10.10.0 0.0.0.255 any
    60 permit tcp any host 192.168.10.4 eq smtp

Old documenation will tell you that you can't edit a numbered ACL, but that's actually not true anymore.