Default User Role in Firepower

When I took over the Firepower system at work, we had a long standing issue relating to a couple of users. Everyone in Security Operations is an admin in the FMC, however, one of our SecOps guy could not see all the events in the FMC. He can see everything in the SIEM, just not in the FMC, verifying that this is indeed a FMC issue.  2 separate TAC cases had been raised before, neither of which got to the bottom of the matter.

One of the first tasks I had in Firepower was adding in the remaining IR guys who did not have accounts yet. One if the IR guy immediately noticed the same behavior as the previous mentioned SecOps guys. He could see some events, just not all of them.

Here's what happened. In the FMC, we had a default user role defined for RADIUS users under External Authentications. This user role had Restrictive Search enabled everywhere possible in the user role. User role permissions are cumulative, however it appears that Restrictive Search overrides everything.

Because this is the default role, it gets applied to all RADIUS authenticated users upon login. Even if you uncheck the role from their user profile later, it will get rechecked the next time they login. You can see that this is the case when you look at the user profile and see that the role is marked Externally Set.

We were hesitant to change the default user role to Security Analyst (Read Only) which is what the IR guys were getting because the more restrictive role was set up for users outside of the SOC to view events related to their devices and it was feared that a new non-SOC user in the future would accidentally get more access than they required.

So the answer was to create a new user role with zero access, set that as the default, and then add another role on top of that be it Administrator, Security Analyst (Read Only), or the previous mentioned custom role with the Restricted Search filters.

Database Integrity Check Failed

A quick tip for Firepower when you have a database error.  This seems to come up a lot especially if you're manually importing snort rules from an outside source such as Emerging Threats.

Every once in a while you'll get a database error that will cause an operation to fail. In my case, it's the scheduled backup. The web page doesn't give you a lot go to on at first glance, but under your schedule, it should tell you more at the bottom of the calendar.

If you ssh in and run DBCheck.pl, you'll see something like this:

[Tue Feb  5 13:15:50 2019][FATAL]  [missing eo reference]    database [sybase],table [rule_header] column [uuid], uuid [fcd5fc7a-2889-11e9-ab40-b1c5df20061d],rev_uuid [NULL]

indicating a bad database entry for a snort rule.

This is easily fixed by typing 

OmniQuery.pl -db sdb -e "delete from rule_header where uuid ='fcd5fc7a-2889-11e9-ab40-b1c5df20061d';"

Pay close attention to the table (rule_header in this case). I've also seen EOPermissions, EORevisionStore, and EOContainerStore (or all 4). 

Again, this seems to only be a problem here for the Emerging Threats rules that I import, I've never seen it happen for the default rule set downloaded and installed automatically by Firepower.

TAC also likes to suggest the "slow repair,"

for i in `ls -1|egrep -v "\.BAK$"|sed 's/\....$//'|sort|uniq`; do repair_table.pl -frames $i;done 

but I've never had that actually fix anything. Note here that you have to actually be in /var/lib/mysql/sfsnort for this to actually do anything despite TAC's insistence that it can be run from anywhere in the file system.

Chicken Chicken

Chicken Chicken Chicken Chicken.

https://isotropic.org/papers/chicken.pdf

My New Favorite ISE Setting

And by favorite, of course I mean least favorite ever.

This one has been a thorn in my side for a while now. In the User Password Policy (Administration > Identity Management > Settings > User Password Policy), under Password Lifetime, there's a setting called "Disable user account after ____ days if password not changed." This setting is enabled by default, with a value of 60 days.

What happened is that I set up a pretty weak password policy for convenience including passwords that do not expire (it's a lab environment, and I only have the one local user account in ISE) but seemingly out of the blue, the account would become disabled. At one point, I even deleted the account and created a new one. Who notices it's been exactly 60 days? I'd reenable the account, and the next day it would be disabled again. The only thing I use this ISE local account for is the automated testing feature on my 802.1x enabled switches. This account becoming disabled leads to a long list of nasty red failed authentications in the Live Log.

For reference, here is the setting:

802.1x in GNS3

After tiring of the trips up and down the stairs dealing with a physical 3750 switch and Windows 7 workstation testing 802.1x, I decided to give it a shot virtually. And I'm pleased to report that it works like a champ.  This post is not a how to guide on setting up 802.1x, it's to show another complex configuration topic that can be done exclusively in the virtual domain.  And while I've never done it personally, I've seen elsewhere that it's also possible to do in EVE-ng.

I setup a basic topology in GNS3. In the light blue oval is the connection to the physical network.  The cloud connects to VLAN10 on my physical 3750 switch, which is the home network. There's a generic GNS3 switch there just in case I want to tie anything in at that point in the future, but at this point it's unnecessary and not really being utilized. Next we have BB2, which is a IOU router (i86bi-linux-l3-adventerprisek9-15.4.1T.bin if you care) that provides a degree of separation between the "real network" and the test area. SW11 is a L2 IOU image (i86bi-linux-l2-adventerprisek9-15.2d.bin) playing the role of the supplicant in the chain.  And finally we have fn-ws70008, which is a domain connected Windows 7 workstation which runs in qemu on the GNS3 VM.  The Yellow section branches out into another area of the topology that isn't relevant to this discussion.

Be advised that using a nested VM like this (qemu VM running inside of the GNS3 VM) is horrible from a performance standpoint. However the performance is not bad enough that delays lead to timeouts, so I can live with it for now.

At this time, I'm using the Windows 7 VM to confirm both 802.1x and MAB operation. I had a basic VPCS host that I tried to use for MAB, but for whatever reason, the switch didn't want to play nice with that host so I just set it to try both but prefer 802.1x on the Windows 7 workstation so I could see both play out in the ISE logs. If you can get a MAB authentication to trigger with a VPCS host or a router, I'd love to hear about it.

Turning our attention over to ISE, the next 2 images show the basic NAD configuration for SW11. Nothing out of the ordinary here, straight out of the documentation.

 


The switch configuration comes straight from the SISAS certification guide in Chapter 12.  You can also find a great stand alone PDF right here from Cisco.  As far as I recall, everything in the cert guide and the PDF work in this version of the L2 IOU image except for LLDP device tracking (Edit: its working now with no changes on my part.  No idea why it didn't before). CDP and DHCP tracking work, so that's definitely not the end of the world.  One caveat to keep in mind is that ip routing is enable by default on these switches which led to a short session of head scratching when I couldn't reach anything off of the local network from the switch the first time I started seriously labbing with it.

From what I can tell, the L2 IOUv works great as well, however I haven't labbed to seriously with it yet since the L2 IOU image does everything I've asked of it and I've been using it since before IOUv was a thing. As another note, there is a significant syntax difference with IOU 15.6 so I'm holding off on that one as well.

Configuring ISE for 802.1x is also straight from the SISAS certification guide, so no need to beat a dead horse rehashing it here.  Chapters 10 and 11 cover AuthZ and AuthC policy creation.  So with all that completed, let's check the Live Log in ISE. For the record, I'm running ISE 2.0 with no patches applied.

Here we'll see the workstation performing MAB and 802.1x authentication successfully. You'll also note the switch doing the automate-tester authentication with the local-bob account. For ease of use, I have an AD account called domain-bob, a local account on ISE called local-bob, and each router has a username called router-bob. Thanks to Keith Barker inspiring me to name all the users Bob, though I at least saw the potential for confusion with both the domain user and ISE user being just "bob."

And as one last show of compatibility, I've configured sw11 to send syslog to my local Splunk server. I do use the ISE interface for debugging as much as possible while labbing, as you should when you're trying to learn everything about ISE, having one location for long term log retention for troubleshooting purposes is very valuable when the problem involves multiple devices, and/or you want to look back at the messages for a similar situation that did work.

BB2 is also configured with ip-helper to forward DNS requests to one of my Windows domain controllers, both of which exist outside of this GNS3 topology. That also works flawlessly. 

So bottom line is that I'm really liking these IOU images (both L2 and L3) and they're almost fully meeting my labbing needs.  Since more recent ISE versions appear to work in GNS3 as well, you could theoretically work through the SISAS almost completely in GNS3 at this point. Connecting the dots in the exam topics tell me I still want to know ACS, though I cannot say for sure. It's not specifically listed on the topics and I haven't taken the exam yet. But the exam is written for ISE 1.2 (which doesn't have TACACS+ support) and the topics do say implement device administration with both TACACS+ and RADIUS.

Next up, 802.1s on NX-OS using the Cisco Nexus 1000V switch image, though I'm not sure when I'll get around to that. Nexus switches do not appear anywhere in the security track (yet) but I do want some exposure to them. Maybe I can integrate a 1000V image into ESXi and ditch the nested Windows 7 VM.

Troubleshooting With Near Zero Access

Early one morning last week I attempted to RDP into my lab to test something out I was looking into at work. Access to my terminal server was fine, but from there, I was unable to access any other system on my network. Every system that I attempted to RDP into came back stating that my user account was unauthorized for RDP access on that system. The user is a Domain Admin so there should be no reason for that. Not too long after, I noticed that the terminal server was asking for a username and password for everything with is out of character for my user account. And after accounting, I get access denied errors for anything requiring elevated privileges. My first thought was that my network was compromised.

Check for Updates

If you're a Windows 10 user, you might want to think twice about clicking on the Check for Updates button.

http://disq.us/t/39kbbzc

New FAQ

1. What kind of Questions should I expect to see on the CCNA exam?

All the information that is known about the exam is available on the Cisco Certifications pagerelating to the exam in question. The Cisco NDA forbids us from telling you anything more. This includes how many of each type of question we saw, whether or not a specific topic is covered heavily, and what specific questions we were asked.

2. How much does the CCNA exam cost?

You can find the cost of the exam in your country at the above mentioned site where you register. Again, don't take anyone else's word for it. You're going to pay Pearson Vue to take the exam, not me. So it only makes sense to see what they say it costs, right?

3. How do I register to take the exam?

Cisco exams are proctored Pearson VUE. You will find everything you need to know about registration and exam locations on their site. Don't just take the word of some random person on Facebook, get the information directly from the authoritative source.

4. What book should I read for the CCNA?

I've blogged about this before very recently, and you can find that post elsewhere on my blog. But to summarize, I'm always going to recommend the Official Certification Guide from Cisco Press. I don't care if someone said Wendell Odom is too dry, or the books are too long. There's a lot to know, and you may as well understand now that there are no shortcuts.

Many like Todd Lammle's book because it's shorter and it reads more like a stand up comedy routine.  First, Lammle assumes you have a certain level of knowledge already before you pick up his book.  Odom doesn't.  Lammle gives you the bare minimum needed for the exam.  Odom doesn't.  If passing the exam is your only priority, you're in for a rude awakening on your first day on the job. The router that's down isn't going to go easy on you because the root cause wasn't a CCNA exam topic.

You can find a large list of free resources around the Internet to assist your studies.

5. I have a question about my final results.

There's a great post over at the Cisco Learning Network titled Demystifying the Cisco Score Report. It says it's posted by the Certs & Lab Admin, so I'm going to assume this is legit. And while this particular breakdown is for a CCIE Written exam, pretty much everything that is said in the post applies to any written exam.

6. What degree should I get for networking?

Nobody can speak for every hiring manager out there. Every one is different, so there is no answer for this. In my experience, a college degree was a yes/no question without too much focus on what the degree was in. Some hiring managers don't care about a degree while others only care that you have one, and others still want it to be something specific. I chose a degree that closely matched the work I wanted to do in the future. You can't go wrong with that way of going about it. One thing I can guarantee is that nobody is looking specifically for somebody with a masters in Botany to work in their NOC.

7. Why did I get banned from the CCNA group?

The group has rules, and the rules are enforced. Its nobody else's fault that you either posted without knowing the rules, or you knew the rules but chose to disregard them. Every Facebook group, Website, Online Forum, etc. has rules and it's up to you, the user, to seek them out and follow them. This is true with anything you do in life.

8. Will you do my homework for me?

No, we absolutely will not do your homework. Do your own homework. I already know the material and your homework is one of the steps involved in your education. You want to pas the exam and earn the certification? Then you need to learn this stuff, not just write down what I told you. Obvious homework questions will be deleted, and the poster will be mocked. I don't care if you say it's not homework, we know homework when we see it.

Besides that, half of the "answers" I see given to homework questions are wrong anyway.

9. Which certification pays more, Cert A or Cert B?

Follow your heart, not your greed. If you want to strike it rich, maybe come up with the next Google or Facebook.  Every certification path has great opportunities, and you'll be paid well in any IT specialty if you do your job well. And you will do your job a lot better if you're doing something that you actually enjoy doing. If you don't know what you enjoy doing yet, you should be spending more time getting a job and gaining experience and less time worrying about 3 jobs ahead.

Bottom line, if it wasn't a marketable job skill, Cisco wouldn't bother maintaining a certification program for the skill.

10. What is router?

This question, and any basic question like it, doesn't have any place in a Facebook group or an online forum. It's something that you can easily Google, and not waste everyone's time. In case you're still unsure how, I wrote a post on how to Google things. Try it sometime, you may like it.

"What is the CCNA" is especially offensive. Why did you join our group if you don't even know what it is? If you don't know, then Google it.

When in doubt, try this post on how to ask better questions. Better questions get better answers. Take note when you will not show any router configs but still want us to figure out why can't ping across your 5 router topology.

11. Any question that conntains "Class A," "Class B," and/or "Class C".

Let me stop you right there. Classful networking is a relic of ancient days, and we simply don't use it any more. CIDR and VLSM aren't something that's off in the distant future (insert IPv6 or Duke Nuke 'em Forever joke here), they are the here and now. If your "practice test" or study guide that you obtained from some shady website talks about classful networking, you're going to fail.

12. I don't like the rules, who can I complain to?

Basically nobody. The admins set and enforce the rules. And as long as we do not violate any part of Facebook's TOS, that will not change. If you dislike the group that much, there's certainly other groups elsewhere that you would like better. Try one of them.

Of course, if you do want to scream at somebody, then by all means send me a private message on Facebook, even if it was a different admin that annoyed you. I've actually grown to enjoy the abuse.

13. I was treated unfairly by an admin, who can I complain to?

See question #12. I have nothing but respect for the other admins of this group, otherwise they wouldn't be admins. It's not very likely I'll take your side over them.

But again, feel free to scream at me all you want on Facebook.  I love the abuse.

14. Will the CCNA help me find a job?

The same goes for this as it does for the degree question. While experience trumps all, certifications and a degree will help you stand out from other similar candidates who you are otherwise similar to. But at the end of the day, it's up to the individual company to decide what they want in a candidate. Some want a degree, some want certifications, some want something else still. The job posting will tell you exactly what they're looking for in a candidate.

The bottom line is that having a degree and a CCNA helped me get my first job in I.T. Everyone had zero experience at one point, and yet the senior members of the group are all working in the field, right?

15. I has cert, job me now!

Sorry, but this isn't a jobs board, and these types of posts can and will drown out everything else if we allow them.. Since there's really no such thing as a "CCNA Job" despite what many posters say, there's no connection here. If you wish to explore career opportunities in your area, seek out a group dedicated to that topic.

16. I'm the network administrator or engineer and everything is broke. Help me!

Let me guess, you passed the CCNA by using Testking, and somehow lucked your way through the job interview. The hiring manager must have figured they could pay you less since you really didn't interview well. Now you see why you shouldn't have cheated. You can't afford my hourly rate as a consultant, so pick up a book and start learning. You're in for a long night.

That'll Never Work!

While reading through a discussion on Slashdot today, someone linked a thread from 1999 as proof that the Slashdot community has always been pretty dumb collectively.  The thread was on the upcoming release of VMWare, a class of products that clearly hasn't been heard of on PCs at the time.

The thread can be found here.  It's amazing how a concept so ingrained today appears to be so foreign just 18 years ago.  My personal favorite is the very bottom comment "a great idea ROFL."

Moving to IPv6 in the Lab

IPv6 is one of those technologies that I've been wanting to dig into further.  I know enough that I can get through the certification exam of the day with a little book time to refresh, but I don't know it well enough.  It's not something I've been avoiding, just something that I've kept putting off because something was more pressing, more interesting, or potentially more useful.  So there's no time like the present. Let's get started.

I began by reconfiguring the network to better align with all the blog posts and docs that I've read to date.  I originally had the 3750 doing the intraVLAN routing, but I decided to simplify and push everything out to the 2821 at the edge for now.  So the 2821 and 3750 are doing router on a stick.  There are 2 VLANs I'll be using (10 and 20 for now, additional VLANs are there but not IPv6 enabled yet), so the /60 Comcast is currently handing out that can be broken down into 16 /64's will suffice.  I think a lot of areas are getting more than a /60, but it's more than enough for now.

On the 2821, we'll start by enabling ipv6 routing.  Naturally, the commands are a bit different here and there.

ipv6 unicast-routing
ipv6 cef

Then on the outside interface, we'll pull our /60.  If your ISP is handing out bigger chunks, adjust your hint accordingly.

interface GigabitEthernet0/1
 ipv6 enable
 ipv6 address autoconfig default
 ipv6 dhcp client pd hint ::/60
 ipv6 dhcp client pd COMCAST

First we enable ipv6 on the interface and then pull a /60 and put it into a pool called COMCAST.  In a lot of other docs online, I see the addition of "ipv6 address dhcp" added on the outside interface as well.  But my router/IOS combination wouldn't take that command and it's working fine without it, so keep this in the back of your mind.

Next, we'll go onto the inside interfaces.  We'll set up the IPv6 addresses and have a little ROAS review here too.

interface GigabitEthernet0/0
 no ip address
interface GigabitEthernet0/0.10
 encapsulation dot1Q 10
 ipv6 address COMCAST ::1/64
 ipv6 dhcp server COMCASTPOOL
interface GigabitEthernet0/0.20
 encapsulation dot1Q 20
 ipv6 address COMCAST ::2:0:0:0:1/64
 ipv6 dhcp server COMCASTPOOL

What we've done here is put the first /64 from the COMCAST pool onto VLAN 10, and the second /64 onto VLAN 20.  The next line on the interface sets up the dhcp options for the two VLANS.  The only options that I've currently configured are the DNS servers.  I'm actually using my own Domain Controllers (which is what you should use if you have them), but for here I'll put in Google's.  There's some timers that may need tweaked in regards to neighbor discovery, but that's a little beyond my understanding at this point.  I'll get into that at a later date.

ipv6 dhcp pool COMCASTPOOL
 dns-server 2001:4860:4860::8888
 dns-server 2001:4860:4860::8844

So now we have full IPv6 connectivity on just about everything in the lab (for some reason, none of my Virtualbox guests can ping past their own Ethernet NIC, but that's a topic for another day).  I've disabled IPv4 completely on a test machine (Server 2008 Enterprise) and loaded up Yahoo.

So far so good.  We've got connectivity.  The NIC settings are shown to demonstrate that IPv4 is indeed disabled.

What's next?  I would like to move intraVLAN routing back down to the 3750 and have a single routed link between it and the 2821. Then I want to move the DHCPv6 functionality for each VLAN down to the domain controllers so I can manage all the IPv6 bits with Windows IPAM as I do now with the IPv4 bits.  And finally, I need to update the IOS on my 3750 to an image that supports IPv6, among other shortcomings I'm currently hampered by.

But first things first, I'm going to move my Hyper-V servers from Server 2012r2 to 2016 and finally get them into a failover cluster.  Between that and getting some shared storage together for the cluster should get me through a good section of the MCSA 2016 topics.