Which VPN tunnel matches my traffic?

When we manage a VPN concentrator with thousands of active tunnels, we might face conflicts between crypto maps. This is not so easy to realize and we might spend a lot of time before we figure out that another tunnel is forwarding our traffic.

Here are the steps that I would use to check which tunnel matches my packets.

debug menu

ASA command reference page does not include a detailed explanation for the debug menu command, therefore I collected the details from a device CLI. It's not recommended to use this command without TAC supervision, but some of them are really useful (check debug menu ssh). Some options might not be available on the OS version that you are running.

Analyzing logs to filter traffic that hits an access list entry

If you have a generic access rule and you want to make it more specific, you can use logs to help you on filtering sources, destinations and/or services that hit the rule.

By default, ASA only logs denied packets (message %ASA-4-106023), but we can use the log option to enable logging for specific rules.

Log messages %ASA-6-106100 and access list entries hash codes are helpful to find out what traffic matches an specific rule. Let's use the following ACL as an example:

asa# sh access-list
access-list inside line 1 extended permit tcp any any eq www log informational interval 300 (hitcnt=1) 0xaa22f669
access-list inside line 2 extended permit object-group services any any log informational interval 300 0x3d0397cc
 access-list inside line 2 extended permit icmp any any log informational interval 300 (hitcnt=0) 0x55873f33
 access-list inside line 2 extended permit tcp any any eq telnet log informational interval 300 (hitcnt=1) 0x44b661f2

The ACL has two rules and log option is enabled on both. One is a single rule and the other one uses an object-group for service definition. Note that we have a hash code for each rule, but we also have one hash code for every group member (expanded rule elements).

If we want to filter logs that match the first rule, we use the following command:

!access-list inside line 1 extended permit tcp any any eq www:
asa# sh log | i 106100.*0xaa22f669
Mar 03 2016 01:11:21: %ASA-6-106100: access-list inside permitted tcp inside/10.0.0.10(48338) -> outside/192.0.2.2(80) hit-cnt 1 first hit [0xaa22f669, 0x0]

This is a single rule, thus only one hash code is present in the log message.

If we want to find out what traffic matches the second rule, we can filter for one specific object member or the group ACE:

!access-list inside line 2 extended permit tcp any any eq telnet:
asa(config)# sh log | i 106100.*0x44b661f2
Mar 03 2016 01:12:09: %ASA-6-106100: access-list inside permitted tcp inside/10.0.0.10(44036) -> outside/192.0.2.2(23) hit-cnt 1 first hit [0x3d0397cc, 0x44b661f2]

!access-list inside line 2 extended permit object-group services any any:
asa# sh log | i 106100.*0x3d0397cc
Mar 03 2016 01:32:04: %ASA-6-106100: access-list inside permitted tcp inside/10.0.0.10(37284) -> outside/192.0.2.2(23) hit-cnt 1 first hit [0x3d0397cc, 0x44b661f2]
Mar 03 2016 01:32:54: %ASA-6-106100: access-list inside permitted tcp inside/10.0.0.10(64900) -> outside/192.0.2.2(21) hit-cnt 1 first hit [0x3d0397cc, 0x7a45343d]

Based on these log messages, we could replace the existing generic ACEs with more specific rules.

Dynamic NAT and no matching global

The ASA translates an address when a NAT rule matches the traffic. If no NAT rule matches, processing for the packet continues. A single dynamic NAT overload rule is created with the following commands:

nat (inside) 1 192.168.0.0 255.255.255.0
global (outside) 1 interface

Therefore any packet coming from 192.168.0.0/24 and going to the Internet, is translated to the outside interface address.

However, ASA 8.2 adds the following entries to the NAT table when you multiple active interfaces and a nat statement is defined:

match ip inside 192.168.0.0 255.255.255.0  outside any
    dynamic translation to pool 1 (192.0.2.1 [Interface PAT])
match ip inside 192.168.0.0 255.255.255.0  DMZ any
    dynamic translation to pool 1 (No matching global)
match ip inside 192.168.0.0 255.255.255.0  Guest any
    dynamic translation to pool 1 (No matching global)

It creates translation conditions for all the active interfaces, but if global statements are not defined for all of them, packets will be dropped due to "no matching global". In other words, the translation rule is not complete and the ASA cannot process the packet.
Then, if we have traffic going to networks connected to the other interfaces and translations are not required, we must create additional rules to handle exceptions (NAT exempt or NAT 0 or NONAT rules).

ASA 8.3 and later releases do not create those additional translation conditions, then NAT exempt rules for other interfaces are not required. This is a NAT table on new releases for the scenario described above:

1 (inside) to (outside) source dynamic net-192.168.0.0-24 interface

arp permit-nonconnected

You have a NAT block on your firewall but it is not a directly connected subnet. It used to work, but after upgrading to 8.4 it doesn't work anymore. What happened?

NAT Exemption for intra-interface traffic

Both sites A and B have IPsec L2L tunnels to HQ ASA. Remote users send traffic to the Web through the VPN tunnels and also communicate with each other.

HQ ASA has dynamic PAT rules to translate traffic coming from remote sites using the outside interface IP address before routing the traffic to the Web. It is also configured to allow intra-interface traffic:


nat (outside) 1 10.2.2.0 255.255.255.0
nat (outside) 1 10.3.3.0 255.255.255.0
nat (inside) 1 0 0
global (outside) 1 interface
same-security-traffic permit intra-interface


For traffic coming from a higher security level interface to a lower one (outbound traffic), you don't need to create a rule to exempt returning traffic from NAT:

Source: 172.16.1.0/24 (inside)
Destination: 10.2.2.0/24 (outside)

access-list inside-nonat permit ip 172.16.1.0 255.255.255.0 10.2.2.0 255.255.255.0
nat (inside) 0 access-list inside-nonat

However, if source and destination are routed through the same interface, you need to create two ACEs, otherwise returning traffic would match the PAT rule:

Multi-context FWSM ACL partition

When you convert a FWSM from single to multiple mode (security contexts), the system creates pools of resources (aka partitions). These pools limit the number of rules (ACEs, AAA rules, Policy NAT, and others) that can be created on each context. The FWSM uses 12 partitions by default (maximum value) and each context is assigned to its own partition, unless you have more than twelve contexts. In this case, the system will assign more than one context to each partition, sharing resources between them.

What is the reason for the log %ASA-6-106015?

When the ASA receives a packet it checks the conn table and whether a connection entry is found for that packet, it is handled by the Fast Path and bypass the ACLs. It is true for any packet that doesn't require application inspection, otherwise it is handled by session management path or control plane path.


So what if you see the following logs:

Feb 20 2012 08:15:08: %ASA-6-302013: Built outbound TCP connection 7985447 for outside:192.168.0.35/80 (192.168.0.35/80) to inside:10.0.0.20/45494 (192.0.2.20/45494)
Feb 20 2012 08:15:38: %ASA-6-302014: Teardown TCP connection 7985447 for outside:192.168.0.35/80 to inside:10.0.0.20/45494 duration 0:00:30 bytes 0 SYN Timeout
Feb 20 2012 08:15:54: %ASA-6-106015: Deny TCP (no connection) from 192.168.0.35/80 to 192.0.2.20/45494 flags SYN ACK  on interface outside

Disabling idle timeout for specific traffic

The ASA enforces a timeout for idle connections and the default value is one hour for TCP connections. It helps to save resources and avoid overloads, but it can also crash some applications. We can disable this feature at all, but it is not a good idea as it can impact the firewall performance. Thus the best thing to do when you are running some application which you expect to have idle connections for long time is disabling the idle control for that traffic only. We can do that with Advanced Connection Settings.

Static route issue

If you made a typo when setting up static routes, it could result in a persistent route. For example, if you want to implement a static route for 10.10.10.0/24 but you write a wrong netmask, you are not able to remove that route:

asa(config)# route inside 10.10.10.0 255.255.25.0 192.168.100.254
asa(config)# no route inside 10.10.10.0 255.255.25.0 192.168.100.254
%No matching route to delete

Tips on output filtering

During troubleshooting sessions, you execute several show commands to collect information. Oftentimes you don't need all the output, then you use the vertical bar (aka pipe symbol) to filter. In this article I will show some useful filtering expressions.

The gateway address is wrong, but I have access to the Internet. Why?

In order to reach unknown networks, a host needs to use a gateway to forward the traffic. The gateway is some device connected to the local network and at least one external network. That device should be able to route packets, then it can forward the traffic to reach unknown destinations. Ok, we know that. What's the big news?

Is it possible to reach the Internet whether the gateway address is wrong? Yes. Maybe. I will demonstrate how, using the following scenario:

Static NAT - Does the order of commands matter?

I've implemented the following scenario to demonstrate the behavior of the ASA/PIX that we could see after to change the order of the Static NAT rules.

Debugging interface status on failover pairs

Should static routes interfere with the operation of ASA/PIX failover pairs? I don't think so. However, it is possible! So I'm going to describe a scenario where that issue might happen.

Syslog over IPSec

I've identified an issue when ASAs are configured to send syslog messages over an IPSec tunnel. For some unknown reason, I've seen some devices trying to establish the connection to the log server without forward the traffic through the tunnel. Thus, the log messages are not saved, since the remote peer blocks the unencrypted packets.