Debugging IPSEC

Today I had occasion to do some IPSEC debugging.  There were multiple tunnels between two endpoints (using OpenS/Wan's "leftsubnets={...}" and "rightsubnets={...}" declarations) and one of the tunnels was reported as being dead.  The remaining tunnels were carrying a lot of traffic and couldn't be shut down.

Usually I'd ping some stuff on the other side of the VPN and tcpdump the connection to see if the pings are being encapsulated.  But with multiple tunnels between the same endpoints, just filtering the traffic by source and destination address doesn't really help - you'd end up capturing traffic for all of the tunnels.

Each tunnel has its own pair of identifiers (SPIs) - one for each direction.  So first thing to do is run "ip xfrm policy show" and you get something like:

# ip xfrm policy show
src 10.0.0.0/24 dst 203.0.113.1
    dir out priority 2343 ptype main
    tmpl src 198.51.100.1 dst 203.0.113.1
        proto esp reqid 16389 mode tunnel
src 192.168.0.0/24 dst 10.0.0.0/24
    dir fwd priority 2343 ptype main
    tmpl src 203.0.113.1 dst 198.51.100.1
        proto esp reqid 16389 mode tunnel
src 192.168.0.0/24 dst 10.0.0.0/24
    dir in priority 2343 ptype main
    tmpl src 203.0.113.1 dst 198.51.100.1
        proto esp reqid 16389 mode tunnel

(The output will actually be much longer if you've got multiple tunnels).  From this we can extract the reqid from the tunnels we care about - 16389 in this case.

Now we do "ip xfrm state show":

# ip xfrm policy show
src 203.0.113.1 dst 198.51.100.1
    proto esp spi 0x01234567 reqid 16389 mode tunnel
    replay-window 32 flag 20
    auth hmac(sha1) 0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    enc cbc(des3_ede) 0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
src 198.51.100.1 dst 203.0.113.1
    proto esp spi 0x89abcdef reqid 16389 mode tunnel
    replay-window 32 flag 20
    auth hmac(sha1) 0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    enc cbc(des3_ede) 0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

(Again, the output will be much longer than this).  Now look up the reqid and make a note of the associated spis (0x01234567 and 0x89abcdef).


Finally, we can ask tcpdump to only show us traffic that matches those SPIs:

# tcpdump -n -i internet 'ip[20:4] = 0x01234567' or 'ip[20:4] = 0x89abcdef'

In this case I found that the encapsulated traffic was being transmitted ok, but we weren't receiving any from the other end.  Turns out the other side's firewall was dropping their traffic.