How does traceroute work?

This one likes to make an appearance now and then so let’s dive into the lab and get the answer. First of all…my own definition of traceroute: “To identify all Layer 3 network devices in a given path to a destination

Topology

4 routers in the lab running OSPF in area 0. Everyone is advertising their networks into Area 0, so full end to end reachability is possible.

Scenario

Let’s perform a traceroute from vIOS1 to vIOS4:

  • Source: 10.1.0.1/24 (vIOS1)
  • Destination: 10.3.0.4/24 (vIOS4)
  • ***This traceroute on Cisco IOS will use UDP for it’s transport

Exercise

Ok so here we go… lets perform the standard traceroute and get the basic output:

vIOS1#traceroute 10.3.0.4
Type escape sequence to abort.
Tracing the route to 10.3.0.4
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.0.2 6 msec 5 msec 4 msec
  2 10.2.0.3 7 msec 5 msec 5 msec
  3 10.3.0.4 7 msec *  18 msec

We have a path shown from this source to the destination. Observations.. why do we have 3 responses on each hop? This is down to the probe value and can be set using the extended traceroute command as per below:

In this example we change the probe count to 1 and sure enough we only see 1 response now on each hop

So with this in place.. lets now dive deeper into how exactly traceroute is working. Let’s stick with the probe count of 1 as it will our lives easier in the packet captures. Here we go..

Packet Captures

So if you do a quick Google on how traceroute works, you will more than likely start reading about TTL values and how they change along the way. Before we go any further, lets check this in the capture and then go from there with the how and why.

So I have setup captures on each routers interface and will set off a traceroute and then talk through the results. Let’s go..

vIOS1 Gig0/0 to vIOS2 Gig0/0

vIOS1#traceroute
Protocol [ip]: ip
Target IP address: 10.3.0.4
Ingress traceroute [n]:
Source address: 10.1.0.1
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]: 1
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.
Tracing the route to 10.3.0.4
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.0.2 7 msec
  2 10.2.0.3 9 msec
  3 10.3.0.4 9 msec

What did we see in Wireshark?

vIOS1

Here is the entire capture from the point of starting the traceroute:

PCAP of traceroute

What is exactly going on?

Packet 112 and 113

A UDP packet (112) is encapsulated and the IP header confirms the source and destination IP addresses as expected:

In the same IP header we also see the TTL (Time to Live) field set as 1:

This particular UDP stream contains 1 ICMP packet (113). Let’s take a look at this in more detail:

The IP header now contains different Source and Destination addresses

The Source of this ICMP packet is from 10.1.0.2, which is infact vIOS2 and the Destination of this packet is 10.1.0.1 which is vIOS1. Remember that 10.1.0.2 is the 1st hop that we see in the traceroute.

What else is going on here? Well we can also see the original Src and Dst IP header (10.1.0.1 to 10.3.0.4) encapsulated in the ICMP packet along with the UDP connection details:

What about this TTL stuff? Well let’s take a look at the ICMP packet and what it is telling us about TTL:

TTL of 0 = expired in transit = drop packet!
  • vIOS2 has decremented the TTL of 1 by 1, which leaves 0.
  • A TTL of 0 means the packet is dropped.
  • This is communicated from vIOS2 back to vIOS1 within this ICMP packet.

So what next? How does that help us? Well infact vIOS1 has now discovered the 1st hop in the path to our desired destination. That hop is 10.1.0.2 which we do see within the traceroute. So again… what next?!

vIOS2 now wants to discover the router detail for the 2nd hop in the path. So guess what? The TTL is increased to a count of 2. So we now move to the next UDP stream for this information.

Packet 114 and 115

Lets take a look at that UDP packet..

The Src and Dst IP addresses are still the same as per our traceroute. (vIOS1 and vIOS4)

But wait we can now see the TTL has increased to 2. Knowing what we know so far, we are expecting an ICMP response from vIOS2 right? Maybe! Maybe not! Let’s take a look at packet 115 for this detail:

The ICMP response this time is infact from Source IP 10.2.0.3 which is actually the L3 interface on vIOS3:

Let’s not forget that we are seeing this from the perspective of vIOS1 at this stage.

The Destination address here is also important. This is targeted at vIOS1 on 10.1.0.1, which is infact always going to be the Destination address for these ICMP responses along the way:

The Destination IP is always the same with the ICMP packet targeted each time at the source of the traceroute on vIOS1

But wait, the Source of this ICMP packet is also the 2nd hop address in our traceroute:

So we are starting to learn more about the path and have now discovered hop 1 and hop 2. But wait what about the concept with the TTL? What happened here with this 2nd ICMP packet?

The response was sure enough ‘Time to live exceeded in transit’ which must mean the reported router had decremented the TTL to the point of 0 and is reporting back to drop the packet. Let’s expand on this…

The reporting router is 10.2.0.3, so we need to shift out attention to this router what it has done in order to help discover the path.

Pcaps on vIOS2

Here are captures on vIOS2. Let’s start with packet 104:

Packet 104 shows the UDP connection arrive from vIOS1 / 10.1.0.1 destined for 10.3.0.4. Note the TTL is 1!

Packet 105 shows the ICMP packet sent back to vIOS1 / 10.1.0.1:

  • TTL is 0, therefore expired in transit.
  • The original IP header is encapsulated in the ICMP packet as before.
  • The ICMP response is a Type 11

So really this is just confirmation around the 1st UDP / ICMP connection. Let’s shift our attention back to that 2nd UDP stream and ICMP response..

Packet 106 and 107

Ok so here things are really heating up on this whole TTL concept. Let’s break this down clearly:

  • 106 – vIOS2 has received the UDP packet from vIOS1 with our original Src and Dest IP addresses as per the trace
  • 106 – The TTL field is now set to 2. Why is this set to 2? Because vIOS1 set it as 2! Remember this packet is from vIOS1 who is attempting to discover the network path to 10.3.0.4. vIOS2 will therefore pass this onto the next hop router and will NOT drop the packet. So we should be expecting a response from vIOS3 right?
  • 107 – ICMP packet from 10.2.0.3 / vIOS3 – Correct! The ICMP response shows the TTL of 0 and therefore expired in transit with the response going back to 10.1.0.1 being vIOS1.

So lets come to earth and summarise this before we go any deeper:

  • Each time vIOS1 / Source of the traceroute learns about a next hop in the path it will increment the TTL by 1.
  • Packet 108 the 3rd UDP stream confirms this in the IP header with a TTL of 3:
  • For each UDP packet that we send, we will always expect the ICMP response to come back from the next hop in the path
  • The destination IP for each ICMP response will always be the Source IP of the traceroute (We are the ones who need to know the path remember!)

Final Response / Last Hop

Back to vIOS1 now and let’s look at ICMP packet 117 from 10.3.0.4:

Sure enough as it was the 3rd hop we are attempting to discover, we sent a TTL of 3.

The ICMP response this time is a little different. This time the response is from 10.3.0.4 which is the final hop in the traceroute and infact our destination in the traceroute. What is going on here then..

Well the ICMP response this time states ‘Destination Unreachable’ with a Type 3 ICMP response:

‘The ICMP Destination Unreachable message is sent by a router in response to a packet which it cannot forward because the destination (or next hop) is unreachable or a service is unavailable’ My definition of this is.. ‘There is nowhere else to go!’ Therefore a Type 3 ICMP packet is sent back to vIOS1.

TTL by design

Outside of traceroute.. TTL exists within the IP Header for what purpose?

The point of the TTL/hop limit is to keep streams of undeliverable packets stuck in routing loops (perhaps due to incorrect routing tables) from circulating forever and clogging up the networks in question

How does it normally operate outside of a traceroute?

‘An IP TTL is set initially by the system sending the packet. It can be set to any value between 1 and 255; different operating systems set different defaults. Each router that receives the packet subtracts at least 1 from the count; if the count remains greater than 0, the router forwards the packet, otherwise it discards it and sends an Internet Control Message Protocol (ICMP) message back to the originating host, which may trigger a resend.’

Infact we can actually see this happening in this lab example with the IP Header in the ICMP responses:

Packet 113 with a received TTL of 255 from 10.1.0.2:

Packet 115 with a received TTL of 254 from 10.2.0.3:

Packet 117 with a receive TTL of 253 from 10.3.0.4:

Summary

Traceroute is essentially taking advantage of the existing TTL functionality in the IP Header (In combination with ICMP) to discover each next hop router in the path to the destination.

Starting with a TTL of 1, when we discover each next hop, we increment the TTL by 1 to then pass the 1st known router to the 2nd, then onto the 3rd, 4th, 5th etc… each time with the TTL being set and decremented accordingly to allow the router to drop the packet and respond back to the originator via ICMP.

BGP Lab 3 – Route Filtering

Ok lets stick with the same again, but this time let’s filter out some routes. Lets filter out 6.6.6.6/32 that is directly connected to the Internet router. Let’s make this happen on R1 as we are the customer and are in control of this router.

!To confirm we have a route for this prefix via ISP1!

R1#sh ip bgp
BGP table version is 23, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       0.0.0.0                  0         32768 i
 *>i 2.2.2.2/32       10.1.0.2                 0    100      0 i
 *>i 3.3.3.3/32       10.2.0.3                 0    100      0 i
 *>  4.4.4.4/32       10.4.0.2                               0 65003 65004 65002 i
 *>  5.5.5.5/32       10.4.0.2                 0             0 65003 i
 *   6.6.6.6/32       10.4.0.2                               0 65003 65004 i
 *>                   10.3.0.2                             100 65002 65004 i
 *>  10.1.0.0/24      0.0.0.0                  0         32768 i
 * i                  10.1.0.2                 0    100      0 i
 *>i 10.2.0.0/24      10.1.0.2                 0    100      0 i
 *>  10.3.0.0/24      0.0.0.0                  0         32768 i
 *>  10.4.0.0/24      0.0.0.0                  0         32768 i
 *>  172.16.1.0/24    10.4.0.2                               0 65003 65004 i
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.16.2.0/24    10.4.0.2                               0 65003 65004 i
 *>  172.16.3.0/24    10.4.0.2                               0 65003 65004 i

Let’s filter this route out on R1. We can use: (There are more ways..)

  1. Distribute List
  2. Prefix List

Distribute List

! Filter out 6.6.6.6/32 using a Distribute List !

ip access-list standard DENY_6666
 deny   6.6.6.6
 permit any

! Apply to each ISP inbound !

R1(config-router)#neighbor 10.3.0.2 distribute-list DENY_6666 in
R1(config-router)#neighbor 10.4.0.2 distribute-list DENY_6666 in
R1#clear ip bgp * soft

R1#sh ip bgp
BGP table version is 25, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       0.0.0.0                  0         32768 i
 *>i 2.2.2.2/32       10.1.0.2                 0    100      0 i
 *>i 3.3.3.3/32       10.2.0.3                 0    100      0 i
 *>  4.4.4.4/32       10.4.0.2                               0 65003 65004 65002 i
 *>  5.5.5.5/32       10.4.0.2                 0             0 65003 i
 *>  10.1.0.0/24      0.0.0.0                  0         32768 i
 * i                  10.1.0.2                 0    100      0 i
 *>i 10.2.0.0/24      10.1.0.2                 0    100      0 i
 *>  10.3.0.0/24      0.0.0.0                  0         32768 i
 *>  10.4.0.0/24      0.0.0.0                  0         32768 i
 *>  172.16.1.0/24    10.4.0.2                               0 65003 65004 i
 *>  172.16.2.0/24    10.4.0.2                               0 65003 65004 i
 *>  172.16.3.0/24    10.4.0.2                               0 65003 65004 i

Sure enough the routes are not in the BGP table but other routes are. Let’s verify:

R1#show access-lists
Standard IP access list DENY_6666
    10 deny   6.6.6.6 (2 matches)
    20 permit any (10 matches)

Prefix List

! Filter out 6.6.6.6/32 using a Prefix List !

R1#sh run | sec prefix-list
ip prefix-list DENY_6666 seq 5 deny 6.6.6.6/32
ip prefix-list DENY_6666 seq 10 permit 0.0.0.0/0 ge 32

! Apply to ISP1 and ISP2 neighbors !


R1(config-router)#neighbor 10.3.0.2 prefix-list DENY_6666 in
R1(config-router)#neighbor 10.4.0.2 prefix-list DENY_6666 in
clear ip bgp * soft

! Check BGP table !

R1#sh ip bgp
BGP table version is 32, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       0.0.0.0                  0         32768 i
 *>i 2.2.2.2/32       10.1.0.2                 0    100      0 i
 *>i 3.3.3.3/32       10.2.0.3                 0    100      0 i
 *>  4.4.4.4/32       10.4.0.2                               0 65003 65004 65002 i
 *>  5.5.5.5/32       10.4.0.2                 0             0 65003 i
 *>  10.1.0.0/24      0.0.0.0                  0         32768 i
 * i                  10.1.0.2                 0    100      0 i
 *>i 10.2.0.0/24      10.1.0.2                 0    100      0 i
 *>  10.3.0.0/24      0.0.0.0                  0         32768 i
 *>  10.4.0.0/24      0.0.0.0                  0         32768 i

So this has worked all OK and we can no longer see the 6.6.6.6/32 prefix. To confirm also we can see the expected prefixes from ISP1 and ISP2. Success!

Posted in BGP

BGP Lab 2 – Route Summarization

Ok so same network as before but let’s do some basic route summarization with BGP.

Addition of 3 x 172.x.x.x network on the Internet router to advertise into BGP

Lets create and add the networks on the Internet router:

interface Loopback1
 ip address 172.16.1.1 255.255.255.0
!
interface Loopback2
 ip address 172.16.2.1 255.255.255.0
!
interface Loopback3
 ip address 172.16.3.1 255.255.255.0
Internet#sh ip route connected | inc 172.16
      172.16.0.0/16 is variably subnetted, 6 subnets, 2 masks
C        172.16.1.0/24 is directly connected, Loopback1
L        172.16.1.1/32 is directly connected, Loopback1
C        172.16.2.0/24 is directly connected, Loopback2
L        172.16.2.1/32 is directly connected, Loopback2
C        172.16.3.0/24 is directly connected, Loopback3
L        172.16.3.1/32 is directly connected, Loopback3

Let’s advertise them into BGP:

Internet(config)#router bgp 65004
Internet(config-router)#network 172.16.1.0 mask 255.255.255.0
Internet(config-router)#network 172.16.2.0 mask 255.255.255.0
Internet(config-router)#network 172.16.3.0 mask 255.255.255.0

Lets check how R1 sees the advertisement:

R1#sh ip bgp
BGP table version is 21, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       0.0.0.0                  0         32768 i
 *>i 2.2.2.2/32       10.1.0.2                 0    100      0 i
 *>i 3.3.3.3/32       10.2.0.3                 0    100      0 i
 *>  4.4.4.4/32       10.4.0.2                               0 65003 65004 65002 i
 *>  5.5.5.5/32       10.4.0.2                 0             0 65003 i
 *   6.6.6.6/32       10.4.0.2                               0 65003 65004 i
 *>                   10.3.0.2                             100 65002 65004 i
 *>  10.1.0.0/24      0.0.0.0                  0         32768 i
 * i                  10.1.0.2                 0    100      0 i
 *>i 10.2.0.0/24      10.1.0.2                 0    100      0 i
 *>  10.3.0.0/24      0.0.0.0                  0         32768 i
 *>  10.4.0.0/24      0.0.0.0                  0         32768 i
 *>  172.16.1.0/24    10.4.0.2                               0 65003 65004 i
 *>  172.16.2.0/24    10.4.0.2                               0 65003 65004 i
 *>  172.16.3.0/24    10.4.0.2                               0 65003 65004 i

To clarify reachability from R1:

R1#ping 172.16.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/6/10 ms

R1#ping 172.16.2.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/7 ms

R1#ping 172.16.3.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/8 ms

Now how about the summarization and where will it be applied exactly?

Lets try and do this firstly on the Internet router with the aggregate-address command globally in BGP:

Internet#sh run | sec bgp
router bgp 65004
 bgp log-neighbor-changes
 network 6.6.6.6 mask 255.255.255.255
 network 172.16.1.0 mask 255.255.255.0
 network 172.16.2.0 mask 255.255.255.0
 network 172.16.3.0 mask 255.255.255.0
 aggregate-address 172.16.0.0 255.255.0.0 summary-only
 neighbor 10.5.0.1 remote-as 65002
 neighbor 10.6.0.1 remote-as 65003

How does this look elsewhere in the network?

R1#sh ip bgp
BGP table version is 19, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       0.0.0.0                  0         32768 i
 *>i 2.2.2.2/32       10.1.0.2                 0    100      0 i
 *>i 3.3.3.3/32       10.2.0.3                 0    100      0 i
 *>  4.4.4.4/32       10.4.0.2                               0 65003 65004 65002 i
 *>  5.5.5.5/32       10.4.0.2                 0             0 65003 i
 *   6.6.6.6/32       10.4.0.2                               0 65003 65004 i
 *>                   10.3.0.2                             100 65002 65004 i
 *>  10.1.0.0/24      0.0.0.0                  0         32768 i
 * i                  10.1.0.2                 0    100      0 i
 *>i 10.2.0.0/24      10.1.0.2                 0    100      0 i
 *>  10.3.0.0/24      0.0.0.0                  0         32768 i
 *>  10.4.0.0/24      0.0.0.0                  0         32768 i
 *>  172.16.0.0       10.4.0.2                               0 65003 65004 i

Success! We see whatever we specified in the aggregate-address command.

Posted in BGP

BGP Lab 1 (iBGP + eBGP)

Ok so time to do some BGP! In this lab we will setup iBGP and eBGP peerings with different autonomous systems. We will setup and advertise loopback interfaces on each router for testing BGP functionality and reachability of these prefixes.

Here is the topology:

iBGP + eBGP Topology

iBGP – AS 65001 (R1, R2 + R3)

R1#sh run | sec bgp
router bgp 65001
 bgp router-id 1.1.1.1
 bgp log-neighbor-changes
 network 1.1.1.1 mask 255.255.255.255
 network 10.1.0.0 mask 255.255.255.0
 network 10.3.0.0 mask 255.255.255.0
 network 10.4.0.0 mask 255.255.255.0
 neighbor 10.1.0.2 remote-as 65001
 neighbor 10.1.0.2 next-hop-self
 neighbor 10.3.0.2 remote-as 65002
 neighbor 10.4.0.2 remote-as 65003
  • iBGP peering with R2 in AS 65001
  • Next Hop Self, to ensure any advertisements into 65001 do no show the next hop as ISP1 or ISP2 which is the default behaviour with an eBGP neighbor
  • R1 has eBGP peerings to ISP1 and ISP2 (65002 and 65003)
  • Underlay network and Loopback IP address advertised into BGP
R2#sh run | sec bgp
router bgp 65001
 bgp router-id 2.2.2.2
 bgp log-neighbor-changes
 network 2.2.2.2 mask 255.255.255.255
 network 10.1.0.0 mask 255.255.255.0
 network 10.2.0.0 mask 255.255.255.0
 neighbor 10.1.0.1 remote-as 65001
 neighbor 10.1.0.1 route-reflector-client
 neighbor 10.2.0.3 remote-as 65001
 neighbor 10.2.0.3 route-reflector-client
  • R2 is also a Route Reflector to enable prefixes to be advertised to R3 in the absence of a full mesh. Another iBGP default rule.
  • iBGP peerings with R1 and R3
  • Underlay network and Loopback IP address advertised into BGP
R3#sh run | sec bgp
router bgp 65001
 bgp router-id 3.3.3.3
 bgp log-neighbor-changes
 network 3.3.3.3 mask 255.255.255.255
 network 10.2.0.0 mask 255.255.255.0
 neighbor 10.2.0.2 remote-as 65001
  • iBGP peering with R2
  • Underlay network and Loopback IP address advertised into BGP

eBGP (R1 / ISP1, ISP2 and Internet)

Again let’s revisit R1:

R1#sh run | sec bgp
router bgp 65001
 bgp router-id 1.1.1.1
 bgp log-neighbor-changes
 network 1.1.1.1 mask 255.255.255.255
 network 10.1.0.0 mask 255.255.255.0
 network 10.3.0.0 mask 255.255.255.0
 network 10.4.0.0 mask 255.255.255.0
 neighbor 10.1.0.2 remote-as 65001
 neighbor 10.1.0.2 next-hop-self
 neighbor 10.3.0.2 remote-as 65002
 neighbor 10.4.0.2 remote-as 65003
  • We have eBGP peerings to ISP1 and ISP2

ISP1 and ISP2

ISP1#sh run | sec bgp
router bgp 65002
 bgp log-neighbor-changes
 network 4.4.4.4 mask 255.255.255.255
 neighbor 10.3.0.1 remote-as 65001
 neighbor 10.5.0.2 remote-as 65004
ISP2#sh run | sec bgp
router bgp 65003
 bgp log-neighbor-changes
 network 5.5.5.5 mask 255.255.255.255
 neighbor 10.4.0.1 remote-as 65001
 neighbor 10.6.0.2 remote-as 65004
  • ISP1 and ISP2 both have eBGP peerings from their local AS to 65001 (Let’s call this the customer)
  • We are only advertising the loopback addresses into our BGP peers

Internet

Internet#sh run | sec bgp
router bgp 65004
 bgp log-neighbor-changes
 network 6.6.6.6 mask 255.255.255.255
 neighbor 10.5.0.1 remote-as 65002
 neighbor 10.6.0.1 remote-as 65003
  • This router is really acting as our destination for testing, so 6.6.6.6/32 is the Loopback advertised into BGP

Exercise 1

Reachability of 6.6.6.6/32 from R3

!R3

R3>sh ip bgp
BGP table version is 39, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 1.1.1.1/32       10.1.0.1                 0    100      0 i
 *>i 2.2.2.2/32       10.2.0.2                 0    100      0 i
 *>  3.3.3.3/32       0.0.0.0                  0         32768 i
 *>i 4.4.4.4/32       10.1.0.1                 0    100      0 65002 i
 *>i 5.5.5.5/32       10.1.0.1                 0    100      0 65003 i
 *>i 6.6.6.6/32       10.1.0.1                 0    100      0 65003 65004 i
 *>i 10.1.0.0/24      10.2.0.2                 0    100      0 i
 * i 10.2.0.0/24      10.2.0.2                 0    100      0 i
 *>                   0.0.0.0                  0         32768 i
 *>i 10.3.0.0/24      10.1.0.1                 0    100      0 i
 *>i 10.4.0.0/24      10.1.0.1                 0    100      0 i

So we have a prefix for 6.6.6.6/32 with a next hop of 10.1.0.1 which is the interface on R1 in AS 65001. We also have the prefix for 10.1.0.0/24 to the next hop address on R2, therefore this is a valid route and best.

Can we get there?

R3>ping 6.6.6.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 7/9/18 ms

Success! 🙂

Exercise 2

From R1, influence the outbound ISP that we use to reach 6.6.6.6/32

!Current route detail for 6.6.6.6/32 on R1

R1#sh ip bgp 6.6.6.6/32
BGP routing table entry for 6.6.6.6/32, version 15
Paths: (2 available, best #1, table default)
  Advertised to update-groups:
     30         31
  Refresh Epoch 7
  65003 65004
    10.4.0.2 from 10.4.0.2 (5.5.5.5)
      Origin IGP, localpref 100, valid, external, best
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 7
  65002 65004
    10.3.0.2 from 10.3.0.2 (4.4.4.4)
      Origin IGP, localpref 100, valid, external
      rx pathid: 0, tx pathid: 0

So our valid and best path is currently (in a default state) via 10.4.0.2 which is ISP2.

Let’s use the ‘Weight’ path attribute to influence which outbound path we take. We need to use a route map and access list to help us here.

Current the weight is 0:


     Network          Next Hop            Metric LocPrf Weight Path

 *>  6.6.6.6/32       10.4.0.2                               0 65003 65004 i
 *                    10.3.0.2                               0 65002 65004 i

Route-Map Configuration:

R1#sh run | sec access-list
ip access-list standard MATCH_6666
 permit 6.6.6.6


R1#sh run | sec route-map
route-map PREFER_ISP1 permit 10
 match ip address MATCH_6666
 set weight 100

So we are looking to match 6.6.6.6 and if we do, set the weight to 100. Let’s now apply this to the neighbor statement to the path we wish to take via ISP1:

R1#sh run | sec bgp
router bgp 65001
 bgp router-id 1.1.1.1
 bgp log-neighbor-changes
 network 1.1.1.1 mask 255.255.255.255
 network 10.1.0.0 mask 255.255.255.0
 network 10.3.0.0 mask 255.255.255.0
 network 10.4.0.0 mask 255.255.255.0
 neighbor 10.1.0.2 remote-as 65001
 neighbor 10.1.0.2 next-hop-self
 neighbor 10.3.0.2 remote-as 65002
 neighbor 10.3.0.2 route-map PREFER_ISP1 in
 neighbor 10.4.0.2 remote-as 65003

Lets clear BGP soft:

clear ip bgp * soft

…and now check the BGP table for the 6.6.6.6/32 prefix:

R1#sh ip bgp
BGP table version is 18, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  1.1.1.1/32       0.0.0.0                  0         32768 i
 *>i 2.2.2.2/32       10.1.0.2                 0    100      0 i
 *>i 3.3.3.3/32       10.2.0.3                 0    100      0 i
 *>  4.4.4.4/32       10.4.0.2                               0 65003 65004 65002 i
 *>  5.5.5.5/32       10.4.0.2                 0             0 65003 i
 *   6.6.6.6/32       10.4.0.2                               0 65003 65004 i
 *>                   10.3.0.2                             100 65002 65004 i
 *>  10.1.0.0/24      0.0.0.0                  0         32768 i
 * i                  10.1.0.2                 0    100      0 i
 *>i 10.2.0.0/24      10.1.0.2                 0    100      0 i
 *>  10.3.0.0/24      0.0.0.0                  0         32768 i
 *>  10.4.0.0/24      0.0.0.0                  0         32768 i

We can see the valid and best route is now via 10.3.0.2 which is ISP1 so looking good. Also the Weight attribute is now set to 100 as per the instruction from the route map. Success! 🙂

To confirm we can still reach 6.6.6.6/32 via ISP1:

R3>ping 6.6.6.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/10/14 ms

R1#ping 6.6.6.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/7/14 ms

If we wanted to, we could do the same thing using Local Pref with a higher value that whatever is in the BGP table. Remember though that Weight is earlier in the path selection process than Local Pref.

Conclusion

Weight and Local Pref are typically used to influence outbound routes to multiple ISPs.

What about inbound? Well this is is where we would typically use AS_PATH (Lower AS PATH preferred) or MED (Lower MED preferred) Again, in the same way using a route-map applied to a neighbor statement.

Posted in BGP

Cisco Identity Services Engine Lab 6 – Authorization Policy / DACL

What about if we want to have a switchport locked down with zero or minimal connectivity and then permit this port to have full connectivity after successful dot1x authentication? Well we can sure do this with an authorization profile leveraging a Downloadable Access Control List.

This is where we see great value with ISE and it’s capabilities around network security.

Scenario

Lets block ICMP to 8.8.8.8/32, but permit this when the supplicant has been successfully authenticated by ISE. Sound good? Lets do it..

We are now bringing a physical endpoint and switch onto the network as follows: (Will explain later)

Topology

Steps

  1. Create an ACL to deny traffic to 8.8.8.8/32
  2. Apply the ACL to the switchport inbound
  3. Verify traffic is blocked to 8.8.8.8/32
  4. Create an Authorization Profile with a DACL that permits all traffic
  5. Once dot1x authentication passed, the DACL via authorization will be applied to the switchport
  6. Verify this and that traffic is permitted to 8.8.8.8/32

Block ACL

Setup a blocking ACL to deny traffic from the host / any source to 8.8.8.8/32

ip access-list extended Block_8888
 deny   icmp any host 8.8.8.8
 permit ip any any

This is now blocking and no ICMP is possible from the host to 8.8.8.8.

How do we approach this with ISE? Let’s take a look..

Downloadable ACL

Lets use the default ACL for permit all ipv4 traffic:

Authorization Profile

Lets create a new authorization profile that references the ACL:

Policy Set

What do we now do with the auth profile? Well we need to apply it to the current policy set as follows:

Notice that we have disabled the rule below, which was previously being matched to authorize clients. Now we are expecting to match the ‘Permit 8888’ authorization rule and download an ACL to permit full IP any any.

As per the topology I am now doing the dot1x from my iMac:

Connected all OK. 🙂 Can we ping 8.8.8.8…..

Hell yeah!

How do we know the DACL has kicked in?

Authentication has passed all OK as per dot1x success. The authorization profile has also kicked in as we can see the DACL applied!

Lessons Learnt

So I lost A LOT of time doing this lab using Cisco VIRL images for the authenticating switch. Basically the DACL never applied to the port and in the end after a lot of time I gave up and instead spent around £20 on a used v2 3750 PoE switch and guess what.. the DACL applied 1st time to the 3750 and would never work with the VIRL image. Keep an eye out for that one! Sometimes you can’t beat real gear.. (For you Olly)

Posted in ISE

Deploy and use the CSR1000v in AWS

Introduction

So let’s add a CSR1000v into AWS and actually use it to route out to the internet.

In the lab below we will enable access to the Internet from a Private Subnet via the CSR1000v which is attached to the Internet Gateway as per my default VPC.

Normally the Private Subnet cannot reach the Internet unless:

  • It is in a Public Subnet attached to an Internet Gateway
  • Or is in a Private Subnet attached to a NAT Gateway

Topology

Lab Topology

Steps

VPC

  • Create VPC
  • Create 2 x subnets: (Same AZ)
  • 10.0.10.0/24 – Dunder-DMZ
  • 10.0.20.0/24 – Dunder-LAN

Internet Gateway

  • Setup Internet Gateway – Dunder-IGW
  • Attach to the VPC

Route Tables

  • Default – Edit and add default route to the IGW / associate subnet with 10.0.10.0/24
  • Create LAN Route Table / associate with subnet 10.0.20.0/24

Instances

  • DUN-MGMT-LINUX – (Public)
  • DUN-LAN-LINUX – (Private)
  • DUN-INTERNET-CSR1000V – (Public) **AWS Marketplace

Network Interfaces

  • Add Network Interface for CSR1000V LAN side (10.0.20.100/24)
  • Attach to the CSR1000v instance

Connect to EC2 Instances via Public IP / Key Pair

  • Connect to CSR1000v via Public IP (Add private key for EC2 instance access, username = ‘ec2-user’ for CSR and ‘ubuntu’ for ubuntu hosts)
  • (If needed and not done when spinning up the instance) Manually add the IP to the router as per what AWS have assigned the instance

Route Tables – Edit

  • LAN – Edit and add default route via the CSR1000V Network Interface – 10.0.20.100
  • Verify routing from DUN-LAN-LINUX to CSR1000v = OK due to route table
  • Verify routing from CSR1000v to Internet = OK due to default NAT and Internet Gateway
  • Verify routing from LAN Linux box to Internet = FAIL… why? Read on…

NAT

  • DUN-LAN-LINUX now needs to get Internet Access, but is in a Private subnet – How to tackle this?
  • **CSR1000v has a NAT configuration by default, Public subnet can reach the internet all OK
  • We need NAT for the Private Subnet
  • Add to the CSR1000v:

! Verify / update NAT configuration

!ACL for NAT source subnet
ip access-list extended LAN
permit 10.0.20.100 0.0.0.255 any

!NAT Configuration
int gig1
ip nat outside

int gig 2
ip nat inside

ip nat inside source list LAN interface Gig0/1 overload

Verify

  • Jump from DUN-MGMT-LINUX in Public Subnet to DUN-LAN-LINUX in Private Subnet = PASS
  • From DUN-LAN-LINUX verify Internet access / connectivity = Working!
  • NAT translations are present on the CSR1000v
  • Lab is complete!

Extra!

I was stuck with one thing.. I couldn’t ping out to the internet from the host on the LAN, but could all OK from the inside interface of the CSR (In the same subnet as the LAN host..) So was a bit of a headscratcher.

After some digging.. on the CSR LAN facing network interface:

Source / Dest Check

Disable / Untick them boom it came to life:

Untick / Disable

Cisco Identity Services Engine Lab 5 – AD Authentication

1 – Verify basic DNS

Verify ISE can resolve DNS names via CLI (It can)

2 – Join ISE to AD

Join to the domain:

ISE is now an AD member

Join computer account to AD

Host3 will be joined to the AD domain:

Host 3 will be our test 802.1x AD user

Also we need a AD user account for this test endpoint:

Almost ready! 🙂

Create and use Identity Source Sequence

With the below configuration we will use AD 1st and then if the user is not found we will try the next search list of Internal / Local Users:

Update Policy Set Authentication Policy

We need to tell the policy to use the new identify source sequence for 802.1x:

Verify

Great so we enable 802.1x in Windows as per previous labs and either trust or don’t trust the certificate on the ISE.

A default worth mentioning for MSCHAP:

So our test user is ready to go!

Disable / Reenable the NIC in Windows and we are pinging all ok!

Also ISE looks happy and we can see the authentication from the AD1 store:

What about the switchport?

Switch#sh dot1x interface GigabitEthernet1/2 details

Dot1x Info for GigabitEthernet1/2
-----------------------------------
PAE                       = AUTHENTICATOR
QuietPeriod               = 60
ServerTimeout             = 0
SuppTimeout               = 30
ReAuthMax                 = 2
MaxReq                    = 2
TxPeriod                  = 10

Dot1x Authenticator Client List
-------------------------------
EAP Method                = (25)
Supplicant                = 5000.0006.0000
Session ID                = C0A844010000000F0025EDC7
    Auth SM State         = AUTHENTICATED
    Auth BEND SM State    = IDLE
Switch#show authentication sessions interface gigabitEthernet 1/2

Interface    MAC Address    Method  Domain  Status Fg Session ID
----------------------------------------------------------------------
Gi1/2        5000.0006.0000 dot1x   DATA    Auth      C0A844010000000F0025EDC7


Key to Session Events Blocked Status Flags:

  A - Applying Policy (multi-line status for details)
  D - Awaiting Deletion
  F - Final Removal in progress
  I - Awaiting IIF ID allocation
  N - Waiting for AAA to come up
  P - Pushed Session
  R - Removing User Profile (multi-line status for details)
  U - Applying User Profile (multi-line status for details)
  X - Unknown Blocker

Runnable methods list:
  Handle  Priority  Name
    5        5      dot1x
    9        10     mab
    7        15     webauth

Looking good!

So we now have Jim Halpert authenticating his network interface using dot1x (PEAP Outer and MS-CHAPV2 Inner) via his AD account. 🙂 Great stuff! 🙂

Posted in ISE

Cisco Identity Services Engine Lab 4 – MAB

1 – Add additional RADIUS attribute (Let ISE server know the MAC address will be sent over)

Switch(config)#radius-server attribute 6 on-for-login-auth

2 – Setup switchport Gig1/2 for test MAB host:

CURRENT

Switch#sh run int gigabitEthernet 1/2
Building configuration...

Current configuration : 365 bytes
!
interface GigabitEthernet1/2
 switchport access vlan 68
 switchport mode access
 media-type rj45
 negotiation auto
 authentication host-mode multi-auth
 authentication open
 authentication port-control auto
 authentication periodic
 authentication timer reauthenticate server
 dot1x pae authenticator
 dot1x timeout tx-period 10
 spanning-tree portfast edge
end

REVISED FOR MAB

Switch#sh run int gigabitEthernet 1/3
Building configuration...

Current configuration : 253 bytes
!
interface GigabitEthernet1/3
 description MAB
 switchport access vlan 68
 switchport mode access
 media-type rj45
 negotiation auto
 authentication order mab dot1x
 authentication priority mab dot1x
 mab
 dot1x pae authenticator
 dot1x timeout tx-period 10
 spanning-tree portfast edge
end

Breakdown of commands:

! Enable MAB on port
mab

! Authentication order - MAB 1st, then dot1x
authentication order mab dot1x

! Authentication priority - dot1x preferred, but MAB if supplicant can't support dot1x

authentication priority dot1x mab

3 – Add MAC address to ISE

Add MAC address of MAB host to ISE Endpoints

Now the MAC address is in the ISE endpoints database and we should be able to pass authentication.

I had a lot of problems with MAB in the lab and wasn’t very pleased with how it went.. maybe one to come back to another day. Basically I struggled getting MAB / RADIUS to kick in to the ISE, then it just worked randomly and the logging and verification looks wrong. Possibly a weird VIRL thing.. I want to try this on real kit soon or another image to see if any difference.

Posted in ISE

Cisco Identity Services Engine Lab 3 – 802.1x Signed Certificate

Ok so in this lab we now step up and issue a signed identity certificate to the ISE server. With this in place we can then force the client to validate the ISE identity certificate and valid the issuer of the root certificate from the AD CS. We can also connect to the web interface using a DNS name and HTTPS.

  1. Download the root certificate from the AD CS

URL http://192.168.68.250/certsrv/

2. Upload the root trust certificate to ISE

Import cert / the cert is already imported to the trust cert store on ISE

3. Upload the root cert to Host 1 trust store

**As I want to use this certificate for ISE admin, I want the client to be able to connect to https://ise1.dundermifflin.com using DNS / TLS.

Root cert now in store allowing the HTTPS web connection to the ISE web interface from Host 1.

Now lets test HTTPS access from Host1 to the ISE web admin:

We are looking good for HTTPS using the signed cert from the AD CS

4. Verify that Host1 can use 802.1x with the trusted certificate

Forcing the supplicant to verify the ISE certificate

How does the port look on the switch?

vIOS-L2-01#show authentication sessions interface gigabitEthernet 0/2

Interface    MAC Address    Method  Domain  Status Fg Session ID
----------------------------------------------------------------------
Gi0/2        5000.0003.0000 dot1x   DATA    Auth      000000000000000D00090E5F


Key to Session Events Blocked Status Flags:

  A - Applying Policy (multi-line status for details)
  D - Awaiting Deletion
  F - Final Removal in progress
  I - Awaiting IIF ID allocation
  P - Pushed Session
  R - Removing User Profile (multi-line status for details)
  U - Applying User Profile (multi-line status for details)
  X - Unknown Blocker

Runnable methods list:
  Handle  Priority  Name
    8        5      dot1x
    12       10     mab
    10       15     webauth

Also if we add details to the end:

Switch#show authentication sessions int gig1/1 details
            Interface:  GigabitEthernet1/1
          MAC Address:  5000.0003.0000
         IPv6 Address:  Unknown
         IPv4 Address:  192.168.68.2
            User-Name:  mscott
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-auth
     Oper control dir:  both
      Session timeout:  N/A
    Common Session ID:  C0A844010000000C003A4635
      Acct Session ID:  0x00000001
               Handle:  0x42000001
       Current Policy:  POLICY_Gi1/1

Local Policies:
        Service Template: DEFAULT_LINKSEC_POLICY_SHOULD_SECURE (priority 150)
      Security Policy:  Should Secure
      Security Status:  Link Unsecure

Server Policies:

Method status list:

       Method           State
       dot1x            Authc Success

**Ignore the switch port number here! 😛 This command was added to this post from a different lab.

We are good and in an ‘Auth’ status.

Also in ISE:

Verification

We can verify authentication in more detail via:

Lots of great information comes back:

Posted in ISE

Cisco Identity Services Engine Lab 2 – 802.1x Self Signed Certificate

802.1x with Self Signed Certificate

In this lab we will setup basic 802.1x authentication with PEAP, however we currently have a self signed certificate that we will not be able to validate. (In future labs we will bring the CA in and issue a trusted identity certificate to ISE)

The aim is for Michael Scott to be able to use 802.1x with a local identity from ISE. (In future labs we will add AD as an Identity Store)

Lab Topology

Method List for 802.1x

SW1(config)#aaa authentication dot1x default group radius

Use ISE for AAA network access (DACL/VLAN membership etc… more to follow)

SW1(config)#aaa authorization network default group radius

Use ISE for accounting

SW1(config)#aaa accounting dot1x default start-stop group radius

Include IP of Host/Supplicant as part of Authentication Requests that go to ISE:

**8 Framed IP address attribute

C3560X(config)#radius-server attribute 6 on-for-login-auth
C3560X(config)#radius-server attribute 8 include-in-access-req
C3560X(config)#radius-server attribute 25 access-request include

Enable 802.1x on switch globally

SW1(config)#dot1x system-auth-control

Edge Port Configuration

Current

SW1#sh run int gig0/2
Building configuration...

Current configuration : 105 bytes
!
interface GigabitEthernet0/2
 description Access Port to Host1
 media-type rj45
 negotiation auto
end

New

SW1#sh run int gig0/2
Building configuration...

Current configuration : 372 bytes
!
interface GigabitEthernet0/2
 description Access Port to Host1
 switchport mode access
 media-type rj45
 negotiation auto
 authentication host-mode multi-auth
 authentication open
 authentication port-control auto
 authentication periodic
 authentication timer reauthenticate server
 dot1x pae authenticator
 dot1x timeout tx-period 10
 spanning-tree portfast edge
end

802.1x commands / context


! Authenticate all MAC addresses on this port
 authentication host-mode multi-auth

! Leave the port open even if auth fails. I actually took this off to prove the concepts
 authentication open

! When we remove the 'open' authentication, enable 802.1x control the port
 authentication port-control auto

! recurring authentication
 authentication periodic

! let the ISE server decide how often this will occur
 authentication timer reauthenticate server

! set this port as an authenticator
 dot1x pae authenticator

! supplicant retry timeout
 dot1x timeout tx-period 10

Windows Host

  • Enable Supplicant / 802.1x
  • Start ‘Wired AutoConfig’ Service
  • Edit Network Connection
  • Edit PEAP – do not validate identity cert (As currently not part of the configuration)
  • Edit Additional Settings (To set a local account to be used instead of domain auth)

Enter the credentials!

If you are debugging radius on the switch then you will see lots of action when this kicks in.

This is what we want to see for successful authentication:

SW1#show authentication sessions interface GigabitEthernet 0/2

Interface    MAC Address    Method  Domain  Status Fg Session ID
----------------------------------------------------------------------
Gi0/2        5000.0003.0000 dot1x   DATA    Auth      C0A844FC0000000D00337637


Key to Session Events Blocked Status Flags:

  A - Applying Policy (multi-line status for details)
  D - Awaiting Deletion
  F - Final Removal in progress
  I - Awaiting IIF ID allocation
  N - Waiting for AAA to come up
  P - Pushed Session
  R - Removing User Profile (multi-line status for details)
  U - Applying User Profile (multi-line status for details)
  X - Unknown Blocker

Runnable methods list:
  Handle  Priority  Name
    5        5      dot1x
    9        10     mab
    7        15     webauth

Don’t forget if there are any issues we can always enable a radius debug as before.

Our host is authenticated and has connectivity!

Remember if the host fails to authenticate, the port is not in OPEN mode, so no traffic will flow. We can set the mode as OPEN so even if there is a 802.1x authentication issue, traffic will still pass.

We will iterate and improve this lab to use trusted certificates with PEAP.

Posted in ISE