Ever wanted to play around with Cisco’s fancy STP killer but don’t feel like mortgaging your house to buy some Nexus 5 and 7k switches for your lab? Well as it turns out we can use VIRL to play with FabricPath.

What is FabricPath and why do I care?

FabricPath is a L2 implementation of the IS-IS routing protocol that aims to replace STP in Data Center networks. Why would we pick on STP? Because STP works by preventing certain links from sending traffic and in demanding data centers, every link counts.

It runs on Nexus 5ks, 6ks, and 7ks switches with F cards and needs the Enhanced L2 license to work.

FabricPath doesn’t use ethernet and instead uses the L2-IS-IS Ethertype (0x22f4)

FabricPath Terminology

Classic Ethernet – Refers to the Ethernet we all know and love, runs STP.

Leaf Switch – This is the edge switch that connects to the CE domain, everything connects to the leaf switches.

Spine Switch – This is the backbone switch, all ports are fabricpath ports.

FabricPath Core Ports – Ports that connects  the leaf switches to the spine or the spine switches to each other.

CE Edge Ports – Ports that connect the CE domain to the Leaf switches.

Let’s give it a go

To play with FabricPath we will make a typical spine and leaf topology using NX-OSv nodes. I’m going to use the server node as endpoints but it doesn’t really matter if you choose to use routers or switches instead.

We’ll start by building a topology like so.

Next we’ll rename the nodes to be a bit more descriptive, the Spine switches are on the top and the Leaf switches are below and the servers on the bottom.

Next we connect both Spine switches to each Leaf switch. In retrospect I should have connected the spine switches together but what am I going to do? Go back and update the screenshot? Don’t be ridiculous!

Then we’ll connect our servers to the Leaf switches.

When we’re done we’ll press the ‘Build configuration’ button and customize our servers.
If you don’t want to change the username like below the login will be cisco/cisco.

We’ll make the server an IP of 192.168.123.10x/24 where X is the host number.

I should also point out that the Server node in VIRL is a Ubuntu VM and does allow sudo so you can change the IPs etc manually if you want.

Now we’ll boot up the topology which should look something like this.

Now the time to actually configure FabricPath! The first thing we need to do is to install the fabricpath feature-set license and then activate the feature-set on each Nexus switch.

SPINE01(config)# install feature-set fabricpath 
SPINE01(config)# fea2016 Feb 7 00:02:50 SPINE01 %LICMGR-2-LOG_LIC_NO_LIC: No license(s) present for feature ENHANCED_LAYER2_PKG. Application(s) shut down in 119 days.
SPINE01(config)# feature-set fabricpath

When we do so we’ll see a log message about FP needing the Enhanced L2 license and it will shut down in 4 months or so.

Next it is a good idea to set the FP switch-id, if we don’t do it the value will be randomly assigned which can make things a bit messy. I’m going to make the Spine switches 7X and the Leafs 5X where X is the host number.

SPINE01(config)# fabricpath switch-id 71
SPINE02(config)# fabricpath switch-id 72
LEAF01(config)# fabricpath switch-id 51
LEAF02(config)# fabricpath switch-id 52
LEAF03(config)# fabricpath switch-id 53

With that out of the way we simply need to tell all the FP Core Ports that they need to run in fabricpath mode instead of regular old boring ethernet. One thing to keep in mind is that the Nexus ports will be in routed by default so we need to change them back to switchports.

First we’ll do the Spine switches

SPINE01(config-if-range)# show cdp neighbors interface e2/1-8 | begin ID
Device-ID Local Intrfce Hldtme Capability Platform Port ID
LEAF01(TB3E14E88BB)
 Eth2/1 153 R S s N7K-C7018 Eth2/1 
LEAF02(TB3E43932BB)
 Eth2/2 157 R S s N7K-C7018 Eth2/1 
LEAF03(TB3E784C95B)
 Eth2/3 155 R S s N7K-C7018 Eth2/1

SPINE01(config)# int e2/1-3
SPINE01(config-if-range)# switchport 
SPINE01(config-if-range)# switchport mode fabricpath

SPINE02(config)# show cdp neighbors interface e2/1-8 | begin ID
Device-ID Local Intrfce Hldtme Capability Platform Port ID
LEAF01(TB3E14E88BB)
 Eth2/1 141 R S s N7K-C7018 Eth2/2 
LEAF02(TB3E43932BB)
 Eth2/2 145 R S s N7K-C7018 Eth2/2 
LEAF03(TB3E784C95B)
 Eth2/3 143 R S s N7K-C7018 Eth2/2 

SPINE02(config)# interface e2/1-3
SPINE02(config-if-range)# switchport 
SPINE02(config-if-range)# switchport mode fabricpath

And then the Leaf switches.

LEAF01(config)# show cdp neighbors interface e2/1-8 | begin ID
Device-ID Local Intrfce Hldtme Capability Platform Port ID
SPINE01(TB3EB8B028B)
 Eth2/1 167 R S s N7K-C7018 Eth2/1 
SPINE02(TB3E9DE6AFB)
 Eth2/2 142 R S s N7K-C7018 Eth2/1

LEAF01(config)# int e2/1-2
LEAF01(config-if-range)# switchport 
LEAF01(config-if-range)# switchport mode fabricpath

LEAF02(config)# int e2/1-2
LEAF02(config-if-range)# switchport 
LEAF02(config-if-range)# switchport mode fabricpath

LEAF03(config)# int e2/1-2
LEAF03(config-if-range)# switchport 
LEAF03(config-if-range)# switchport mode fabricpath

Once we are done with all that there will be a flood of console as FP figures itself out and IS-IS forms neighbors. After a minute we can verify that IS-IS is happy with:

SPINE01(config)# show fabricpath isis adjacency 
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
LEAF01 N/A 1 UP 00:00:25 Ethernet2/1
LEAF02 N/A 1 UP 00:00:26 Ethernet2/2
LEAF03 N/A 1 UP 00:00:26 Ethernet2/3

SPINE02(config)# show fabricpath isis adjacency 
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
LEAF01 N/A 1 UP 00:00:23 Ethernet2/1
LEAF02 N/A 1 UP 00:00:29 Ethernet2/2
LEAF03 N/A 1 UP 00:00:23 Ethernet2/3

Of course FabricPath doesn’t really do anything useful without L2 traffic so lets make vlan 123 on each switch, assign it to the server ports, and give it a SVI on each Leaf with a 192.168.123.5x/24

SPINE01(config)# vlan 123
SPINE01(config-vlan)# name Servers
SPINE01(config-vlan)# exit

LEAF01(config)# int e2/3
LEAF01(config-if)# switchport
LEAF01(config-if)# switchport mode access 
LEAF01(config-if)# switchport access vlan 123

LEAF01(config)# feature interface-vlan
LEAF01(config)# int vlan 123
LEAF01(config-if)# ip add 192.168.123.51/24
LEAF01(config-if)# no shut

So the question is, are we working right now?

meow@S1:~$ ping 192.168.123.102
PING 192.168.123.102 (192.168.123.102) 56(84) bytes of data.
From 192.168.123.101 icmp_seq=1 Destination Host Unreachable
From 192.168.123.101 icmp_seq=2 Destination Host Unreachable
From 192.168.123.101 icmp_seq=3 Destination Host Unreachable

No we are not! The reason why is vlan 123 isn’t actually using the FabricPath topology because it is still in CE mode. As a sidenote each switch thinks it is the root for vlan 123 because no one is answering any BPDUs.

We can change the vlan type by doing the following on each switch:

SPINE01(config)# vlan 123
SPINE01(config-vlan)# mode fabricpath
SPINE01(config-vlan)# exit

Now our servers can talk to each other!

meow@S1:~$ ping 192.168.123.102
PING 192.168.123.102 (192.168.123.102) 56(84) bytes of data.
64 bytes from 192.168.123.102: icmp_seq=1 ttl=64 time=16.9 ms
64 bytes from 192.168.123.102: icmp_seq=2 ttl=64 time=5.57 ms
64 bytes from 192.168.123.102: icmp_seq=3 ttl=64 time=6.37 ms

Note: You can normally see FabricPath information in show mac address-table but it doesn’t seem to be displaying correctly in NX-OSv

If you are security minded we can also add authentication to the FP Hellos by key chain and adding the authentication to the interface.

SPINE01(config)# key chain FP_HELLO_AUTH
SPINE01(config-keychain)# key 1
SPINE01(config-keychain-key)# key-string MEOW
SPINE01(config-keychain-key)# int e2/1
SPINE01(config-if)# fabricpath isis authentication-type md5 
SPINE01(config-if)# fabricpath isis authentication key-chain FP_HELLO_AUTH

SPINE01(config-if)# show fabricpath isis interface e2/1 | in Auth
 Authentication type MD5
 Authentication keychain is FP_HELLO_AUTH
 Authentication check specified

We can also authenticate the IS-IS LSPs like so:

SPINE01(config)# key chain FP_LSP_AUTH
SPINE01(config-keychain)# key 1
SPINE01(config-keychain-key)# key-string CATS!
SPINE01(config-keychain-key)# exit
SPINE01(config-keychain)# exit

SPINE01(config)# fabricpath domain default 
SPINE01(config-fabricpath-isis)# authentication-type md5 
SPINE01(config-fabricpath-isis)# authentication key-chain FP_LSP_AUTH

But(t) what about BUM (Broadcast, Unknown Unicast, and Multicast) traffic you ask? It is handled by FP’s Multi Destination Tree, the second tree handles multicast traffic while the first tree handles everything else. Generally we want the Spines to win the election so we can change the priority to be higher. The highest priority wins the first tree and the sencond highest wins the second tree.

SPINE01(config-fabricpath-isis)# show fabricpath isis topology summary 
FabricPath IS-IS Topology Summary
Fabricpath IS-IS domain: default
MT-0
 Configured interfaces: Ethernet2/1 Ethernet2/2 Ethernet2/3
 Max number of trees: 2 Number of trees supported: 2
 Tree id: 1, ftag: 1, root system: fa16.3ec2.1786, 52
 Tree id: 2, ftag: 2, root system: fa16.3e82.919b, 51

Currently we see that Leaf2 is the Tree 1 root and Leaf1 is the Tree 2 root.
Lets change it to so the spines do the heavy lifting.

SPINE01(config)# fabricpath domain default 
SPINE01(config-fabricpath-isis)# root-priority 255

SPINE02(config)# fabricpath domain default
SPINE02(config-fabricpath-isis)# root-priority 254

SPINE01(config-fabricpath-isis)# show fabricpath isis topology summary 
FabricPath IS-IS Topology Summary
Fabricpath IS-IS domain: default
MT-0
 Configured interfaces: Ethernet2/1 Ethernet2/2 Ethernet2/3
 Max number of trees: 2 Number of trees supported: 2
 Tree id: 1, ftag: 1, root system: fa16.3e72.7069, 71
 Tree id: 2, ftag: 2, root system: fa16.3e81.f493, 72

Last thing worth mentioning for now is that because FP runs on IS-IS we can adjust the routing metric to do some traffic engineering with the fabricpath isis metric command under a interface.