I have been buried in APIC-EM stuff for awhile and since the new version of the CCNA is adding some APIC-EM topics this September I figured I may as well talk about the solution now so I can shamelessly be ahead of some of the other blogs out there 🙂

So what is APIC-EM?  The Application Policy Infrastructure Controller – Enterprise Module is one of Cisco’s SDN controllers that is focused on easing IWAN & QoS deployments, and providing an easier troubleshooting solution.

APIC-EM is free paradoxically has a steep price in the form of very large system requirements so your ability to follow along will depend on how big your ESX server is.

Server	APIC-EM
VMware Version	5.1/5.5/6.0
Virtual CPUs	6
Memory	64GB for a single node or 32GB per Node in cluster
HDD Size	500 GB
Throughput (Disk IOPS)	200 MB/s

Topology

The physical topology I’m using for this is fairly straight forward, I’m using VIRL to deploy 9 x CSRs  and 6 x IOSv routers with external flat networks. Server wise we’ll setup the APIC-EM server (obviously!) and I also have a Prime Infrastructure 3.1 server that we’ll use for some integration stuff. Because of the nature of testing all the things APIC-EM can do I’ll use a variety of real gear to help show some features.

Installation

The process of installing APIC-EM isn’t terribly complicated but as luck would have it I needed to rebuild my APIC-EM virtual machine anyway so lets go through the steps.

1. Accept the License Agreement

1. The installer will then validate your disk throughput, if you don’t make the cut the installer will give you an option to ignore the warning and proceed anyway.

1. Now we can either create a new cluster or join an existing one, since this is our first (and only) server we will pick “Create a new APIC-EM cluster”.

1. Then we enter in the IP information for the server and any static routes the server may need. Only thing of interest to note is that it is often best to specify a Virtual IP because APIC-EM so you can easily grow the cluster later.

1. Next we need to enter your Cisco CCO information.

1. Then we enter the password for the Linux server, you can optionally have the server randomly generate a password if you aren’t feeling particularly creative.

1. Same thing again for the admin username and password.

1. Next we enter in some NTP servers, I have one in my LAN so I’ll just use that.

1. On this page we decide whether to format the disks or not, I picked “yes” for fun but it doesn’t really matter for a first time install. You would leave it as “no” if you wanted to preserve existing data.

1. Now we can finally start the install, this will take a LONG time so make yourself a drink and find a good TV show to watch. You can also choose to save the configuration and exit if you have commitment issues.

Here is a kitty gif to help you pass the time

1. Once the install gets to Step 8 or so you’ll have an option of logging into Grapevine to watch the services starting if you are interested in seeing such a thing. The grapevine URL is: https://<apic-em&gt;:14141

The APIC-EM Grapevine appliance is Ubuntu so once you get on the command line you can adjust the server as needed. One thing you may want to consider is adding a few extra hard drives and configuring LVM to help easy the high disk speed requirements.

Depending on your setup you may want to setup GRE tunnels and/or Quagga to help simplify the routing to your lab.A pro-tip for using VIRL would be to install openvpn so you can connect APIC-EM directly to VIRL’s flat network.

APIC-EM

The server is made up of several applications that each have their own focus.

• Discovery / Inventory / Topology – This creates an network inventory and topology map of the network.

• Network Plug and Play – This is focused on easing router deployments by allowing routers to contact the APIC-EM server to download configuration and firmware.
• Path Visualization – This application shows the path traffic would take to each its destination, this is very handy in more complicated IWAN networks.
• IWAN – This app tries to make configuring IWANs a GUI based experience with minimal need for using the CLI.
• EasyQoS – This app allows QoS policies to be managed in a web GUI. It is currently in beta at the moment.

The APIC-EM server also allows REST-API access so you can mess around with the solution using Python. We’ll check that out later on.

Here is the expanded menu that shows all the APIC-EM applications, for the rest of this I’ll keep the menu collapsed to save a bit of screen space since this will be a pretty screenshot heavy blog post.

Settings

Let’s start by looking at the most logical place – the settings page!
We’ll call out some of the more useful options.

Prime Credentials – This page allows you to integrate the APIC-EM controller with a Prime Infrastructure server, since I happen to have a PI server I put in the IP address and the login info.

Discovery Credentials – This page allows you to save the login and SNMP information used by the discovery process.

EasyQoS – This page allows you to enable the EasyQoS feature
Note: Enabling the feature is permanent and can only be undone by factory defaulting the server.

Update – This allows you upgrade APIC-EM to the latest and greatest version.

Discovery

The Discovery application’s job is to find and inventory the devices in your network. It uses the following methods to scan the device and figure out all its info:

• Cisco Discovery Protocol (CDP)
• Link Layer Discovery Protocol (LLDP)
• LLDP-MED
• SNMPv2
• SNMPv3
• IP Device Tracking

To discover your devices, enter in your network range as well as the saved SNMP and login credentials. You can also choose to enter in the login information directly by pressing the Enter job specific button under SNMP and CLI credentials.

CDP can also be used by specifying a seed IP that APIC-EM can query and figure out all the devices connected to it. APIC-EM will follow CDP up to 16 levels to track down devices.

Once discovery is completed devices will be put into the Inventory application and will be periodically re-run to keep things accurate. The discovery job can’t be edited through the web interface once it is created so you need to make a new one if you need to edit something.

Inventory

Once the discovery is done any network devices and hosts that are found will be kept track of in either the Network or Host inventory. This database is shared with other features such as Path Visualization or the Topology map.

Network Inventory

After discovery completes the routers and switches will eventually show up in the network inventory application. If the status is Managed then discovery has completed successfully, if you see Partial Collection Failure then you need to make sure your login information is accurate. If devices are missing then you need to make sure APIC-EM can reach all the devices properly. One thing I found with testing with VIRL is that the IOSvL2 node isn’t supported and it seems to only import a single switch even if you are using multiple switches.

It is possible to customize the discovered devices by adding its location and tags to the object.

You can use the Set Location button to…well…set the device’s location. To set a location, create a new location and then use Add Marker to put mark the device’s location.

The Set Device Tag button lets you add a tag for device types that you can use with other features to help you filter things.

Likewise the Set Policy Tag does the same thing except your grouping by policy, this is especially useful for the EasyQoS feature.

If you click a device name it will provide an overview of the device’s hardware, interfaces, OS, Uptime, and MGMT IP.

You can also press the Layout button to customize the information you see on the screen.

For example, after selecting Config, it is now possible to see the running-configuration on the device.

Host Inventory

In contract to the network inventory with its many options, the host inventory simply keeps tracks of any hosts it finds in the network and really just adds the information to the topology map.

Topology

The topology application attempts to draw out the topology based on the network and host inventory. Because using VIRL and CSRs doesn’t lead to particularly interesting maps I will also use a few other topologies and hardware platforms to help show off the feature.

As I mentioned earlier the IOSvL2 nodes are unsupported so they show up as a question mark and doesn’t quite map right. Here we can a standard 9 x CSR topology connected to a virtual switch. The cloud node represents the external connection to APIC-EM.

We can make the topology a bit cleaner by aggregating the cloud node and the switch node by selecting the two nodes and pressing the Aggregate Selected button.

Now we see a single connection from the routers to the aggregate node instead of two, you can also do this for things like LACP links or wherever else it makes sense.

Here is a topology with actual routers and switches, notice the topology map is now able to place hosts in the network.

We can get more information by clicking on either the device or the connections.

Here is a bit more of an interesting topology for us to play with.

The Layer button allows you to highlight the portions of the network that use particular Layer2 vlans, Layer 3 routing protocols, and/or VRFs.

The tag button allows you to highlight devices that use a particular tag.

We can also use the Topology Structure button to organize the diagram based on one of the following

Enterprise: the map is built based on the device name / IP scheme

Device Type & Role: Keeps device types together

Connections: Focuses on how devices are connected to each other.

Clicking on a node will give you more information about the device and what is connected to it. You can also specify Device Tagging from here.

Network Plug in Play

This application allows you to easily provision new routers into your network by having the router automatically download its firmware and configuration from APIC-EM and/or Prime Infrastructure. If you have been around for awhile you may think this sounds fairly similar to the old service config feature.

To configure plug in play we create a PNP profile and point it to the APIC-EM server.
Note: APIC-EM seems pretty picky when it comes to mgmt vrfs so it is best to avoid them at this time.

R01(config)#pnp profile PNP-Profile
R01(config-pnp-init)# transport http ipv4 10.20.2.210 port 80

 

You can also use DHCP discovery by using DHCP option 43/60 to point to the APIC-EM address or to a DNS record pnpserver.localhost that resolves to the APIC-EM address.

Since the point of Network PNP is to push configuration and firmware, lets upload a IOS image.

We’ll also type up a simple config to upload to our routers.

You can also use the Bulk Import button to upload a CSV template with all the information PNP needs.

After a bit we should see the routers eventually show up in the unplanned devices tab.

To provision a device, select our PNP.txt config for each router, select everything and click Claim. Since I’m using CSRs at the moment we can’t play with the firmware aspect but we would just set the image as well on the screen.

After a while the routers should show as provisioned, behind the scenes the router has added APIC-EM as a PKI trustpoint, changed the PNP configuration to use HTTPS instead.

R01(config-router)#do sh run | s pnp
crypto pki trustpoint pnplabel
 enrollment pkcs12
 revocation-check none
crypto pki certificate chain pnplabel
 certificate ca 1511890402066176111485434789897892148683
 308203BE 308202A6 A0030201 02021415 11890402 06617611 14854347 89897892 
 14868330 0D06092A 864886F7 0D01010B 05003081 88312D30 2B060355 04031324 
 31653935 66623639 2D316338 622D3431 34342D38 6639352D 33633430 66346662 
 37343363 310B3009 06035504 06130255 53311330 11060355 0408130A 43616C69 
 666F726E 69613110 300E0603 55040713 0753616E 4A6F7365 31133011 06035504 
 0B130A41 50494345 4D2D5344 4E310E30 0C060355 040A1305 43697363 6F301E17 
 0D313630 35313831 35353031 365A170D 32313035 31373135 35303136 5A305D31 
 14301206 03550403 130B3130 2E32302E 322E3231 30311330 11060355 0408130A 
 43616C69 666F726E 6961310B 30090603 55040613 02555331 0E300C06 0355040A 
 13054369 73636F31 13301106 0355040B 130A4150 4943454D 2D53444E 30820122 
 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101 00C7A5AD 
 569B1BB2 7F4EC08A 6F5AC9B3 87D0968B B880EEBC B4132275 FA45E92F AF85F042 
 B4B41115 81F5406C BB11C2DC 07839500 547B09D5 5BFECA7F 19620746 CC701803 
 7D077335 2BED38AA FB4B19B3 5D401692 DE30816F 62A20EB6 1591038E B6DEB7F0 
 23BDF944 98C15BC8 8162FEF1 887DA53E A1457952 A07C89D3 919DEA97 D06DB511 
 17CBC4A9 8D0B48C1 117E3827 8C0B1D6A 3BB81597 558B12F4 E0FA2B09 2440C1F1 
 8B32D72F 868CDB30 F632990A 48338D0E 4B1FFDDE B9A2A0CB 973358E6 452E377F 
 00B7C13A EF91ABA3 3C9F85CB B948940E 05914283 2C31D308 A78CD7F9 3055FDFC 
 B83F53B8 1075D9D5 659ABC4E 39C88466 4EBF2A64 7DDAD3AE 5BB682FB 91020301 
 0001A34A 30483009 0603551D 13040230 00300B06 03551D0F 04040302 05A0301D 
 0603551D 25041630 1406082B 06010505 07030106 082B0601 05050703 02300F06 
 03551D11 04083006 87040A14 02D2300D 06092A86 4886F70D 01010B05 00038201 
 010019FE 5787FF0D 7D4ED52B 5EC74BB1 10F2B246 0F74EDF3 65745CF3 4A00EF4B 
 E6CAAF53 B8BE1B60 D640CCD2 48F2A6E3 B23CF475 C3A29C49 09C6D1D3 4EA58539 
 BC839BDB C4B639EA BFDF03BC 1E43DC19 116F79E0 5168A1E4 422A7128 DE905B4C 
 9C618677 168E3F50 4A9CCB5E A3172C70 763B7D44 8CB389EB 6A4A1C4B 0DC217BA 
 330741BF C9F296DB 5828B035 8AF4F895 0E8F8253 C3BFC4BB 67B65E3F 83C75393 
 51A448CE A86EA502 01F21379 7C60D9D5 573AB37E 37976D5F D57C3674 6BB0D93A 
 26FDC540 6FA79A29 1AB6BA66 2FB57D87 73EB5539 D3AA4F64 E404B8DE F780AB22 
 8CB6D2A8 6D4F8301 144C6035 149CC8D7 C2B87554 517AF783 AB5782EC D7A7B4BF 
 A80F
 quit
pnp profile PNP-Profile
 transport https ipv4 10.20.2.210 port 443

APIC-EM automatically kicksoff a discovery scan for the new router

The Projects tab lets you organize routers into groups by creating a project and adding the router into the group. You can optionally choose to use an external TFTP server for a project.

If you setup Prime Infrastructure and APIC-EM to talk to each other then Prime will automatically learn about nodes that contact APIC-EM via Network PNP.

In Prime go to Adminstration -> Servers -> APIC-EM and add the server VIP address.

Also make sure that Global PnP / Zero Touch Provisioning is set to APIC-EM.

After this is done nodes will magically appear in PI as they try to register.

In Prime Infrastructure you can also create templates that can be applied to the routers as part of the PNP process. Once this is done Prime creates a APIC-EM PNP project for the selected routers and pushes the config as part of the provisioning process.

 

This post is getting kinda long so here is another kitty

Path Visualization

If you are going for the new CCNA then this section will be the most relevant to you, Path Visualization draws out the path a packet takes to get to its destination, kinda like a visual traceroute. This tool is especially handy in more complicated networks such as IWAN deployments since you can see the actual physical path a packet takes and you can see where the packet dies if it doesn’t make it to its destination.

To do a trace:

1. Click Start a new Path Trace
2. Select the source router
3. Select the source interface
4. Select the destination router
5. Select the destination interface

You can also choose to include port information which is useful for making sure ACLs aren’t blocking traffic and for testing ECMP since CEF uses the port information to figure out the load balancing. You can also enable interface and QoS stats and have APIC-EM continuously run the test every 30 seconds so you can see any changes that may occur.

In this example the several CSR routers are connected a virtual switch and each router is logically connected to its neighboring router via sub-interfaces.

 

In this example we can see a mock MPLS network with typical PE and P routers (however MPLS is not actually turned on because APIC-EM didn’t like the VRFs) we can see the router passes through OSPF, ISIS, and EIGRP and also uses ECMP through the “core”.

We can also turn on interface and QoS statistics for a bit more information.

For fun, I broke the reverse path in the above example. APIC-EM was able to figure out there was a loop in the reverse path. We can have Path Visualization figure out the reverse path  by pressing the reverse button.

This example just shows a real switch in the mix.

EasyQoS

EasyQoS attempts to ease QoS configurations by allowing advanced QoS policies to be created by just clicking things.

Because the feature is still in beta you have to first enable it in the settings (seen above), the feature also uses Policy Tags so you assign routers you want to receive the QoS policy in Network Inventory.

To configure a policy:

1. Assign a group of routers a policy tag, I’m using Branch in this example.
2. In EasyQoS click the tag name and create a policy name, as you can see the name doesn’t like spaces.
3. Click create and then edit policy.

From here you can search for the applications you want to adjust or you can create a new application. I’ll create a new application for the fun of it.

Enter in the application details and then press create application.

Now we can pick one of three values: Business Relevant, Default, Business irrelevant.
Basically relevant traffic is deemed important and gets more priority and irrelevant doesn’t. Click Apply policy when your done.

Here is the config that EasyQoS puts on our routers. That is a lot of typing!

C4K-R01#sh run | s class-map|policy-map
class-map match-any prm-EZQOS_12C#REALTIME
match dscp cs4
class-map match-all prm-MARKING_IN#CONTROL
match protocol attribute traffic-class network-control
match protocol attribute business-relevance business-relevant
class-map match-all prm-MARKING_IN#MM_STREAM
match protocol attribute traffic-class multimedia-streaming
match protocol attribute business-relevance business-relevant
class-map match-all prm-MARKING_IN#OAM
match protocol attribute traffic-class ops-admin-mgmt
match protocol attribute business-relevance business-relevant
class-map match-all prm-MARKING_IN#SCAVENGER
match protocol attribute business-relevance business-irrelevant
class-map match-all prm-MARKING_IN#SIGNALING
match protocol attribute traffic-class signaling
match protocol attribute business-relevance business-relevant
class-map match-all prm-MARKING_IN#BROADCAST
match protocol attribute traffic-class broadcast-video
match protocol attribute business-relevance business-relevant
class-map match-all prm-MARKING_IN#BULK_DATA
match protocol attribute traffic-class bulk-data
match protocol attribute business-relevance business-relevant
class-map match-all prm-MARKING_IN#MM_CONF
match protocol attribute traffic-class multimedia-conferencing
match protocol attribute business-relevance business-relevant
class-map match-any prm-EZQOS_12C#TRANS_DATA
match dscp af21
match dscp af22
match dscp af23
class-map match-all prm-MARKING_IN#VOICE
match protocol attribute traffic-class voip-telephony
match protocol attribute business-relevance business-relevant
class-map match-all prm-MARKING_IN#TUNNELED-NBAR
match protocol capwap-data
class-map match-all prm-MARKING_IN#REALTIME
match protocol attribute traffic-class real-time-interactive
match protocol attribute business-relevance business-relevant
class-map match-any prm-EZQOS_12C#CONTROL
match dscp cs6
class-map match-any prm-EZQOS_12C#MM_STREAM
match dscp af31
match dscp af32
match dscp af33
class-map match-any prm-EZQOS_12C#OAM
match dscp cs2
class-map match-any prm-EZQOS_12C#SCAVENGER
match dscp cs1
class-map match-any prm-EZQOS_12C#SIGNALING
match dscp cs3
class-map match-any prm-EZQOS_12C#BROADCAST
match dscp cs5
class-map match-any prm-EZQOS_12C#BULK_DATA
match dscp af11
match dscp af12
match dscp af13
class-map match-any prm-EZQOS_12C#MM_CONF
match dscp af41
match dscp af42
match dscp af43
class-map match-all prm-MARKING_IN#TRANS_DATA
match protocol attribute traffic-class transactional-data
match protocol attribute business-relevance business-relevant
class-map match-any prm-EZQOS_12C#VOICE
match dscp ef

policy-map prm-dscp#QUEUING_OUT
class prm-EZQOS_12C#VOICE
police rate percent 10
priority
class prm-EZQOS_12C#BROADCAST
police rate percent 10
priority
class prm-EZQOS_12C#REALTIME
police rate percent 13
priority
class prm-EZQOS_12C#MM_CONF
bandwidth remaining percent 15
fair-queue
random-detect dscp-based
class prm-EZQOS_12C#MM_STREAM
bandwidth remaining percent 15
fair-queue
random-detect dscp-based
class prm-EZQOS_12C#CONTROL
bandwidth remaining percent 3
class prm-EZQOS_12C#SIGNALING
bandwidth remaining percent 3
class prm-EZQOS_12C#OAM
bandwidth remaining percent 5
class prm-EZQOS_12C#TRANS_DATA
bandwidth remaining percent 15
fair-queue
random-detect dscp-based
class prm-EZQOS_12C#BULK_DATA
bandwidth remaining percent 6
fair-queue
random-detect dscp-based
class prm-EZQOS_12C#SCAVENGER
bandwidth remaining percent 1
class class-default
bandwidth remaining percent 37
fair-queue
random-detect dscp-based

policy-map prm-MARKING_IN
class prm-MARKING_IN#VOICE
set dscp ef
class prm-MARKING_IN#BROADCAST
set dscp cs5
class prm-MARKING_IN#REALTIME
set dscp cs4
class prm-MARKING_IN#MM_CONF
set dscp af41
class prm-MARKING_IN#MM_STREAM
set dscp af31
class prm-MARKING_IN#CONTROL
set dscp cs6
class prm-MARKING_IN#SIGNALING
set dscp cs3
class prm-MARKING_IN#OAM
set dscp cs2
class prm-MARKING_IN#TRANS_DATA
set dscp af21
class prm-MARKING_IN#BULK_DATA
set dscp af11
class prm-MARKING_IN#SCAVENGER
set dscp cs1
class prm-MARKING_IN#TUNNELED-NBAR
class class-default
set dscp default

ip nbar attribute-map Something
 attribute category other
 attribute sub-category other
 attribute traffic-class voip-telephony
 attribute business-relevance business-relevant
ip nbar attribute-map APIC-A_M-RELEVANT
 attribute business-relevance business-relevant
ip nbar attribute-map APIC-A_M-DEFAULT
 attribute business-relevance default
ip nbar attribute-map APIC-A_M-SCAVENGER
 attribute business-relevance business-irrelevant
ip nbar custom Something transport tcp id 37633
 ip address 1.1.1.1 
 port 9999 
 direction any
ip nbar attribute-set Something Something

Lastly there is a Dynamic QoS feature that attempts to adjust the QoS policy based the actual network traffic. You can turn it on by pressing the big ON button  on the bottom.

 

IWAN Application

I’m going to be covering DMVPN and IWAN in another post so I’ll cover this app there.

Python

Part of the power of APIC-EM is that it fully supports REST-API / Python so you can write your own scripts to achieve whatever you want to do. I won’t dive too deeply into this since I haven’t touched on python yet in this blog so I’ll instead use a slightly modified python script that Cisco provided on APIC-EM’s site.

This script queries the network inventory application and displays the software versions of all the nodes.

import requests,json
import re

### Disable invalid certificate warnings.
requests.packages.urllib3.disable_warnings()

apicem_ip = "10.20.2.210"

def createserviceticket():
    response = requests.post(
        url="https://"+apicem_ip+"/api/v1/ticket",
        headers={
            "Content-Type": "application/json",
        },
        verify=False,
        data=json.dumps({
            "username": 'admin',
            "password": 'meowcat'
        })
    )
    output = ('Response HTTP Response Body: {content}'.format(content=response.content))
    match_service_ticket = re.search('serviceTicket":"(.*cas)', output, flags=0)
    service_ticket = match_service_ticket.group(1)
    return service_ticket

url = "https://"+apicem_ip+"/api/v1/network-device"

response = requests.get(url,headers={"X-Auth-Token": createserviceticket(),"Content-Type": "application/json",},verify=False)

data = response.json()

device_list = data['response']
for device in device_list:
    print 'Hostname: %s' % device['hostname']
    print '     Software Version: %s '% device['softwareVersion']

Results

C:\Anaconda2\python.exe C:/Users/TPT/PycharmProjects/untitled/ListSoftwareVersions.py
Hostname: P01.testlab.com
 Software Version: 15.6(2)T 
Hostname: P02.testlab.com
 Software Version: 15.6(2)T 
Hostname: P03.testlab.com
 Software Version: 15.6(2)T 
Hostname: P04.testlab.com
 Software Version: 15.6(2)T 
Hostname: PE1.testlab.com
 Software Version: 15.6(2)T 
Hostname: PE2.testlab.com
 Software Version: 15.6(2)T 
Hostname: PE3.testlab.com
 Software Version: 15.6(2)T 
Hostname: R01.testlab.com
 Software Version: 15.6(1)S 
Hostname: R02.testlab.com
 Software Version: 15.6(1)S 
Hostname: R03.testlab.com
 Software Version: 15.6(1)S 
Hostname: R11.testlab.com
 Software Version: 15.6(1)S 
Hostname: R12.testlab.com
 Software Version: 15.6(1)S 
Hostname: R21.testlab.com
 Software Version: 15.6(1)S 
Hostname: R22.testlab.com

This concludes today’s post, see ya next time when I tackle a topic that requires less screenshots.