Today’s post will quickly go over a neat feature of IOS-XE that lets you run a virtual machine directly off your Cisco router, the feature is meant for the ASRs and the 4K routers but since CSRs use the same code base we can have a virtual router host a virtual machine! This is different from using a UCS-E blade to run a full blown ESXi install, rather the router uses KVM to directly run the VM, it also supports containers which we’ll play with on some Nexus 9ks.
The first thing to do is to deploy a CSR to your Vmware ESXi host, I tested the feature with Vmware Workstation but it always froze and I’m not sure where the fault lies. So ESXi it is!
Once you deploy the CSR, make sure you enable Hardware Virtualization, it is also a good idea to bump up the RAM and CPU (I gave it 8gb) since the router will be running Fedora as well as being a router.
With that out of the way let’s download the Fedora cloud qemu2 file on ideally a Linux box since we have to do some prep work, though if you want to try out Window’s new bash support you should be able to mostly follow along 🙂
We’ll also need to download libguestfs-tools tools so we can add the root password to the image since cloud images don’t ship with credentials since they assume your using them with OpenStack.
[root@centos7-1 tmp]# yum install libguestfs-tools -y
The password is set by using the virt-sysprep command as shown here, it will take a couple minutes to work its magic.
[root@centos7-1 ~]# virt-sysprep -a fedora.qcow2 --root-password password:meowcat [ 0.0] Examining the guest ... [ 23.0] Performing "abrt-data" ... [ 23.0] Performing "bash-history" ... [ 23.0] Performing "blkid-tab" ... [ 23.0] Performing "crash-data" ... [ 23.0] Performing "cron-spool" ... [ 23.0] Performing "dhcp-client-state" ... [ 23.0] Performing "dhcp-server-state" ... [ 23.0] Performing "dovecot-data" ... [ 23.0] Performing "logfiles" ... [ 23.0] Performing "machine-id" ... [ 23.0] Performing "mail-spool" ... [ 23.0] Performing "net-hostname" ... [ 23.0] Performing "net-hwaddr" ... [ 23.0] Performing "pacct-log" ... [ 23.0] Performing "package-manager-cache" ... [ 23.0] Performing "pam-data" ... [ 23.0] Performing "puppet-data-log" ... [ 23.0] Performing "rh-subscription-manager" ... [ 23.0] Performing "rhn-systemid" ... [ 23.0] Performing "rpm-db" ... [ 23.0] Performing "samba-db-log" ... [ 23.0] Performing "script" ... [ 23.0] Performing "smolt-uuid" ... [ 23.0] Performing "ssh-hostkeys" ... [ 23.0] Performing "ssh-userdir" ... [ 23.0] Performing "sssd-db-log" ... [ 23.0] Performing "tmp-files" ... [ 23.0] Performing "udev-persistent-net" ... [ 23.0] Performing "utmp" ... [ 23.0] Performing "yum-uuid" ... [ 23.0] Performing "customize" ... [ 23.0] Setting a random seed [ 23.0] Setting passwords [ 28.0] Performing "lvm-uuids" ...
Next we need to make a OVA file that consists of a few files:
package.yaml – This file contains instructions for how to run the VM, it is similar to a VMX file from VMware.
version.ver – This is a simple version control doc
qemu2 file – This is the actual Fedora HD that we will boot
mf file – This is a sha1 checksum file.
Here is an example of the package.yaml file, for the most part you can copy and paste this for now. YAML is nice because it is very easy to understand what it is doing if you want to adjust a few things.
the-packet-thrower@HOME01:/mnt/c/Fedora$ cat package.yaml manifest-version: 1.0 info: name: fedora description: "KVM Fedora 24" version: 1.1 app: # Indicate app type (vm, paas, lxc etc.,) apptype: vm resources: cpu: 10 memory: 2048000 vcpu: 1 disk: - target-dev: hdc file: fedora.qcow2 interfaces: - target-dev: net1 serial: - console - aux # Specify runtime and startup startup: runtime: kvm boot-dev: hd
The version.ver can just be left at 1.0
the-packet-thrower@HOME01:/mnt/c/Fedora$ cat version.ver 1.0
Next we need to generate the sha1 checksums for the mf file
Note: Yes I have switched over to Windows 10 Bash, Bash on Windows! Awesome!!!
the-packet-thrower@HOME01:/mnt/c/Fedora$ sha1sum * dd1f574bc17fd642dfcad10e846f98f8568cc81f fedora.mf c3b63972a7eae9da368e71eaf54040f5179e511f fedora.qcow2 0b3b9b7a70e1be1c6dd70f8c64b04df66a4807ec package.yaml 61652cd1568dcf2614df833eba241755eee34e89 version.ver
Then we make put all the checksums into the mf file like so, the filename should match the qemu2 name.
the-packet-thrower@HOME01:/mnt/c/Fedora$ cat fedora.mf SHA1(fedora.qcow2)= c3b63972a7eae9da368e71eaf54040f5179e511f SHA1(package.yaml)= 0b3b9b7a70e1be1c6dd70f8c64b04df66a4807ec SHA1(version.ver)= 61652cd1568dcf2614df833eba241755eee34e89
With that all done we can now use the tar utility to make an OVA file out of them.
the-packet-thrower@HOME01:/mnt/c/Fedora$ tar -cvf Fedora.ova fedora.qcow2 fedora.mf package.yaml version.ver fedora.qcow2 fedora.mf package.yaml version.ver
Having an OVA file isn’t going to help us unless we add it to the router so lets setup SCP.
CSR01(config)#ip domain-name testlab.com CSR01(config)#crypto key generate rsa modulus 2048 % You already have RSA keys defined named CSR01.testlab.com. % They will be replaced. % The key modulus size is 2048 bits % Generating 2048 bit RSA keys, keys will be non-exportable... [OK] (elapsed time was 0 seconds) CSR01(config)#ip scp server enable CSR01(config)#username admin privilege 15
Because the unlicensed CSR doesn’t allow much bandwidth through it, this will take awhile so now would be a good time to make a tea.
$ scp Fedora.ova admin@10.10.2.119:Fedora.ova Password: Fedora.ova 100% 199MB 166.1KB/s 20:24 Connection to 10.10.2.119 closed by remote host.
Once that is uploaded we will need to tell the router that is fine to run unsigned OVA files
CSR01(config)#virtual-service CSR01(config-virt-serv-global)# signing level unsigned
Now we’re at the point were we actually install the image! The name we specify is how we interact with the VM so you might want to use a easy to type name.
CSR01#virtual-service install name Fedora package bootflash:fedora.ova Installing package 'bootflash:/fedora.ova' for virtual-service 'Fedora'. Once the install has finished, the VM may be activated. Use 'show virtual-service list' for progress.
While it does its thing we can check the status of the install as well as list any installed VMs like so
CSR01(config)# do show virtual-service list Virtual Service List: Name Status Package Name ------------------------------------------------------------------------------ csr_mgmt Installed iosxe-remote-mgmt.16.03.01.ova Fedora Activated Fedora.ova
We can also use show virtual-service to verify the router supports KVM (probably should have done that first eh?)
CSR01(config)# do show virtual-service Virtual Service Global State and Virtualization Limits: Infrastructure version : 1.7 Total virtual services installed : 2 Total virtual services activated : 1 Machine types supported : KVM, LXC Machine types disabled : none Maximum VCPUs per virtual service : 1 Resource virtualization limits: Name Quota Committed Available -------------------------------------------------------------- system CPU (%) 75 10 65 memory (MB) 4096 2000 2096 bootflash (MB) 20000 342 5638
In order to get network connectivity to the VM we create a Virtual Port Group, if you want to use multiple NICs with the VM then you will have to edit the package.yaml file to allow it. You can either add a IP or use unnumbered depending on what suites your environment, I’m going to opt to add a IP.
CSR01(config)#int virtualportGroup 1 CSR01(config-if)#ip add 10.10.3.1 255.255.255.0 CSR01(config-if)#exit
I’m also going to setup NAT so the VM can get online.
CSR01(config)#ip access-list extended NAT CSR01(config-ext-nacl)# permit ip 10.10.3.0 0.0.0.255 any CSR01(config-ext-nacl)#exit CSR01(config)#interface virtualportgroup 1 CSR01(config-if)#ip nat inside CSR01(config-if)#int g1 CSR01(config-if)#ip nat outside
Then we go under the VM configuration to assign the Virtual Port Group and assign the IP address for the VM.
Note: The guest ip address does not configure the Fedora OS, nor does the VM have to use that exact IP, it is more of a visual than anything else.
Lastly when we are done configuring we turn on the VM with the activate keyword.
I would give the VM a few minutes to boot before you use the virtual-service connect command unless you want to see all the boot-up messages, also if you try it too early the router might kick you out of the console session.
Enter the root password we setup earlier and we should be on the Fedora box!
CSR01#virtual-service connect name Fedora console Connected to appliance. Exit using ^c^c^c Fedora 24 (Cloud Edition) Kernel 4.5.5-300.fc24.x86_64 on an x86_64 (ttyS0) localhost login: root Password: Last login: Sat Aug 20 19:01:05 on ttyS0
Because the linux OS isn’t configured at all, I’ll quickly configure a IP address, default gateway, and dns server
[root@localhost ~]# ifconfig eth0 10.10.3.100/24 [root@localhost ~]# route add default gateway 10.10.3.1 SIOCADDRT: File exists [root@localhost ~]# ping 4.2.2.2 -c 5 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_seq=1 ttl=54 time=38.5 ms 64 bytes from 4.2.2.2: icmp_seq=2 ttl=54 time=37.7 ms 64 bytes from 4.2.2.2: icmp_seq=3 ttl=54 time=37.4 ms 64 bytes from 4.2.2.2: icmp_seq=4 ttl=54 time=37.6 ms 64 bytes from 4.2.2.2: icmp_seq=5 ttl=54 time=37.8 ms --- 4.2.2.2 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4007ms rtt min/avg/max/mdev = 37.450/37.841/38.522/0.400 ms. [root@localhost ~]# echo "nameserver 8.8.8.8" > /etc/resolv.conf [root@localhost ~]# ping google.ca PING google.ca (216.123.194.113) 56(84) bytes of data. 64 bytes from cache.google.com (216.123.194.113): icmp_seq=1 ttl=61 time=7.43 ms 64 bytes from cache.google.com (216.123.194.113): icmp_seq=2 ttl=61 time=5.87 ms
Lets end this lab by setting up Quagga so that the Cisco router can run OSPF with the linux VM.
[root@localhost ~]# dnf install quagga -y Last metadata expiration check: 0:14:13 ago on Sat Aug 20 21:48:19 2016. Dependencies resolved. ================================================================================t Package Arch Version Repository Size ================================================================================ Installing: ....SKIPPED.... Complete!
Next I’ll quickly setup Quagga, basically I’m enabling IP forwarding, creating some config for the ospfd, enabling the daemons and starting it. Quagga will be covered in more depth next post.
[root@localhost ~]# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf [root@localhost ~]# sysctl -p /etc/sysctl.conf net.ipv4.ip_forward = 1 [root@localhost ~]# setsebool -P zebra_write_config 1 echo 'WATCH_DAEMONS="zebra bgpd ospfd ospf6d ripd ripngd"'>> /etc/sysconfig/quagga [root@localhost ~]# echo "enable password cisco" > /etc/quagga/ospfd.conf [root@localhost ~]# chmod 744 /etc/quagga/ospfd.conf [root@localhost ~]# systemctl enable ospfd Created symlink from /etc/systemd/system/multi-user.target.wants/ospfd.service to /usr/lib/systemd/system/ospfd.service. [root@localhost ~]# systemctl enable zebra Created symlink from /etc/systemd/system/multi-user.target.wants/zebra.service to /usr/lib/systemd/system/zebra.service. [root@localhost ~]# systemctl start zebra [root@localhost ~]# systemctl start ospfd
Lastly we connect to Quagga and enable OSPF. Now our router is routing with the VM! This can be handy for generating test hosts or running tools like IPERF or SNMP tools from the VM.
[root@localhost ~]# vtysh Hello, this is Quagga (version 0.99.24.1). Copyright 1996-2005 Kunihiro Ishiguro, et al. localhost.localdomain.localdomain# conf t localhost.localdomain.localdomain(config)# hostname Quagga Quagga(config)# do wr Building Configuration... Configuration saved to /etc/quagga/zebra.conf Can't backup old configuration file /etc/quagga/ospfd.conf.sav. [OK] Quagga(config)# router ospf Quagga(config-router)# network 10.10.3.0/24 area 0 Quagga(config-router)# *Aug 20 22:12:20.391: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.3.100 on VirtualPortGroup1 from LOADING to FULL, Loading Done
Quagga# show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel, > - selected route, * - FIB route K>* 0.0.0.0/0 via 10.10.3.1, eth0 O>* 10.10.2.0/24 [110/11] via 10.10.3.1, eth0, 00:00:04 O 10.10.3.0/24 [110/10] is directly connected, eth0, 00:00:08 C>* 10.10.3.0/24 is directly connected, eth0 O>* 10.200.1.0/24 [110/12] via 10.10.3.1, eth0, 00:00:04 O>* 10.200.2.0/24 [110/12] via 10.10.3.1, eth0, 00:00:04 O>* 10.200.3.0/24 [110/12] via 10.10.3.1, eth0, 00:00:04 C>* 127.0.0.0/8 is directly connected, lo Quagga#
Containers
The Nexus 9k switches ship with a guestshell container that allows lets you switch to a Red Hat shell in case you want to do some Linux based functions.
NX01(config)# show virtual-service list Virtual Service List: Name Status Package Name ----------------------------------------------------------------------- guestshell+ Activated guestshell.ova
You can activate the shell by typing guestshell enable which is similar to the activate command we used above.
To switch over to the shell simply type guestshell
NX01# guestshell enable 2016 Aug 21 01:40:38 NX01 %$ VDC-1 %$ %VMAN-2-ACTIVATION_STATE: Activating virtual service 'guestshell+' NX01# guestshell [guestshell@guestshell ~]$
From here we are connected to the Linux shell and and do all sorts of Linux things. One neat feature is the dohost command which allows you to access the Nexus shell to do regular CLI commands, here is an example of checking the routing table
[guestshell@guestshell ~]$ dohost "show ip route" IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string> 2.2.0.1/32, ubest/mbest: 1/0 *via 10.11.22.22, Eth1/4, [110/65536], 00:24:46, ospf-1, inter 2.2.1.1/32, ubest/mbest: 1/0 *via 10.11.22.22, Eth1/4, [110/65536], 00:24:46, ospf-1, inter 2.2.2.1/32, ubest/mbest: 1/0 *via 10.11.22.22, Eth1/4, [110/65536], 00:24:46, ospf-1, inter 2.2.3.1/32, ubest/mbest: 1/0 *via 10.11.22.22, Eth1/4, [110/65536], 00:24:46, ospf-1, inter 2.2.4.1/32, ubest/mbest: 1/0 *via 10.11.22.22, Eth1/4, [110/65536], 00:24:46, ospf-1, inter 10.1.2.0/24, ubest/mbest: 1/0, attached *via 10.1.2.1, Vlan12, [0/0], 00:28:39, direct 10.1.2.1/32, ubest/mbest: 1/0, attached *via 10.1.2.1, Vlan12, [0/0], 00:28:39, local 10.11.22.0/24, ubest/mbest: 1/0, attached *via 10.11.22.11, Eth1/4, [0/0], 00:27:17, direct 10.11.22.11/32, ubest/mbest: 1/0, attached *via 10.11.22.11, Eth1/4, [0/0], 00:27:17, local
Of course entering a linux shell to check the routing table isn’t terribly useful, however we can use the bash shell to say…ping the OSPF routes like so:
[guestshell@guestshell ~]$ for i in `dohost "show ip route" | grep -B 1 ospf | grep -v via | \
egrep -o '([0-9]+\.){3}[0-9]+'`; do ping $i -c 3; done
PING 2.2.0.1 (2.2.0.1) 56(84) bytes of data.
64 bytes from 2.2.0.1: icmp_seq=1 ttl=255 time=0.608 ms
64 bytes from 2.2.0.1: icmp_seq=2 ttl=255 time=3.06 ms
64 bytes from 2.2.0.1: icmp_seq=3 ttl=255 time=1.01 ms
--- 2.2.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.608/1.561/3.063/1.075 ms
PING 2.2.1.1 (2.2.1.1) 56(84) bytes of data.
64 bytes from 2.2.1.1: icmp_seq=1 ttl=255 time=0.974 ms
64 bytes from 2.2.1.1: icmp_seq=2 ttl=255 time=0.737 ms
64 bytes from 2.2.1.1: icmp_seq=3 ttl=255 time=0.713 ms
--- 2.2.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.713/0.808/0.974/0.117 ms
PING 2.2.2.1 (2.2.2.1) 56(84) bytes of data.
64 bytes from 2.2.2.1: icmp_seq=1 ttl=255 time=2.23 ms
64 bytes from 2.2.2.1: icmp_seq=2 ttl=255 time=0.683 ms
64 bytes from 2.2.2.1: icmp_seq=3 ttl=255 time=0.741 ms
--- 2.2.2.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.683/1.221/2.239/0.720 ms
PING 2.2.3.1 (2.2.3.1) 56(84) bytes of data.
64 bytes from 2.2.3.1: icmp_seq=1 ttl=255 time=0.503 ms
64 bytes from 2.2.3.1: icmp_seq=2 ttl=255 time=0.759 ms
64 bytes from 2.2.3.1: icmp_seq=3 ttl=255 time=0.761 ms
--- 2.2.3.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.503/0.674/0.761/0.122 ms
PING 2.2.4.1 (2.2.4.1) 56(84) bytes of data.
64 bytes from 2.2.4.1: icmp_seq=1 ttl=255 time=1.86 ms
64 bytes from 2.2.4.1: icmp_seq=2 ttl=255 time=0.770 ms
64 bytes from 2.2.4.1: icmp_seq=3 ttl=255 time=0.804 ms
--- 2.2.4.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.770/1.145/1.863/0.508 ms
Hey! Great post. I was trying something along these lines and wanted to see if you can help me proceed:
When I use tar and create the OVA (using -cvf) it creates the OVA successfully but when I try and copy over the file to my flash on an ISR G3, it somehow seems to extract every file (withing the OVA individually) and then copy it, something like below:
ISR4451-1#copy tftp: flash:
Address or name of remote host [x.x.x.x]?
Source filename [Sonar1.ova]?
Destination filename [Sonar1.ova]?
Accessing tftp://x.x.x.x/Sonar1.ova…
Loading Sonar1.ova from x.x.x.x(via GigabitEthernet0/0/0): !
Expanding bootflash:/Sonar1.ova._Sonar.qcow2 (223 bytes):
Expanding bootflash:/Sonar1.ovanar.qcow2 (430309376 bytes): !!!!!!
This is inturn creating an issue for me becasue I do not have a single OVA file to use with the install command. Even if I try, it gives me an error.
Any ideas? I can see that for all of the other OVAs that I can get a hold of (like the demo ones on Cisco’s website) this is not the case, the OVA doesn’t break into multiple components while I copy.
Hmm maybe try SCP to copy the file instead? Or maybe create new directory in flash and copy to there.
Hello! Found this very useful and opened up a few ideas on which I’ve been working on.
I have recently downloaded a demo OVA from Cisco’s devenet site. This is a Fedora VM with a few net perf tools (pretty similar to what you have running on your CSR). I have been able to install and activate it successfully. However, I’m unable to establish connectivity between the router and the KVM – more specifically I’m unable to ping the vnic gateway (virtual port group) from Fedora and the other way around too.
I was able to get this working on an ISR 4K without any issues. The exact same config on CSR doesn’t seem to work and my pings to the vnic gateway are failing.
I have been advised that this could be due to possible interface driver compatibility issue between what CSR supports and what Fedora is using (possibly e1000). Not sure if this stems from the qemu that’s been used to create the OVA. Could you give us the link from where you downloaded the Fedora qemu file and we can package the OVA to check if that fixes our issue?
Hello,
Still waiting on your reply 🙂
To add a few things here. If you downloaded the Fedora from the fedoraproject.org website, we did the same and downloaded the 663 MB Fedora Atomic qcow2 file. But when we tried to launch it, it didn’t seem to work. It runs into Kernel panic
Followed your document to try to set the root password, but we get the error:
“virt-sysprep: error: no operating systems were found in the guest image”
We were trying to look for other Fedora qcow images but failed to get any.
It would be great if you could assist us with the right qcow2 image
Thanks
I used the cloud image in my post.