Management VRF (multiple routing tables and forwarding) provides routing separation between the out-of-band management network and the in-band data plane network. When management VRF is enabled, applications running on control plane processor communicate out from the management network unless configured otherwise.

Management VRF creates two routing tables within the Linux kernel:

  • main: This is the routing table for all the data plane switch ports. 
  • mgmt: This is the routing table for eth0. 

Cumulus Linux only supports eth0 as the management interface. VLAN subinterfaces, bonds, bridges and the front panel switch ports are not supported as management interfaces.

Management VRF assumes all traffic generated by the switch (except via Quagga) will exit eth0 by default, so unless there is application-level intervention, any packet generated by an application on the switch will only reference the eth0 routing table (the mgmt table). Applications that need to communicate over the data plane network (the main table) must bind to the loopback IP address.

For example, if the switch is responding to an inbound SSH connection or inbound ping, management VRF does not force the traffic out through eth0. However, if you attempt to SSH from the switch outbound, then management VRF will force the traffic to exit eth0, unless you specify otherwise. For example, when initiating an SSH connection, you can use -b <loopback IP address> to SSH to a device via the data plane network.

Contents

 Click here to expand...

Enabling Management VRF

To enable management VRF, complete the following steps:

  1. Update the apt source list:
    $ sudo apt-get update
    
  2. Install the management VRF package:
    sudo apt-get install cl-mgmtvrf
    
  3. Run the management VRF script:
    sudo cl-mgmtvrf --enable
    

    Management VRF has hooks in the eth0 DHCP client to force the correct mgmt table routes when the DHCP address is obtained. If you use static IP address assignment on eth0, you have to manually configure the routes before you execute this step. See the 'Using Static IP Addresses on eth0' section below for more information.

  4. Restart Quagga:

    sudo service quagga restart
        

    You can also bounce adjacency to the peer advertising the default route to get the default route from the data plane network into the main routing table.

Verifying Management VRF

To check the status of management VRF, run:

cl-mgmtvrf --status

This will display cl-mgmtvrf is NOT enabled or cl-mgmtvrf is enabled, depending upon whether management VRF is disabled or enabled.

Disabling Management VRF

To disable managment VRF, run:

sudo cl-mgmtvrf --disable
If management VRF is disabled and the data plane adds a default route, the default route via the management interface will not be added to main routing table.

Using ping or traceroute

By default, issuing a ping or traceroute assumes the packet should be sent to the dataplane network (the main routing table). If you wish to use ping or traceroute on the control plane network, use the -I flag for ping and -i for traceroute.

ping -I eth0

or

sudo traceroute -i eth0
DNS does not work with traceroute or ping unless you explicitly add support for the DNS server in the main routing table.

OSPF and BGP

In general, no changes are required for either BGP or OSPF. Quagga was updated in Cumulus Linux 2.5.3 to be aware of the management VRF and automatically sends packets based on the switch port routing table. This includes BGP peering via loopback interfaces. BGP does routing lookups in the default table. However, one modification you may consider has to do with how your routes get redistributed.

Redistributing Routes in Management VRF

The control that management VRF has over the local routing table does not extend to how the routes are redistributed when using routing protocols such as OSPF and BGP.

To redistribute the routes in your network, use the redistribute connected command under BGP or OSPF. This enables the directly connected network out of eth0 to be advertised to its neighbor.

This also creates a route on the neighbor device to the management network through the data plane, which may not be desired.

Cumulus Networks recommends you always use route maps to control the advertised networks redistributed by the redistribute connected command. For example, you can specify a route map to redistribute routes in this way (for both BGP and OSPF):

<routing protocol> 
redistribute connected route-map redistribute-connected

route-map redistribute-connected deny 100
 match interface eth0
!
route-map redistribute-connected permit 1000

SNMP Traps Use eth0 Only

SNMP cannot currently use a switch port to send data. For any SNMP traps, this traffic gets sent out to eth0. Cumulus Networks will support switch ports in the future.

This restriction only applies to traps; SNMP polling is not affected.

SSH

If you SSH to the switch through a switch port, it works as expected. If you need to SSH from the device out a switch port, use ssh -b <ip_address_of_swp_port>. For example:

cumulus@leaf1$ ip addr show swp17
19: swp17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 500
    link/ether ec:f4:bb:fc:19:23 brd ff:ff:ff:ff:ff:ff
    inet 10.23.23.2/24 scope global swp17
    inet6 fe80::eef4:bbff:fefc:1923/64 scope link
       valid_lft forever preferred_lft forever

cumulus@leaf1$ ssh -b 10.23.23.2 10.3.3.3

Viewing the Routing Tables

When you look at the routing table with ip route show, you are looking at the switch port (main) table. You can also see the dataplane routing table with ip route show table main.

To look at information about eth0 (the management routing table), use ip route show table mgmt.

cumulus@leaf1$ ip route show table mgmt
default via 192.168.0.1 dev eth0

cumulus@leaf1$ ip route show
default via 10.23.23.3 dev swp17  proto zebra  metric 20
10.3.3.3 via 10.23.23.3 dev swp17
10.23.23.0/24 dev swp17  proto kernel  scope link  src 10.23.23.2
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.11

Viewing a Single Route

Note that if you use ip route get to return information about a single route, the command resolves over the mgmt table by default. To get information about the route in the switching silicon, use: 

ip route get <addr> from <loopback IP> 

Or:

sudo cl-rctl ip route show <addr> 

Using Static IP Addresses on eth0

If you're using DHCP on your management network, the Management VRF feature has hooks in the eth0 DHCP client to automatically add the correct default and interface routes into the mgmt table when the DHCP address is obtained.

If a static IP address is used in the eth0 definition, you must manually control the connected and static routes attached to eth0 before running cl-mgmtvrf --enable

To do this change the configuration in the /etc/network/interfaces as follows:

auto eth0
iface eth0 inet static
  address 192.1.1.254/24
  post-up ip route add 192.1.1.0/24 dev eth0 table mgmt
  post-up ip route add default via 192.1.1.1 dev eth0 table mgmt
  post-up ip route del 192.1.1.0/24 dev eth0 table main
  post-down ip route del 192.1.1.0/24 dev eth0 table mgmt
  post-down ip route del default via 192.1.1.1 dev eth0 table mgmt

 

Then bounce eth0:

sudo ifdown eth0; sudo ifup eth0

Enabling management VRF via cl-mgmtvrf --enable after this step should lead to the expected routing behavior. 

The post-down commands are there to ensure that no routing race condition can occur on an interface experiencing route flapping. As a result, the following error messages during a link flap are harmless and can be ignored:

warning: eth0: post-down cmd 'ip route del 192.1.1.0/24 dev eth0 table mgmt' failed (RTNETLINK answers: No such process)
warning: eth0: post-down cmd 'ip route del default 192.1.1.1 via eth0 table mgmt' failed (Error: either "to" is duplicate, or "192.1.1.1" is a garbage.)

Incompatibility with cl-ns-mgmt

If you are using the Cumulus Linux management namespace feature (via the cl-ns-mgmt utility), you cannot enable management VRF, as the two features are incompatible. Management VRF does not run if Cumulus Linux detects that you have management namespaces enabled, and vice versa.

Log Files

  • /var/log/cl-mgmtvrf.log

Caveats and Errata

  • Unlike earlier versions of Cumulus Linux, with management VRF enabled, sFlow now sends and receives packets through switch ports as well as the management port, as long as hsflowd has a route available out of the main table. To enable this, contact the Cumulus Networks support team.
  • If you are using an MLAG configuration when the eth0 management interface is enabled, you cannot specify a backup link (via clagd-backup-ip) over the switch ports.
  • Duplicate IP addresses are not supported i.e. you cannot have the same IP address in both the management network and the data network.
  • DHCP relay does not work with cl-mgmtvrf when the DHCP servers are on the management network. For more information, refer to the knowledge base article.
  • Cumulus Linux supports both DHCP and static DNS entries over management VRF through IP FIB rules. These rules are added to direct lookups to the DNS addresses out of the management VRF. However, nameservers configured through DHCP are automatically updated, while statically configured nameservers (configured in /etc/resolv.conf ) only get updated when you run  ifreload -a.

    Because DNS lookups are forced out of the management interface using FIB rules, this could affect data plane ports if there are overlapping addresses. 

4 Comments

  1. Anonymous

    Hi

    I have a problem

    I created a vrf but ssh client or telnet not working.

    where is the problem.

     

    [~]$ ip -V
    ip utility, iproute2-ss151103
    [~]$ uname -a
    Linux master 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
    [~]$
    
    
    
    
    [~]$ ip link show type vrf
    12: test10: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 9e:64:a7:19:32:ad brd ff:ff:ff:ff:ff:ff
    
    
    
    [~]$ ip link show master test10
    5: vboxnet0: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast master test10 state DOWN mode DEFAULT group default qlen 1000
    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff
    6: vboxnet1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master test10 state UP mode DEFAULT group default qlen 1000
    link/ether 0a:00:27:00:00:01 brd ff:ff:ff:ff:ff:ff
    7: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master test10 state UNKNOWN mode DEFAULT group default qlen 100
    link/none
    
    
    
    [~]$ ip addr show master test10
    5: vboxnet0: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast master test10 state DOWN group default qlen 1000
    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 169.254.1.2/16 brd 169.254.255.255 scope global vboxnet0
    valid_lft forever preferred_lft forever
    inet6 fe80::800:27ff:fe00:0/64 scope link
    valid_lft forever preferred_lft forever
    6: vboxnet1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master test10 state UP group default qlen 1000
    link/ether 0a:00:27:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.10/24 brd 10.1.1.255 scope global vboxnet1
    valid_lft forever preferred_lft forever
    inet6 fe80::800:27ff:fe00:1/64 scope link
    valid_lft forever preferred_lft forever
    7: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master test10 state UNKNOWN group default qlen 100
    link/none
    inet 10.8.3.3/24 brd 10.8.3.255 scope global tun0
    valid_lft forever preferred_lft forever
    
    
    
    [~]$ ping -I test10 10.1.1.100
    ping: Warning: source address might be selected on device other than test10.
    PING 10.1.1.100 (10.1.1.100) from 10.1.1.10 test10: 56(84) bytes of data.
    64 bytes from 10.1.1.100: icmp_seq=1 ttl=64 time=0.249 ms
    64 bytes from 10.1.1.100: icmp_seq=2 ttl=64 time=0.235 ms
    64 bytes from 10.1.1.100: icmp_seq=3 ttl=64 time=0.221 ms
    64 bytes from 10.1.1.100: icmp_seq=4 ttl=64 time=0.245 ms
    ^C
    --- 10.1.1.100 ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3000ms
    rtt min/avg/max/mdev = 0.221/0.237/0.249/0.018 ms
    
    
    
    
    
    [~]$ ssh -b 10.1.1.10 10.1.1.100
    bind: 10.1.1.10: Cannot assign requested address
    ssh: connect to host 10.1.1.100 port 22: Cannot assign requested address
    [~]$

     

     

    1. Have you tried asking about this issue on our community? Someone there might be able to help you. 

  2. Agree with Pete. This is off topic for this docs page. If you send an email to netdev@vger.kernel.org I can respond.

  3. Anonymous

    Thanks. 

    I joined the community.