Management VRF — a subset of VRF (virtual routing tables and forwarding) — provides a separation between the out-of-band management network and the in-band data plane network. For all VRFs, the main routing table is the default table for all of the data plane switch ports. With management VRF, a second table, mgmt, is used for routing through the switch's Ethernet ports. The mgmt name is special cased to identify the management VRF from a data plane VRF. FIB rules are installed for DNS servers since this is the typical deployment case.
Cumulus Linux only supports eth0 as the management interface, or eth1, depending on the switch platform. The Ethernet ports are software-only parts that are not hardware accelerated by
switchd. VLAN subinterfaces, bonds, bridges and the front panel switch ports are not supported as management interfaces.
When management VRF is enabled, logins to the switch are set into the management VRF context. IPv4 and IPv6 networking applications (for example, Ansible, Chef and
apt-get) run by an administrator communicate out the management network by default. This default context does not impact services run through
systemd and the
systemctl command, and does not impact commands examining the state of the switch; for example, using the
ip command to list links, neighbors or routes.
The management VRF configurations in this chapter contain a localhost loopback IP address (127.0.0.1/8). Putting the loopback address in the management VRF's L3 domain prevents issues with applications that expect the loopback IP address to exist in the VRF, such as NTP.
Enabling Management VRF
To enable management VRF on eth0, complete the following steps:
Bringing the Management VRF Up after Downing It with ifdown
If you take down the management VRF using
ifdown, to bring it back up you need to do one of two things:
ifup --with-depends <vrf>
ifreload -a disconnects the session for any interface configured as auto.
Running Services within the Management VRF
It's possible to run a variety of services within the management VRF instead of the default VRF. In most cases, you must stop and disable the instance running in the default VRF before you can start the service in the management VRF. This is because the instance running in the default VRF owns the port across all VRFs. The list of services that must be disabled in the default VRF are:
When you run a service inside the management VRF, that service runs only on eth0; it no longer runs on any switch port. However, you can keep the service running in the default VRF with a wildcard for agentAddress. This enables the service to run on all interfaces no matter which VRF, so you don't have to run a different process for each VRF.
Some applications can work across all VRFs. The kernel provides a sysctl that allows a single instance to accept connections over all VRFs. For TCP, connected sockets are bound to the VRF the first packet was received. This sysctl is enabled for Cumulus Linux.
To enable a service to run in the management VRF, do the following. These steps use the NTP service, but you can use any of the services listed above, except for
dhcrelay (discussed here) and
hsflowd (discussed below).
- Configure the management VRF as described in the Enabling Management VRF section above.
If NTP is running, stop the service:
Disable NTP from automatically starting in the default VRF:
Start NTP in the mgmt VRF.
Enable ntp@mgmt so that it starts when the switch boots:
Once enabled, you can verify that NTP peers are active:
Enabling Polling with snmpd in a Management VRF
When you enable
snmpd to run in the management VRF,
snmpd listens only on eth0; you can no longer listen on a switch port.
However, as mentioned above, you can keep
snmpd running in the default VRF with a wildcard for agentAddress. This enables
snmpd to listen to all interfaces no matter which VRF, so you don't have to run a different
snmpd process for each VRF.
SNMP traps always use eth0 only to send trap-related traffic. SNMP traps cannot use a switch port to send data. Cumulus Networks plans to support switch ports in the future.
If you're using sFlow to monitor traffic in the mgmt VRF, you need to complete the following steps to enable it.
hsflowdprocess to the
systemdconfiguration file in
/etc/vrf/systemd.confin a text editor.
snmpddaemon if it is running:
snmpdto ensure it does not start in the default VRF if the system is rebooted:
snmpdin the the mgmt VRF:
Enable hsflowd@mgmt so it starts when the switch boots:
Verify that the
hsflowdservice is running in the mgmt VRF:
Using ping or traceroute
By default, issuing a
traceroute assumes the packet should be sent to the dataplane network (the main routing table). If you wish to use
traceroute on the management network, use the
-I flag for ping and
OSPF and BGP
In general, no changes are required for either BGP or OSPF. FRRouting was updated in Cumulus Linux 3.0 to be VRF-aware and automatically sends packets based on the switch port routing table. This includes BGP peering via loopback interfaces. BGP does routing lookups in the default table. However, one modification you may consider has to do with how your routes get redistributed.
Redistributing Routes in Management VRF
Management VRF uses the mgmt table, including local routes. It does not affect how the routes are redistributed when using routing protocols such as OSPF and BGP.
To redistribute the routes in your network, use the
redistribute connected command under BGP or OSPF. This enables the directly connected network out of eth0 to be advertised to its neighbor.
This also creates a route on the neighbor device to the management network through the data plane, which may not be desired.
Cumulus Networks recommends you always use route maps to control the advertised networks redistributed by the
redistribute connected command. For example, you can specify a route map to redistribute routes in this way (for both BGP and OSPF):
These commands produce the following configuration snippet in the
Using SSH within a Management VRF Context
If you SSH to the switch through a switch port, it works as expected. If you need to SSH from the device out of a switch port, use
vrf exec default ssh <ip_address_of_swp_port>. For example:
Viewing the Routing Tables
When you look at the routing table with
ip route show, you are looking at the switch port (main) table. You can also see the dataplane routing table with
net show route vrf main.
To look at information about eth0 (the management routing table), use
net show route vrf mgmt.
Viewing a Single Route
Note that if you use
ip route get to return information about a single route, the command resolves over the mgmt table by default. To get information about the route in the switching silicon, use:
To get the route for any VRF, run:
Using the mgmt Interface Class
ifupdown2 interface classes are used to create a user-defined grouping for interfaces. The special class mgmt is available to separate the switch's management interfaces from the data interfaces. This allows you to manage the data interfaces by default using
ifupdown2 commands. Performing operations on the mgmt interfaces requires specifying the
--allow-mgmt option, which prevents inadvertent outages on the management interfaces. Cumulus Linux by default brings up all interfaces in both the auto (default) class and the mgmt interface class when the switch boots.
The management VRF interface class is not supported if you are configuring Cumulus Linux using NCLU.
You configure the management interface in
/etc/network/interfaces. In the example below, the management interface, eth0, and the mgmt VRF stanzas are added to the mgmt interface class:
When you run
ifupdown2 commands against the interfaces in the mgmt class, include
--allow=mgmt with the commands. For example, to see which interfaces are in the mgmt interface class, run:
To reload the configurations for interfaces in the mgmt class, run:
However, you can still bring the management interface up and down using
ifup eth0 and
Management VRF and DNS
Cumulus Linux supports both DHCP and static DNS entries over management VRF through IP FIB rules. These rules are added to direct lookups to the DNS addresses out of the management VRF.
In order for DNS to use the management VRF, the static DNS entries must reference the management VRF in the
/etc/resolv.conf file. For example:
Note that nameservers configured through DHCP are automatically updated, while statically configured nameservers (configured in
/etc/resolv.conf) only get updated when you run
Because DNS lookups are forced out of the management interface using FIB rules, this could affect data plane ports if there are overlapping addresses.
Incompatibility with cl-ns-mgmt
Management VRF has replaced the management namespace functionality in Cumulus Linux. The management namespace feature (via the
cl-ns-mgmt utility) has been deprecated, and the
cl-ns-mgmt command has been removed.