Cumulus Linux provides virtual routing and forwarding (VRF) to allow for the presence of multiple independent routing tables working simultaneously on the same router or switch. This permits multiple network paths without the need for multiple switches. Think of this feature as VLAN for layer 3, but unlike VLANs, there is no field in the IP header carrying it. Other implementations call this feature VRF-Lite.
The primary use cases for VRF in a data center are similar to VLANs at layer 2: using common physical infrastructure to carry multiple isolated traffic streams for multi-tenant environments, where these streams are allowed to cross over only at configured boundary points, typically firewalls or IDS. You can also use it to burst traffic from private clouds to enterprise networks where the burst point is at layer 3. Or you can use it in an OpenStack deployment.
VRF is fully supported in the Linux kernel, so it has the following characteristics:
- The VRF is presented as a layer 3 master network device with its own associated routing table.
- The layer 3 interfaces (VLAN interfaces, bonds, switch virtual interfaces/SVIs) associated with the VRF are enslaved to that VRF; IP rules direct FIB (forwarding information base) lookups to the routing table for the VRF device.
- The VRF device can have its own IP address, known as a VRF-local loopback.
- Applications can use existing interfaces to operate in a VRF context — by binding sockets to the VRF device or passing the
cmsg. By default, applications on the switch run against the default VRF. Services started by
systemdrun in the default VRF unless the VRF instance is used. If management VRF is enabled, logins to the switch default to the management VRF. This is a convenience for users to not have to specify management VRF for each command.
- Listen sockets used by services are VRF-global by default unless the application is configured to use a more limited scope — for example, read about services in the management VRF. Connected sockets (like TCP) are then bound to the VRF domain in which the connection originates. The kernel provides a sysctl that allows a single instance to accept connections over all VRFs. For TCP, connected sockets are bound to the VRF the first packet was received. This sysctl is enabled for Cumulus Linux.
- Connected and local routes are placed in appropriate VRF tables.
- Neighbor entries continue to be per-interface, and you can view all entries associated with the VRF device.
- A VRF does not map to its own network namespace; however, you can nest VRFs in a network namespace.
- You can use existing Linux tools to interact with it, such as
Cumulus Linux supports up to 255 VRFs on a switch.
You configure VRF by associating each subset of interfaces to a VRF routing table, and configuring an instance of the routing protocol — BGP or OSPFv2 — for each routing table.
Each routing table is called a VRF table, and has its own table ID. You configure VRF using NCLU, then place the layer 3 interface in the VRF. You can have a maximum of 255 VRFs on a switch.
When you configure a VRF, you follow a similar process to other network interfaces. Keep in mind the following for a VRF table:
- It can have an IP address, a loopback interface for the VRF.
- Associated rules are added automatically.
- You can also add a default route to avoid skipping across tables when the kernel forwards the packet.
Names for VRF tables can be up to 15 characters. However, you cannot use the name mgmt, as this name can only be used for management VRF.
To configure a VRF, run:
These commands result in the following VRF configuration in the
Specify a Table ID
Instead of having Cumulus Linux assign a table ID for the VRF table, you can specify your own table ID in the configuration. The table ID to name mapping is saved in
/etc/iproute2/rt_tables.d/ for name-based references. So instead of using the
auto option above, specify the table ID like this:
If you do specify a table ID, it must be in the range of 1001 to 1255 which is reserved in Cumulus Linux for VRF table IDs.
Bring a VRF Up after Downing It with ifdown
If you take down a VRF using
ifdown, to bring it back up you need to do one of two things:
ifup --with-depends <vrf>
vrf command returns information about VRF tables that is otherwise not available in other Linux commands, such as
iproute. You can also use it to execute non-VRF-specific commands and perform other tasks related to VRF tables.
To get a list of VRF tables, run:
To return a list of processes and PIDs associated with a specific VRF table, run
vrf task list <vrf-name>. For example:
To determine which VRF table is associated with a particular PID, run
vrf task identify <pid>. For example:
IPv4 and IPv6 Commands in a VRF Context
You can execute non-VRF-specific Linux commands and perform other tasks against a given VRF table. This typically applies to single-use commands started from a login shell, as they affect only AF_INET and AF_INET6 sockets opened by the command that gets executed; it has no impact on netlink sockets, associated with the
To execute such a command against a VRF table, run
vrf task exec <vrf-name> <command>. For example, to SSH from the switch to a device accessible through VRF rocket:
You should manage long-running services with
systemd using the service@vrf notation; for example,
systemctl start ntp@mgmt.
systemd-based services are stopped when a VRF is deleted and started when the VRF is created. For example, restarting networking or running an
Services in VRFs
For services that need to run against a specific VRF, Cumulus Linux uses
systemd instances, where the instance is the VRF. In general, you start a service within a VRF like this:
For example, you can run the NTP service in the turtle VRF using:
In most cases, the instance running in the default VRF needs to be stopped before a VRF instance can start. This is because the instance running in the default VRF owns the port across all VRFs — that is, it is VRF global.
systemd-based services are stopped when the VRF is deleted and started when the VRF is created. For example, when you restart networking or run an
ifup sequence — as mentioned above. The management VRF chapter details how to do this.
In Cumulus Linux, the following services work with VRF instances:
There are cases where
systemd instances do not work; you must use a service-specific configuration option instead. For example, you can configure
rsyslogd to send messages to remote systems over a VRF:
VRF Route Leaking
The most common use case for VRF is to use multiple independent routing and forwarding tables; however, there are situations where destinations in one VRF must be reachable (leaked) from another VRF. For example, to make a service (such as a firewall) available to multiple VRFs or to enable routing to external networks (or the Internet) for multiple VRFs, where the external network itself is reachable through a specific VRF.
Cumulus Linux provides two options for route leaking across VRFs: static route leaking and dynamic route leaking.
- An interface is always assigned to only one VRF; any packets received on that interface are routed using the associated VRF routing table.
- Route leaking is typically used for non-overlapping addresses.
- Route leaking is supported for both IPv4 and IPv6 routes.
- Do not mix static and dynamic route leaking in a fabric.
- VRF route leaking is not supported between the tenant VRF and the default VRF with onlink next hops (BGP unnumbered).
Configure Static Route Leaking
For static route leaking, you configure routes manually in a VRF whose next hops are reachable over an interface that is part of another VRF. This is useful where one or more specific destinations in a different VRF need to be reachable from another VRF. You can use static route leaking to reach remote destinations (through a next hop router) or directly-connected destinations in another VRF.
Cumulus Networks recommends that you use dynamic route leaking instead of static route leaking. Dynamic route leaking is easier to configure, easier to maintain (static route leaking configuration requires changes when you want to leak new routes), and supports route maps for better control. See Configure Dynamic Route Leaking.
To configure static route leaking in a non-EVPN configuration, follow the steps below. To configure static route leaking with EVPN, see Configure Static Route Leaking with EVPN.
/etc/cumulus/switchd.conffile, change the
TRUEand uncomment the line. Then, restart
switchdfor the change to take effect.
Only set the
TRUEfor static VRF route leaking. This option must be set to
falsefor dynamic route leaking.
Use the keyword
nexthop-vrfto specify the VRF through which the next hop router is reachable. The example command below adds a static route (10.1.0.0/24) to a VRF named
turtle, which is reachable through a next-hop router (192.168.200.1) over a different VRF,
Configure Static Route Leaking with EVPN
Static route leaking is supported with EVPN symmetric routing only.
You cannot leak the default route.
To configure static route leaking with EVPN symmetric routing:
configure-static-routing above.Enable VRF route leaking, as shown in step 1 of
Configure static route leaking for EVPN. The following commands provide examples.
To configure static route leaking between VRF1 and VRF2, where VRF1 contains subnets 10.50.1.0/24, 10.50.2.0/24, 10.50.3.0/24, and 10.50.4.0/24 and VRF2 contains subnets 10.60.1.0/24, 10.60.2.0/24, 10.60.3.0/24, and 10.60.4.0/24, run these commands:
To configure static route leaking between the default VRF and VRF1, where swp1s0 is the egress port for subnets under 10.10.0.0/16 in the default VRF, run these commands:
Configure Dynamic Route Leaking
For dynamic route leaking, a destination VRF is interested in the routes of a source VRF. As routes come and go in the source VRF, they are dynamically leaked to the destination VRF through BGP. If the routes in the source VRF are learned through BGP, no additional configuration is necessary. If the routes in the source VRF are learned through OSPF, or if they are statically configured or directly-connected networks have to be reached, the routes need to be first redistributed into BGP (in the source VRF) for them to be leaked.
You can also use dynamic route leaking to reach remote destinations as well as directly connected destinations in another VRF. Multiple VRFs can import routes from a single source VRF and a VRF can import routes from multiple source VRFs. This is typically used when a single VRF provides connectivity to external networks or a shared service for many other VRFs.
You can control the routes that are leaked dynamically across VRFs with a route-map.
Because dynamic route leaking happens through BGP, the underlying mechanism relies on the BGP constructs of the Route Distinguisher (RD) and Route Targets (RTs). However, you do not need to configure these parameters; they are automatically derived when you enable route leaking between a pair of VRFs.
- Dynamic route leaking with EVPN is supported in Cumulus Linux 3.7.4 and later.
- You cannot reach the loopback address of a VRF (the address assigned to the VRF device) from another VRF.
- When using dynamic route leaking, you must use the
redistributecommand in BGP to leak non-BGP routes (connected or static routes); you cannot use the
- Routes in the management VRF with the next-hop as eth0 or the management interface are not leaked.
- Routes learned with iBGP or multi-hop eBGP in a VRF can be leaked even if their next hops become unreachable. Therefore, route leaking for BGP-learned routes is recommended only when they are learned through single-hop eBGP.
- Cumulus Networks recommends that you do not use the default VRF as a shared service VRF. Create another VRF for shared services.
- Broadcom switches have certain limitations when leaking routes between the default VRF and non-default VRFs.
- On Mellanox switches, only leak the specific routes you need from the default VRF; do not include the VTEP routes or filter out the VTEP routes with a route filter.
In the following example commands, routes in the BGP routing table of VRF
rocket are dynamically leaked into VRF
The NCLU commands save the configuration in the
/etc/frr/frr.conf file. For example:
Exclude Certain Prefixes
To exclude certain prefixes from being imported, you can use a route map.
The following example configures a route map to match the source protocol BGP and imports the routes from VRF
turtle to VRF
rocket. For the imported routes, the community is set to 11:11 in VRF
Dynamic Route Leaking Between VRFs where Subnets Extend across Racks
When you configure dynamic VRF route leaking to leak routes between VRFs, especially in an EVPN deployment where subnets are extended across racks, be aware of the the following considerations:
- If end systems (hosts or VMs) are present in one VRF at the router where route leaking is configured that need to be reached from another VRF, you must configure the router to announce a subnet route for the corresponding subnet. Either redistribute the connected subnets into BGP or inject the specific subnet into BGP in the source VRF. This step is necessary because VRF route leaking only leaks routes in the BGP VRF routing table from one VRF into another, which includes only subnet routes and remote host routes, but not routes to local hosts.
- If the leaked routes are being aggregated in the target VRF for route announcement, and if the aggregate matches with a subnet being leaked, you need to configure static routes for the local hosts either in the source VRF or in the destination VRF. Without this configuration, when the aggregate becomes active in the target VRF due to the presence of more-specific contributing leaked routes, a blackhole/discard route is installed in the target VRF, and because this matches one of the leaked subnets, connectivity to any local hosts in that subnet is lost from the target VRF.
Verify Dynamic Route Leaking Configuration
To check the status of dynamic VRF route leaking, run the NCLU
net show bgp vrf <vrf-name> ipv4|ipv6 unicast route-leak command. For example:
- To view the BGP routing table, run the NCLU
net show bgp vrf <vrf-name> ipv4|ipv6 unicastcommand.
- To view the FRR IP routing table, run the NCLU
net show route vrf <vrf-name>command. This command shows all routes, including routes leaked from other VRFs.
The following example command shows all routes in VRF
turtle, including routes leaked from VRF
Delete Dynamic Route Leaking Configuration
To remove dynamic route leaking configuration, run the following commands. These commands ensure that all leaked routes are removed and routes are no longer leaked from the specified source VRF.
The following example commands delete leaked routes from VRF
rocket to VRF
Do not use the kernel commands; they are no longer supported and might cause issues when used with VRF route leaking in FRR.
FRRouting Operation in a VRF
In Cumulus Linux 3.5 and later, BGP, OSPFv2 and static routing (IPv4 and IPv6) are supported within a VRF context. Various FRRouting routing constructs, such as routing tables, nexthops, router-id, and related processing are also VRF-aware.
FRRouting learns of VRFs provisioned on the system as well as interface attachment to a VRF through notifications from the kernel.
You can assign switch ports to each VRF table with an interface-level configuration, and BGP instances can be assigned to the table with a BGP router-level command.
Because BGP is VRF-aware, they support per-VRF neighbors, both iBGP and eBGP as well as numbered and unnumbered interfaces. Non-interface-based VRF neighbors are bound to the VRF, which is how you can have overlapping address spaces in different VRFs. Each VRF can have its own parameters, such as address families and redistribution. Incoming connections rely on the Linux kernel for VRF-global sockets. BGP neighbors can be tracked using BFD, both for single and multiple hops. You can configure multiple BGP instances, associating each with a VRF.
A VRF-aware OSPFv2 configuration also supports numbered and unnumbered interfaces. Supported layer 3 interfaces include SVIs, sub-interfaces and physical interfaces. The VRF supports types 1 through 5 (ABR/ASBR – external LSAs) and types 9 through 11 (opaque LSAs) link state advertisements, redistributing other routing protocols, connected and static routes, and route maps. As with BGP, you can track OSPF neighbors with BFD.
Cumulus Linux does not support multiple VRFs in multi-instance OSPF.
VRFs are provisioned using NCLU. VRFs can be pre-provisioned in FRRouting too, but they become active only when configured with NCLU.
- You pre-provision a VRF in FRRouting by running the command
- A BGP instance corresponding to a VRF can be pre-provisioned by configuring
net add bgp vrf <VRF> autonomous-system <ASN>. Under this context, all existing BGP parameters can be configured: neighbors, peer-groups, address-family configuration, redistribution, and so forth.
- An OSPFv2 instance can be configured using the
net add ospf vrf <VRF>command; as with BGP, all OSPFv2 parameters can be configured.
- Static routes (IPv4 and IPv6) can be provisioned in a VRF by specifying the VRF along with the static route configuration. For example,
ip route prefix dev vrf vrf-name. The VRF has to exist for this configuration to be accepted — either already defined through
/etc/network/interfacesor pre-provisioned in FRRouting. If you want to leak a static route in a VRF, see the note above.
Example BGP and OSPF Configurations
Here's an example VRF configuration in BGP:
These commands produce the following configuration in the
Here is the FRRouting OSPF configuration:
These commands create the following configuration in the
Example Commands to Show VRF Data
There are a number of ways to interact with VRFs, including NCLU,
vtysh (the FRRouting CLI) and
Show VRF Data Using NCLU Commands
To show the routes in the VRF:
To show the BGP summary for the VRF:
To show BGP (IPv4) routes in the VRF:
However, to show BGP IPv6 routes in the VRF, you need to use
vtysh, the FRRouting CLI:
To show the OSPF VRFs:
To show all the OSPF routes in a VRF:
net show vrf list command. The following command shows which interfaces are in the VRFs configured on the switch:To show which interfaces are in a VRF (either BGP or OSPF), run the
To show the interfaces for a specific VRF, run the
net show vrf list <vrf_name> command. The following command shows which interfaces are in VRF turtle:
You can only specify one VRF with the
net show vrf list <vrf_name> command. For example,
net show vrf list mgmt turtle is an invalid command.
To show the VNIs for the interfaces in a VRF, run the
net show vrf vni command. For example:
To see the VNIs for the interfaces in a VRF in JSON format, run the
net show vrf vni json command. For example:
Show VRF Data Using FRRouting Commands
Show all VRFs learned by FRRouting from the kernel. The table ID shows the corresponding routing table in the kernel either automatically assigned or manually defined:
Show VRFs configured in BGP, including the default. A non-zero ID is a VRF that has also been actually provisioned — that is, defined in
Display interfaces known to FRRouting and attached to this VRF:
To show VRFs configured in OSPF:
To show all OSPF routes in a VRF:
To see the routing table for each VRF, use the
show up route vrf all command. The OSPF route is denoted in the row that starts with O:
Show VRF Data Using ip Commands
To list all VRFs provisioned, showing the VRF ID (vrf1012, vrf1013 and vrf1014 below) as well as the table ID:
To list the interfaces attached to a specific VRF:
To show IPv4 routes in a VRF:
To show IPv6 routes in a VRF:
To see a list of links associated with a particular VRF table,
run ip link list <vrf-name>. For example:
To see a list of routes associated with a particular VRF table, run
ip route list <vrf-name>. For example:
You can also show routes in a VRF using
ip [-6] route show vrf <name>. This command omits local and broadcast routes, which can clutter the output.
BGP Unnumbered Interfaces with VRF
BGP unnumbered interface configurations are supported with VRF. In BGP unnumbered, there are no addresses on any interface. However, debugging tools like
traceroute need at least a single IP address per node as the node's source IP address. Typically, this address was assigned to the loopback device. With VRF, you need a loopback device for each VRF table since VRF is based on interfaces, not IP addresses. While Linux does not support multiple loopback devices, it does support the concept of a dummy interface, which is used to achieve the same goal.
An IP address can be associated with the VRF device, which will then act as the dummy (loopback-like) interface for that VRF.
Configure the BGP unnumbered configuration. The BGP unnumbered configuration is the same for a non-VRF, applied under the VRF context (
router bgp asn vrf <vrf-name>).
These commands create the following configuration in the
Here is the FRRouting BGP configuration:
These commands create the following configuration in the
DHCP with VRF
Because you can use VRF to bind IPv4 and IPv6 sockets to non-default VRF tables, you have the ability to start DHCP servers and relays in any non-default VRF table using the
dhcrelay services, respectively. These services must be managed by
systemd in order to run in a VRF context; in addition, the services must be listed in
/etc/vrf/systemd.conf. By default, this file already lists these two services, as well as others like
snmpd. You can add more services as needed, such as
dhcrelay6 for IPv6.
If you edit
sudo systemctl daemon-reload to generate the
systemd instance files for the newly added service(s). Then you can start the service in the VRF using
systemctl start <service>@<vrf-name>.service, where
<service> is the name of the service — such as
dhcrelay — and
<vrf-name> is the name of the VRF.
For example, to start the
dhcrelay service after you configured a VRF named turtle, run:
To enable the service at boot time you should also run
systemctl enable <service>@<vrf-name>. To continue with the previous example:
In addition, you need to create a separate default file in
/etc/default for every instance of a DHCP server and/or relay in a non-default VRF; this is where you set the server and relay options. To run multiple instances of any of these services, you need a separate file for each instance. The files must be named as follows:
See the example configuration below for more details.
Caveats for DHCP with VRF
- Cumulus Linux does not support DHCP server and relay across VRFs, so the server and host cannot be in different VRF tables. In addition, the server and relay cannot be in different VRF tables.
- Typically a service running in the default VRF owns a port across all VRFs. If the VRF local instance is preferred, the global one may need to be disabled and stopped first.
- VRF is a layer 3 routing feature. It only makes sense to run programs that use AF_INET and AF_INET6 sockets in a VRF. VRF context does not affect any other aspects of the operation of a program.
- This method only works with
In the following example, there is one IPv4 network with a VRF named rocket and one IPv6 network with a VRF named turtle.
The IPv4 DHCP server/relay network looks like this:
The IPv6 DHCP server/relay network looks like this:
Configure each DHCP server and relay as follows:
Sample DHCP Server Configuration
Sample DHCP6 Server Configuration
Sample DHCP Relay Configuration
Sample DHCP6 Relay Configuration
ping or traceroute on a VRF
If you want to use
traceroute on a VRF, use the
-I <vrf> flag for ping and
-i <vrf> for
Caveats and Errata
- Switches using the Hurricane2 ASIC (such as the Penguin Computing Arctica 4804IP) do not support VRFs.
- Table selection based on the incoming interface only; currently, packet attributes or output-interface-based selection are not available.
- Setting the router ID outside of BGP via the
router-idoption causes all BGP instances to get the same router ID. If you want each BGP instance to have its own router ID, specify the
router-idunder the BGP instance using
bgp router-id. If both are specified, the one under the BGP instance overrides the one provided outside BGP.
- You cannot configure EVPN address families within a VRF.