EVPN requires Cumulus Linux version 3.2.1 or newer.
Only VNI values less than 65535 are supported.
Ethernet Virtual Private Network (EVPN) provides a control plane for VXLANs in Cumulus Linux, with the following functionality:
- VNI membership exchange between VTEPs using the EVPN type-3 (Inclusive Multicast Ethernet Tag) route.
- Exchange of MACs learned on local bridge ports using the EVPN type-2 (MAC/IP Advertisement) route.
- Support for MAC mobility through exchange of the MAC Mobility Extended community.
- Support for dual-attached hosts via VXLAN active-active mode; note that MAC synchronization between the peer switches is done using MLAG.
Installing the EVPN Package
Any existing EVPN configuration on the system should be deleted prior to installing this version of EVPN.
Before proceeding, ensure the version of Cumulus Linux is 3.2.1 or higher.
To install EVPN on a switch:
/etc/apt/sources.listfile in a text editor.
Uncomment the early access repo lines and save the file:
EVPN has been engineered and tested to be production-ready. However, as EVPN was made Generally Available (GA) post Cumulus Linux 3.2.1 release, the packages have been left in the Early Access (EA) repository to create minimal end-user disruption, and to ease deployment and upgrade processes. These packages will be moved to the main repository in an upcoming release, and will ship as part of the base image.
Run the following commands to install the EVPN package in Cumulus Linux:
Check your Quagga version using the
dpkg -l quaggacommand. Please make sure you are running version 1.0.0+cl3eau8 or newer.
Quagga needs to be enabled prior to using EVPN.
/etc/quagga/daemonsfile in a text editor:
Change the no to yes for
bgpd, then save the file:
Enable and start Quagga using the
Enabling EVPN between BGP Neighbors
You enable EVPN between BGP neighbors by adding the address family evpn to the existing neighbor address-family activation command.
For a non-VTEP device, such as a spine switch, that is merely participating in EVPN route exchange (as in, the network deployment uses hop-by-hop eBGP or the switch is acting as an iBGP route reflector), activating the interface for the EVPN address family is the only configuration needed.
This command does not result in BGP knowing about the local VNIs defined on the system and advertising them to peers. This requires additional configuration, as described below.
EVPN supports the configuration of only these two BGP neighbor address-family-specific configurations:
To configure an EVPN route exchange with a BGP peer, the peer or peer-group must be activated within the EVPN address-family in Quagga:
Enabling EVPN in an iBGP Environment with an OSPF Underlay
EVPN can be deployed with an OSPF or static route underlay if needed. This is a more complex configuration than using eBGP. In this case, iBGP advertises EVPN routes directly between VTEPs, and the spines are unaware of EVPN or BGP.
The leaf switches peer with each other in a full mesh within the EVPN address family without using route reflectors. The leafs generally peer to their loopback addresses, which are advertised in OSPF. The receiving VTEP imports routes into a specific VNI with a matching route target community.
Advertising All VNIs
A single configuration variable enables the BGP control plane for all VNIs configured on the switch. Set the variable
advertise-all-vni to provision all locally configured VNIs to be advertised by the BGP control plane. Quagga is not aware of any local VNIs and MACs associated with that VNI until
advertise-all-vni is configured.
This configuration is only needed on leaf switches that are VTEPs.
EVPN routes received from a BGP peer do get accepted, even without
advertise-all-vni configured. These routes are maintained in the global EVPN routing table. However, they only become effective (that is, imported into the per-VNI routing table and appropriate entries installed in the kernel) when the VNI corresponding to the received route is locally known.
Enabling EVPN with Route Distinguishers (RDs) and Route Targets (RTs)
EVPN can be provisioned with RDs and RTs either by automatically configuring them, or by manually defining them. In each case, the
advertise-vni option is used on each VTEP (as shown in the example configuration section described later in this chapter).
VNIs can be configured in Quagga either prior to or after enabling EVPN to advertise all VNIs (through
advertise-all-vni, as described above). When a local VNI is learned and there is no explicit configuration for that VNI, the route distinguisher (RD) and import and export route targets (RTs) for this VNI are automatically derived — the RD uses “RouterId:VNI” and both RTs use “AS:VNI”.
For eBGP EVPN peering, since the peers are in a different AS, using an automatic RT of "AS:VNI" does not work for route import. Thus, the import RT is actually treated as "*:VNI" for determining which received routes are applicable to a particular VNI. This only applies when the import RT is auto-derived and not configured.
Enabling EVPN with Automatic RDs and RTs
advertise-all-vni option is sufficient for provisioning EVPN with RDs and RTs:
Enabling EVPN with User-defined RDs and RTs for Some VNIs
To manually define RDs and RTs, use the
vni option to configure the switch:
These commands are per VNI and must be specified under
address-family evpn in BGP.
Disabling Data Plane MAC Learning over VXLAN Tunnels
When EVPN is provisioned, data plane MAC learning should be disabled for VxLAN interfaces to avoid race conditions between control plane learning and data plane learning. In the
/etc/network/interfaces file, configure the
bridge-learning value to off:
Handling BUM Traffic
EVPN and VXLAN Active-Active Mode
There is no specific configuration to enable EVPN to work with VXLAN active-active mode. Both switches in the MLAG pair establish EVPN peering with other EVPN speakers (for example, with spine switches, if using hop-by-hop eBGP) and inform about their locally known VNIs and MACs. When MLAG is active, both switches announce this information with the shared anycast IP address.
MLAG syncs information between the two switches in the MLAG pair, EVPN does not synchronize. Therefore, Cumulus Networks recommends you do not configure EVPN peering between the MLAG switches over the peerlink.
The following configurations are used throughout this chapter. You can find the flat-file configurations for the network devices in the Cumulus Networks GitHub repository. Only a subset is shown here for brevity (not shown are configurations for leaf03, leaf04, server03, server04). Here is the topology diagram:
leaf01 and leaf02 Configurations
spine01 and spine02 Configurations
server01 and server02 Configurations
Testing Connectivity between Servers
SSH to server01 and ping the VLAN1 IP address on server02:
The following table lists all the servers IP addresses to test connectivity across the L3 fabric:
Cumulus Linux Output Commands
You can use various
iproute2 commands to examine links, VLAN mappings and displaying bridge FDB data:
- ip [-d] link show
- bridge link show
- bridge vlan show
- bridge [-s] fdb show
For example, the output from the following
bridge fdb show command reveals information relevant only for a VTEP.
- 3 remote VTEPs (10.0.0.5, 10.0.0.6 and 22.214.171.124) for each of the 2 VNIs.
- MAC address 00:02:00:00:00:03 is a local MAC learned over a bond interface while MAC address 00:02:00:00:00:08 is a remote MAC from VTEP 126.96.36.199 for VNI 10100 (vtep100).
- The entries with MAC “00:00:00:00:00:00” are for BUM traffic replication.
BGP Output Commands
The following commands are not unique to EVPN but help troubleshoot connectivity and route propagation. You can display the L3 fabric by running the Quagga
show ip bgp summary command on one of the spines:
You can see the loopback addresses for all the network devices participating in BGP by running the
show ip bgp command:
EVPN Output Commands
The following commands are unique to EVPN address-families and VXLAN. Note that just because two network nodes are BGP peers does not mean they are EVPN address-family peers or are exchanging VXLAN information.
Displaying EVPN address-family Peers
show bgp evpn summarycommand
You cannot configure EVPN address families within a VRF.
You can display the configured VNIs on a network device participating in BGP EVPN by running the
show bgp evpn vni command. This command works only when run on a VTEP.
The following example examines leaf01, where 3 VNIs are configured; VNIs 10100 and 10200 are purely system-defined in
/etc/network/interfaces. The RD and RT values for these VNIs have been automatically derived as described earlier. The * indicates that both VNIs exist in the kernel, whereas VNI 20100 is user-configured in BGP with a non-default RD and export RT, but does not yet exist in the kernel:
The corresponding BGP configuration for VNI 20100 is follows (only the EVPN section is shown):
Displaying EVPN VXLANs
show evpn vni [<vni>] command to list all local configured VXLANs and remote VTEPs. This command only works when run on a VTEP.
The following output indicates that VNI 10100 is present on 3 remote VTEPs (10.0.0.5, 10.0.0.6 and 188.8.131.52) while VNI 10200 is present on 2 remote VTEPs (10.0.0.6 and 184.108.40.206):
To see the EVPN configuration for a single VXLAN:
Examining Local and Remote MAC Addresses for a VNI in Quagga
You can examine all local and remote MAC addresses for a VNI by running
show evpn mac vni <vni>.
You can examine MAC addresses across VNIs using
show evpn mac vni all:
You can examine MAC addresses for a remote VTEP and/or query a specific MAC address. This command only works when run on a VTEP:
Displaying the Global BGP EVPN Routing Table
show bgp evpn route [type <multicast | macip>] command to display all EVPN routes at the same time:
*> :::[10.0.0.14]is explained as follows:
Output Explanation  Type 3 EVPN route  Ethernet tag  IP address length of 32 bits 10.0.0.14 IPv4 address originating this route
Displaying EVPN Type-2 (MAC/IP) Routes
To display only EVPN type-2 (MAC/IP) routes, run
show bgp evpn route type macip. The output displays the EVPN route-type fields followed by type-specific fields:
- Type 2 route: [type]:[ESI]:[ET]:[MAC length]:[MAC]
- Type 2 route with ARP suppression: [type]:[ESI]:[ET]:[MAC length]:[MAC]:[IP length]:[IP]
- The Ethernet Segment Id (ESI) and Ethernet Tag (ET) are always 0.
- Type 3 route: [type]:[ET]:[Originating Router IP]
- The Ethernet Tag (ET) is always 0.
- The "Originating Router IP" is the VTEP local IP for the corresponding VNI.
Displaying a Specific EVPN Route
To drill down on a specific route for more information, run the
show bgp evpn route rd <VTEP:VXLAN> command. The following example shows leaf01 receiving a type-2 route and a type-3 route from two spine switches (s1 and s2). The actual remote VTEP is 220.127.116.11, specified in the next hop of the route. Both routes contain the BGP Encapsulation extended community (ET) with value 8 (VxLAN); the type-2 route also carries the VNI (10100).
- Though the local VNI is included in the type-2 route, the receiver does not use it. It uses the received RT to match the route to an appropriate local VNI and then assumes the remote VTEP uses the same VNI value — that is, global VNIs are in use.
- If MAC mobility extended community is exchanged, it gets shown in the above output.
- If the remote MAC is dual attached, the next hop for the EVPN route is the anycast IP address of the remote MLAG pair (when MLAG is active). You can see this in the above example, where 18.104.22.168 is actually the anycast IP of the MLAG pair.
Displaying the per-VNI EVPN Routing Table
Received EVPN routes are maintained in the global EVPN routing table (described above), even if there are no appropriate local VNIs to import them into. For example, a spine switch maintains the global EVPN routing table even though there are no VNIs present on it. When local VNIs are present, received EVPN routes are imported into the per-VNI routing tables based on the route target attributes. The per-VNI routing table can be examined using
show bgp evpn route vni <vni> [type <multicast | macip>]:
Displaying a Specific MAC or Remote VTEP
You can examine a specific MAC or IP (remote VTEP):
To display the VNI routing table for all VNIs, run
show bgp evpn route vni all:
The primary way to troubleshoot EVPN is by enabling Quagga debug logs. The relevant debug options are:
debug zebra vxlan— which traces VNI addition and deletion (local and remote) as well as MAC and neighbor addition and deletion (local and remote).
debug zebra kernel— which traces actual netlink messages exchanged with the kernel, which includes everything, not just EVPN.
debug bgp updates— which traces BGP update exchanges, including all updates. Output is extended to show EVPN specific information.
debug bgp zebra- which traces interactions between BGP and zebra for EVPN (and other) routes.
The following caveat applies to EVPN in Cumulus Linux 3.2.x:
Only VNI values less than 65535 are supported.
- EVPN in Cumulus Linux does not support the ability to only announce certain VNIs. It supports either all locally defined VNIs (using the configuration specified here) or none of them. Provisioning options to advertise only specific VNIs will be introduced in a future release.
- Only one import or export RT is allowed for a VNI. Support for additional RTs (for import) will be introduced in a future release.
- Support for one shot configuration of import and export RTs, using
route-target both <rt>will be added in a future release.