LNV active-active mode allows a pair of MLAG switches to act as a single VTEP, providing active-active VXLAN termination for bare metal as well as virtualized workloads.
Terminology and Definitions
|vxrd||VXLAN registration daemon. Runs on the switch that is mapping VLANs to VXLANs. The vxrd daemon needs to be configured to register to a service node. This turns the switch into a VTEP.|
|VTEP||Virtual tunnel endpoint. This is an encapsulation and decapsulation point for VXLANs.|
|active-active VTEP||A pair of switches acting as a single VTEP.|
|ToR||Top of rack switch. Also referred to as a leaf or access switch.|
|Spine||The aggregation switch for multiple leafs. Specifically used when a data center is using a Clos network architecture. Read more about spine-leaf architecture in this white paper.|
|vxsnd||VXLAN service node daemon, that can be run to register multiple VTEPs.|
|exit leaf||A switch dedicated to peering the Clos network to an outside network. Also referred to as border leafs, service leafs or edge leafs.|
|anycast||When an IP address is advertised from multiple locations. Allows multiple devices to share the same IP and effectively load balance traffic across them. With LNV, anycast is used in 2 places:|
|ASIC||Application-specific integrated circuit. Also referred to as hardware, or hardware accelerated. Encapsulation and decapsulation are required for the best performance VXLAN-supported ASIC.|
|RIOT||Broadcom feature for Routing in and out of tunnels. Allows a VXLAN bridge to have a switch VLAN interface associated with it, and traffic to exit a VXLAN into the layer 3 fabric. Also called VXLAN Routing.|
|VXLAN Routing||Industry standard term for ability to route in and out of a VXLAN. Equivalent to Broadcom RIOT feature.|
Configuring LNV Active-Active Mode
LNV requires the following underlying technologies to work correctly.
|MLAG||Refer to the MLAG chapter for more detailed configuration information. Configurations for the demonstration are provided below.|
|OSPF or BGP||Refer to the OSPF chapter or the BGP Chapter for more detailed configuration information. Configurations for the demonstration are provided below.|
|LNV||Refer to the LNV chapter for more detailed configuration information. Configurations for the demonstration are provided below.|
|STP||BPDU filter and BPDU guard should be enabled in the VXLAN interfaces if STP is enabled in the bridge that is connected to the VXLAN.|
Configurations for the demonstration are provided below.
active-active VTEP Anycast IP Behavior
Each individual switch within an MLAG pair should be provisioned with a virtual IP address in the form of an anycast IP address for VXLAN data-path termination. The VXLAN termination address is an anycast IP address that you configure as a
clagd parameter (
clagd-vxlan-anycast-ip) under the loopback interface.
clagd dynamically adds and removes this address as the loopback interface address as follows:
MLAG peering takes place, and a successful VXLAN interface consistency check between the switches occurs.
Failure Scenario Behaviors
|The peer link goes down.|
The primary MLAG switch continues to keep all VXLAN interfaces up with the anycast IP address while the secondary switch brings down all VXLAN interfaces and places them in a PROTO_DOWN state. The secondary MLAG switch removes the anycast IP address from the loopback interface and changes the local IP address of the VXLAN interface to the configured unique IP address.
|One of the switches goes down.|
The other operational switch continues to use the anycast IP address.
All VXLAN interfaces are put in a PROTO_DOWN state. The anycast IP address is removed from the loopback interface and the local IP addresses of the VXLAN interfaces are changed from the anycast IP address to unique non-virtual IP addresses.
|MLAG peering could not be established between the switches.|
|When the peer link goes down but the peer switch is up (i.e. the backup link is active).||All VXLAN interfaces are put into a PROTO_DOWN state on the secondary switch.|
|A configuration mismatch between the MLAG switches||The VXLAN interface is placed into a PROTO_DOWN state on the secondary switch.|
Checking VXLAN Interface Configuration Consistency
The LNV active-active configuration for a given VXLAN interface has to be consistent between the MLAG switches for correct traffic behavior. MLAG ensures that the configuration consistency is met before bringing up the VXLAN interfaces.
The consistency checks include:
- The anycast virtual IP address for VXLAN termination must be the same on each pair of switches.
- A VXLAN interface with the same VXLAN ID must be configured and administratively up on both switches.
You can use the
clagctl command to check if any VXLAN switches are in a PROTO_DOWN state.
Configuring the Anycast IP Address
With MLAG peering, both switches use an anycast IP address for VXLAN encapsulation and decapsulation. This allows remote VTEPs to learn the host MAC addresses attached to the MLAG switches against one logical VTEP, even though the switches independently encapsulate and decapsulate layer 2 traffic originating from the host. The anycast address under the loopback interface can be configured as shown below.
Explanation of Variables
|The unique IP address for the |
|The service node anycast IP address in the topology. In this demonstration, this is an anycast IP address being shared by both spine switches.|
|The anycast address for the MLAG pair to share and bind to when MLAG is up and running.|
Example VXLAN Active-Active Configuration
Note the configuration of the local IP address in the VXLAN interfaces below. They are configured with individual IP addresses, which
clagd changes to anycast upon MLAG peering.
Layer 3 IP Addressing
The IP address configuration for this example:
FRRouting OSPF Configuration
The service nodes and registration nodes must all be routable between each other. The L3 fabric on Cumulus Linux can either be BGP or OSPF. In this example, OSPF is used to demonstrate full reachability.
The FRRouting configuration using OSPF:
In this example, the servers are running Ubuntu 14.04. A layer2 bond must be mapped from server01 and server03 to the respective switch. In Ubuntu this is done with subinterfaces.
Enable the Registration Daemon
The registration daemon (
vxrd) must be enabled on each ToR switch acting as a VTEP, that is participating in LNV. The daemon is installed by default.
- Open the
/etc/default/vxrdconfiguration file in a text editor.
Enable the daemon, then save the file.
Configuring a VTEP
The registration node was configured earlier in
/etc/network/interfaces; no additional configuration is typically needed. Alternatively, the configuration can be done in
/etc/vxrd.conf, which has additional configuration knobs available.
Enable the Service Node Daemon
- Open the
/etc/default/vxsndconfiguration file in a text editor.
Enable the daemon, then save the file:
Restart the daemon.
Configuring the Service Node
To configure the service node daemon, edit the
/etc/vxsnd.conf configuration file:
Full configuration of vxsnd.conf
Full configuration of vxsnd.conf
Considerations for Virtual Topologies Using Cumulus VX
vxrd requires a unique
node_id for each individual switch. This
node_id is based off of the first interface's MAC address; when using certain virtual topologies like Vagrant, both leaf switches within an MLAG pair can generate the same exact unique
node_id. One of the
node_ids must then be configured manually (or make sure the first interface always has a unique MAC address), as they are not unique.
To verify the
node_id that gets configured by your switch, use the
vxrdctl get config command:
To set the
/etc/vxrd.confin a text editor.
node_idvalue within the
commonsection, then save the file:
Ensure that each leaf has a separate node_id so that LNV can function correctly.
Bonds with Vagrant
Bonds (or LACP Etherchannels) fail to work in a Vagrant setup unless the link is set to promiscuous mode. This is a limitation on virtual topologies only, and is not needed on real hardware.
For more information on using Cumulus VX and Vagrant, refer to the Cumulus VX documentation.
Troubleshooting with LNV Active-Active
In addition to the troubleshooting for single-attached LNV, there is now the MLAG daemon (clagd) to consider. The
clagctl command gives the output of MLAG behavior and any inconsistencies that may arise between a MLAG pair.
The additions to normal MLAG behavior are the following:
|VXLAN Anycast IP: 10.10.10.30||The anycast IP address being shared by the MLAG pair for VTEP termination is in use and is 10.10.10.30.|
|There are no conflicts for this MLAG Interface.|
Proto-Down Reason: -
|The VXLAN is up and running (there is no Proto-Down).|
In the next example the
vxlan-id on VXLAN10 was switched to the wrong
vxlan-id. When the
clagctl command is run, you will see that VXLAN10 went down because this switch was the secondary switch and the peer switch took control of VXLAN. The reason code is
vxlan-single indicating that there is a
vxlan-id mis-match on VXLAN10
Caveats and Errata
- The VLAN used for the peer link layer 3 subinterface should not be reused for any other interface in the system. A high VLAN ID value is recommended. For more information on VLAN ID ranges, refer to the section above.
- Active-active mode only works with LNV in this release. Integration with controller-based VXLANs such as VMware NSX and Midokura MidoNet will be supported in the future.
- Integrating with VMware NSX
- Integrating with VMware NSX-V
- Integrating Hardware VTEPs with Midokura MidoNet and OpenStack
- Lightweight Network Virtualization - LNV Overview
- Static MAC Bindings with VXLAN
- Ethernet Virtual Private Network - EVPN
- VXLAN Routing
- VXLAN Scale
- VXLAN Hyperloop
- Static VXLAN Tunnels
- Hybrid Cloud Connectivity with QinQ and VXLANs