This chapter covers configuring VXLAN gateways using a loopback cable (which we call a hyperloop) on non-RIOT (VXLAN routing) capable ASICs running Cumulus Linux.
The Broadcom Trident II and Tomahawk ASICs have a limitation where a layer 2 bridge that contains a VXLAN interface can not also have an IP address assigned to it. This is an expected limitation with this ASIC, because of the ordering of the decapsulation. A packet that is decapsulated will already have passed the portion of the ASIC capable of reading the IP address lookup (for example, VXLAN lookup happens before IP address lookup). Please contact your sales team if there is any confusion. Refer to the Cumulus Networks Hardware Compatibility List to determine which ASIC is running on the switch.
- VXLAN hyperloop only works on an ASIC capable of encapsulating and decapsulating VXLAN traffic, which includes:
- Broadcom Tomahawk
- Broadcom Trident II
- Broadcom Trident II+
- Mellanox Spectrum
- VXLAN hyperloop is supported on Cumulus Linux 3.2.1 and later. Make sure to upgrade to the latest version of Cumulus Linux.
If you are using EVPN, you must be running FRRouting version eau8. Use a
dpkg -lto check the FRRouting version:
If you are not running the right version of FRRouting for EVPN, follow these directions to upgrade.
Hyperloop Use Cases
Without native VXLAN routing support, external gateways, firewalls or routers are attached to a VTEP do the routing, as in the diagram below.
It is very common in network virtualization environments for firewalls to sit on internal VXLAN-tied VLANs as well as external VLANs that are routed out to the internet. Consider the following illustration. No special configuration is needed on the switch because the firewall acts as the gateway between the internal VLAN (VLAN20/VNI-20) and the routable external VLAN 10. VLAN 10 could have an SVI (switch virtual interface) configured to route out of the VLAN. This also has the benefit in cases where a VXLAN represents a tenant (or a purposely separated application) — you want to keep the firewall between VXLANs so that traffic can be filtered and sanitized to the network operator's specification.
With integrated VXLAN routing and bridging using a hyperloop:
- You can avoid having to use external gateways/routers. A hyperloop gives you the ability to do integrated VXLAN routing on a non-RIOT (VXLAN routing) ASIC.
- If applications are hosted on the switch and need layer 3 connectivity, then a hyperloop can be used to provide reachability for this application as well.
You should not use a hyperloop under these circumstances:
- If the external firewall is used for routing and security (as shown above), then there is no need for an external loopback as the firewall takes care of routing across subnets.
- If bandwidth for the traffic to be routed is so large that you cannot provision such high bandwidth using a hyperloop, then you should have dedicated gateways connected to exit leafs instead.
- If north-south routing involves edge router functionality, then that functionality cannot be provided by leaf switches; rather, it requires dedicated edge gateways to achieve the same result, like NAT.
Exiting a VXLAN with a Hyperloop
This limitation means a physical cable must be attached from one port on leaf1 to another port on leaf1. One port is a layer 3 port while the other is a member of the bridge. The native VLAN (VLAN ID 1) must be tagged for all traffic going to the hyperloop ports.
For example, following the configuration above, in order for a layer 3 address to be used as the gateway for vni-10, you could configure the following on exit01:
Packet Flow Diagram
Trident II and Tomahawk switchd Flag
In order for the Broadcom Trident II and Tomahawk ASICs to be able to have a hyperloop work correctly, you must configure the following
switchd flag. This change is not needed on any other hardware ASIC.
switchd for the change to take place:
switchd is a disruptive change and affects data plane network traffic.
hal.bcm.per_vlan_router_mac_lookup = TRUE limits the Trident II switch to a configurable 512 local IP addresses (SVIs and so forth), so you should use this only as a last resort. This is only a limitation on this specific ASIC type.
VXLAN Hyperloop Troubleshooting Matrix
Before you follow these troubleshooting steps, make sure your switch meets the requirements specified above.
Are HER (Head End Replication) entries being programmed into the bridge fdb table?
Check for 00:00:00:00:00:00 entries for each VXLAN using
bridge fdb show:
or use NCLU:
If you are not getting HER entries, there are some steps you can try:
- Make sure you are using LNV OR EVPN. You cannot use both at the same time.
- Make sure you are not trying to use any VNI/VXLAN values over 65535. For example, VXLAN 70000 is not supported in Cumulus Linux.
- Make sure you are not using the reserved VLAN range; by default it is 3000-3999. This range is stored in the
resv_vlan_rangevariable in the
If you are using an MLAG VTEP (dual attached), is it set up correctly?
Check the outputs. Often, when VXLAN is considered to be non-working, it's actually due to an incorrect setup on the server OS, whether it's Ubuntu, Microsoft Windows or RHEL.
Can you ping from host to host on the same VXLAN?
For example, in the following network diagram, can server01 ping to server03 on any of the VLANs (VLAN1, VLAN100, VLAN200)?
If you can't even ping from server to server this is not a VXLAN gateway problem, but a problem with the network itself. This must be fixed prior to making a VXLAN gateway, with or without a hyperloop.
Only proceed past this point if you can get server to server connectivity on the same VXLAN.
Is the SVI on a physical interface or on a traditional bridge?
The SVI (switch virtual interface) IP address for a hyperloop MUST be on a traditional bridge. Please follow the configuration guidelines above.
Is the port plugged in where it is supposed to be plugged in?
net show lldp to see what ports are hooked up:
Notice above that swp53 and swp54 are a loopback cable (hyperloop) where it is connected to itself.
Is the VRR MAC address unique per subnet?
Make sure that VRR is configured correctly and that each MAC address is unique per VLAN.