This documentation is for an older version of the software. If you are using the current version of Cumulus Linux, this content may not be up to date. The current version of the documentation is available here. If you are redirected to the main page of the user guide, then this page may have been renamed; please search for it there.

VXLAN Scale

On Broadcom Trident II and Tomahawk hardware running Cumulus Linux, there is a limit on the amount of VXLANs you can configure simultaneously. The limit most often given is 2000 VXLANs that can be run, but network architects want to get more specific and know exactly the limit for their specific design.

The limit is a Physical to Virtual mappings where we can hold 15,000 mappings in hardware before we hit hash collisions. There is also an upper limit of around 3000 VLANs you can configure before you hit the reserved range (3000-3999 by default). When you hear a Cumulus Linux employee say “around” or use a soft number its because the math is unique per customer environment. A internal VLAN is consumed per L3 port, per subinterface, per traditional bridge and 1 is used for a bridge in VLAN-aware mode. This means we have a configurable:

(Total Configurable 802.1q VLANs) - (Reserved VLANS) - (Physical or Logical Interfaces) =
4094-999-eth0-loopback = 3093 by default (without any other configuration)

The math equation for configurable VXLANs looks like the following:

(Amount of Trunks) * (VXLAN/VLANs per Trunk) - (Linux Logical and Physical Interfaces) = 15,000

For a specific example on a 10Gb switch with 48 * 10 G ports and 6 * 40G uplinks we can calculate for X (amount of configurable VXLANs)

48 * X + (48 downlinks + 6 uplinks + 1 loopback + 1 eth0 + 1 bridge) = 15000
48 * X = 14943
X = 311 VXLANs

Similarly this logic can be applied to a 32 * 100G switch where 16 ports have been broken up to 4 * 25 Gbps ports (64 * 25 Gbps ports)

64 * X + (64 downlinks + 16 uplinks + 1 loopback + 1 eth0 + 1 bridge) = 15000

64 * X = 14917

X = 233 VXLANs

The problem is not all ports are trunks for all VXLANs (or at least not all the time). It is much more common for subsets of ports to be used for different VXLANs.

e.g.
On a 10Gb (48 * 10Gb + 6 * 40Gb uplinks) we have the following configuration:

PortsTrunks
swp1-20100 VXLAN/VLANs
swp21-30100 VXLAN/VLANs
swp31-48X VXLAN/VLANs

The equation now looks like this:

20 swps * 100 VXLANs + 10 swps * 100 VXLANs + 18 swps * X VXLANs + (48 downlinks + 6 uplinks + loopback + 1 eth0 + 1 bridge) = 15000

20 swps * 100 VXLANs + 10 swps * 100 VXLANs + 18 swps * X VXLANs = 14943

18 * X = 11943

663 = VXLANS (still configurable) for a total of 863