Static MAC Bindings with VXLAN

Cumulus Linux includes native Linux VXLAN kernel support.

Requirements

A VXLAN configuration requires a switch with the Broadcom Tomahawk, Trident II+, or Trident II ASIC running Cumulus Linux 2.0 or later, or a switch with the Mellanox Spectrum ASIC running Cumulus Linux 3.2.0 or later.

For a basic VXLAN configuration, make sure that:

  • The VXLAN has a network identifier (VNI); do not use 0 or 16777215 as the VNI ID, which are reserved values under Cumulus Linux.
  • The VXLAN link and local interfaces are added to the bridge to create the association between port, VLAN, and VXLAN instance.

Example VXLAN Configuration

Consider the following example:

https://s3-us-west-2.amazonaws.com/dev.docs.cumulusnetworks.com/images/download/attachments/8362790/vxlan.png

Preconfiguring remote MAC addresses does not scale. A better solution is to use the Cumulus Networks Lightweight Network Virtualization feature, or a controller-based option like Midokura MidoNet and OpenStack or VMware NSX.

Configure the Static MAC Bindings VXLAN

To configure the example illustrated above, first create the following configuration on switch1:

cumulus@switch1:~$ net add loopback lo ip address 172.10.1.1
cumulus@switch1:~$ net add loopback lo vxrd-src-ip 172.10.1.1
cumulus@switch1:~$ net add bridge bridge ports swp1-2
cumulus@switch1:~$ net add bridge post-up bridge fdb add 0:00:10:00:00:0C dev vtep1000 dst 172.20.1.1 vni 1000
cumulus@switch1:~$ net add vxlan vtep1000 vxlan id 1000
cumulus@switch1:~$ net add vxlan vtep1000 vxlan local-tunnelip 172.10.1.1
cumulus@switch1:~$ net add vxlan vtep1000 bridge access 10
cumulus@switch1:~$ net pending
cumulus@switch1:~$ net commit 

These commands create the following configuration in the /etc/network/interfaces file:

auto vtep1000
iface vtep1000
    vxlan-id 1000
    vxlan-local-tunnelip 172.10.1.1
 
auto bridge
iface bridge
    bridge-ports swp1 swp2 vtep1000
    bridge-vids 10
    bridge-vlan-aware yes
    post-up bridge fdb add 0:00:10:00:00:0C dev vtep1000 dst 172.20.1.1 vni 1000

Then create the following configuration on switch2:

cumulus@switch2:~$ net add loopback lo ip address 172.20.1.1
cumulus@switch2:~$ net add loopback lo vxrd-src-ip 172.20.1.1
cumulus@switch1:~$ net add bridge bridge ports swp1-2
cumulus@switch2:~$ net add bridge post-up bridge fdb add 00:00:10:00:00:0A dev vtep1000 dst 172.10.1.1 vni 1000
cumulus@switch2:~$ net add bridge post-up bridge fdb add 00:00:10:00:00:0B dev vtep1000 dst 172.10.1.1 vni 1000
cumulus@switch2:~$ net add vxlan vtep1000 vxlan id 1000
cumulus@switch2:~$ net add vxlan vtep1000 vxlan local-tunnelip 172.10.1.1
cumulus@switch2:~$ net add vxlan vtep1000 bridge access 10
cumulus@switch2:~$ net pending
cumulus@switch2:~$ net commit

These commands create the following configuration in the /etc/network/interfaces file:

auto vtep1000
iface vtep1000
    vxlan-id 1000
    vxlan-local-tunnelip 172.20.1.1
 
auto bridge
iface bridge
    bridge-ports swp1 swp2 vtep1000
    bridge-vlan-aware yes
    post-up bridge fdb add 00:00:10:00:00:0A dev vtep1000 dst 172.10.1.1 vni 1000
    post-up bridge fdb add 00:00:10:00:00:0B dev vtep1000 dst 172.10.1.1 vni 1000

Troubleshooting

Use the following commands to troubleshoot issues on the switch:

  • brctl show verifies the VXLAN configuration in a bridge:

    cumulus@switch:~$ brctl show
    bridge name bridge id           STP enabled   interfaces
    bridge      8000.2a179a8cc471   yes           swp1
                                                  swp2
                                                  vni-10
                                                  vni-2000
    
  • bridge fdb show displays the list of MAC addresses in an FDB:

    cumulus@switch1:~$ bridge fdb show
    44:38:39:00:00:18 dev swp1 master bridge permanent
    44:38:39:00:00:1c dev swp2 master bridge permanent
    2a:17:9a:8c:c4:71 dev vni-2000 master bridge permanent
    9a:e8:ef:a1:9d:6f dev vni-10 master bridge permanent
    00:00:10:00:00:0c dev vni-10 dst 172.20.1.1 self permanent
    
  • ip -d link show displays information about the VXLAN link:

    cumulus@switch1:~$ ip –d link show vni-10
    15: vni-10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master bridge state UNKNOWN mode DEFAULT group default
        link/ether 9a:e8:ef:a1:9d:6f brd ff:ff:ff:ff:ff:ff promiscuity 1
        vxlan id 10 remote 10.2.1.3 local 10.2.1.1 srcport 0 0 dstport 4789 ageing 1800
        bridge_slave state forwarding priority 8 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8004 port_no 0x4 designated_port 32772 designated_cost 0 designated_bridge 8000.2a:17:9a:8c:c4:71 designated_root 8000.2a:17:9a:8c:c4:71 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on addrgenmode eui64