Integrating Hardware VTEPs with VMware NSX-V
Switches running Cumulus Linux can integrate with VMware NSX-V to act as hardware VTEP gateways. The VMware NSX-V controller provides consistent provisioning across virtual and physical server infrastructures.

Cumulus Linux also supports integration with VMware NSX in high availability mode. Refer to OVSDB Server High Availability.
Get Started
Before you integrate VXLANs with NSX-V, make sure you have a layer 2 gateway; a switch with Broadcom Tomahawk, Trident II+, Trident II, or Trident III, Maverick, or Mellanox Spectrum or Spectrum A1 running Cumulus Linux. Cumulus Linux includes OVSDB server (ovsdb-server
) and VTEPd (ovs-vtepd
), which support VLAN-aware bridges.
To integrate a VXLAN with NSX-V, you need to:
- Configure the NSX-V integration on the switch.
- Configure the transport and logical layers from the NSX Manager.
- Verify the VXLAN configuration.
Cumulus Linux supports security protocol version TLSv1.2 for SSL connections between the OVSDB server and the NSX controller.
The OVSDB server cannot select the loopback interface as the source IP address, causing top of rack registration to the controller to fail. To work around this issue, run the net add bgp redistribute connected
command followed by the net commit
command.
Configure the Switch for NSX-V Integration
Before you start configuring the gateway service, and logical switches and ports that comprise the VXLAN, you need to enable and start the openvswitch-vtep
service, and configure the NSX integration on the switch, either using the script or performing the manual configuration.
Start the openvswitch-vtep Service
To enable and start the openvswitch-vtep
service, run the following command:
cumulus@switch:~$ sudo systemctl enable openvswitch-vtep.service
cumulus@switch:~$ sudo systemctl start openvswitch-vtep.service
In previous versions of Cumulus Linux, you had to edit the /etc/default/openvswitch-vtep
file and then start the openvswitch-vtep
service . Now, you just have to enable and start the openvswitch-vtep
service .
Configure the NSX-V Integration Using the Configuration Script
A script is available so you can configure the NSX-V integration on the switch automatically.
In a terminal session connected to the switch, run the vtep-bootstrap
command with these options:
controller_ip
is the IP address of the NSX controller (192.168.100.17 in the example command below).- The ID for the VTEP (
vtep7
in the example command below). - The datapath IP address of the VTEP (
172.16.20.157
in the example command below). This is the VXLAN anycast IP address. - The IP address of the management interface on the switch (
192.168.100.157
in the example command below). This interface is used for control traffic.
cumulus@switch:~$ vtep-bootstrap vtep7 --controller_ip 192.168.100.17 172.16.20.157 192.168.100.157
Executed:
create certificate on a switch, to be used for authentication with controller
().
Executed:
sign certificate
(vtep7-req.pem Tue Sep 11 21:11:27 UTC 2018
fingerprint a4cda030fe5e458c0d7ba44e22f52650f01bcd75).
Executed:
define physical switch
().
Executed:
define NSX controller IP address in OVSDB
().
Executed:
define local tunnel IP address on the switch
().
Executed:
define management IP address on the switch
().
Executed:
restart a service
().
Run the following commands in the order shown to complete the configuration process:
cumulus@switch:~$ sudo systemctl restart openvswitch-vtep.service
cumulus@switch:~$ sudo ifreload -a
cumulus@switch:~$ sudo systemctl restart networking.service
Configure the NSX-V Integration Manually
You can configure the NSX-V integration manually for standalone mode only; manual configuration for OVSDB server high availability is not supported.
If you do not want to use the configuration script to configure the NSX-V integration on the switch automatically, you can configure the integration manually, which requires you to perform the following steps:
- Generate a certificate and key pair for authentication by NSX-V.
- Configure a switch as a VTEP gateway.
Generate the Credentials Certificate
In Cumulus Linux, generate a certificate that the NSX controller uses for authentication.
In a terminal session connected to the switch, run the following commands:
cumulus@switch:~$ sudo ovs-pki init Creating controllerca... Creating switchca... cumulus@switch:~$ sudo ovs-pki req+sign cumulus cumulus-req.pem Wed Oct 23 05:32:49 UTC 2013 fingerprint b587c9fe36f09fb371750ab50c430485d33a174a cumulus@switch:~$ ls -l total 12 -rw-r--r-- 1 root root 4028 Oct 23 05:32 cumulus-cert.pem -rw------- 1 root root 1679 Oct 23 05:32 cumulus-privkey.pem -rw-r--r-- 1 root root 3585 Oct 23 05:32 cumulus-req.pem
In the
/usr/share/openvswitch/scripts/ovs-ctl-vtep
file, make sure the lines containing private-key, certificate, and bootstrap-ca-cert point to the correct files; bootstrap-ca-cert is obtained dynamically the first time the switch talks to the controller:# Start ovsdb-server. set ovsdb-server "$DB_FILE" set "$@" -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO set "$@" --remote=punix:"$DB_SOCK" set "$@" --remote=db:Global,managers set "$@" --remote=ptcp:6633:$LOCALIP set "$@" --private-key=/root/cumulus-privkey.pem set "$@" --certificate=/root/cumulus-cert.pem set "$@" --bootstrap-ca-cert=/root/controller.cacert set "$@ " --ssl-protocols=TLSv1,TLSv1.1,TLSv1.2
If files have been moved or regenerated, restart the OVSDB server and VTEPd:
cumulus@switch:~$ sudo systemctl restart openvswitch-vtep.service
Define the NSX Controller Cluster IP address in OVSDB. This causes the OVSDB server to start contacting the NSX controller:
cumulus@switch:~$ sudo vtep-ctl set-manager ssl:192.168.100.17:6632
Define the local IP address on the VTEP for VXLAN tunnel termination. First, find the physical switch name as recorded in OVSDB:
cumulus@switch:~$ sudo vtep-ctl list-ps vtep7
Then set the tunnel source IP address of the VTEP. This is the datapath address of the VTEP, which is typically an address on a loopback interface on the switch that is reachable from the underlying layer 3 network:
cumulus@switch:~$ sudo vtep-ctl set Physical_Switch vtep7 tunnel_ips=172.16.20.157
After you generate the certificate, keep the terminal session active; you need to paste the certificate into NSX Manager when you configure the VTEP gateway.
Enable ovs-vtepd to Use the VLAN-aware Bridge
By default, in stand-alone mode, the ovs-vtep daemon creates traditional bridges for each VXLAN VTEP. To use the VLAN-aware bridge with the VTEPs, edit the /usr/share/openvswitch/scripts/ovs-ctl-vtep
file and uncomment the --enable-vlan-aware-mode
line:
# Start ovs-vtepd
set ovs-vtepd unix:"$DB_SOCK "
set "$@ " -vconsole:emer -vsyslog:err -vfile:info
#set "$@ " --enable-vlan-aware-mode
Then restart the OVSDB server and VTEPd:
cumulus@switch:~$ sudo systemctl restart openvswitch-vtep.service
Provision VMware NSX-V
Configure the Switch as a VTEP Gateway
After you create a certificate, connect to NSX Manager in a browser to configure a Cumulus Linux switch as a hardware VTEP gateway. In this example, the IP address of the NSX Manager is 192.168.110.23.
In NSX Manager, add a new HW VTEP gateway. Click the Network & Security icon, Service Definitions category, then the Hardware Devices tab. Under Hardware Devices, click +. The Create Add Hardware Devices window opens.
In the Name field, provide a name for the HW VTEP gateway.
Enable the BFD service to the service nodes. Select the Enable BFD check box.
From the terminal session connected to the switch where you generated the certificate, copy the certificate and paste it into the Certificate text field. Copy only the bottom portion, including the
BEGIN CERTIFICATE
andEND CERTIFICATE
lines. For example, copy all the highlighted text in the terminal terminal and paste it into NSX Manager:cumulus@switch:~$ cd /var/lib/openvswitch cumulus@switch:/var/lib/openvswitch$ ls conf.db pki vtep7-cert.pem vtep7-privkey.pem vtep7-req.pem cumulus@switch:/var/lib/openvswitch$ cat vtep7-cert.pem
Click OK to save the gateway.
After communication is established between the switch and the controller, a controller.cacert
file is downloaded onto the switch.
Verify that the controller and switch handshake is successful. In a terminal connected to the switch, run this command:
cumulus@switch:~$ sudo ovsdb-client dump -f list | grep -A 7 "Manager"
Manager table
_uuid : 2693ea2e-306-4c23-ac03-934ala304077
inactivity_probe : []
is_connected : true
max_backoff : []
other_config : {}
status : {sec_since_connect="557", state=ACTIVE}
target : "ssl:192.168.110.110:6640"
Configure the Transport and Logical Layers
Configure the Transport Layer
After you finish configuring NSX-V integration on the switch, configure the transport zone and segment ID.
In NSX Manager, click the Logical Network Preparation tab in the Installation category, then click the Segment ID tab.
Click Edit and add the segment IDs (VNIDs) to be used. Here VNIs 5000-5999 are configured.
Click OK to save and provision the segment IDs.
Click the Transport Zones tab, choose the name of the transport zone.
Select Unicast to choose the NSX-V Controller Cluster to handle the VXLAN control plane.
Click OK to save the new transport zone.
Configure the Logical Layer
To complete the integration with NSX-V, you need to configure the logical layer, which requires defining a logical switch (the VXLAN instance) and all the logical ports needed.
To define the logical switch:
In NSX Manager, select the Logical Switches category. Click + to add a logical switch instance.
In the Name field, enter a name for the logical switch.
In the Transport Zone field, add the transport zone that you created earlier.
In the Replication Mode field, select Unicast for replication by the service node. Then check the Enable IP Discovery check box.
Click OK.
To configure the Replication Cluster:
Select the Service Definitions category, then click the Hardware Devices tab. Next to the Replication Cluster field, click Edit.
Hypervisors connected to the NSX controller for replication appear in the Available Objects list. Select the required service nodes, then click the green arrow to move them to the Selected Objects list.
Click OK to save the replication node configuration.
Define Logical Switch Ports
To define the logical switch ports (you can define a VLAN-to-VNI binding for each switch port associated with a particular logical switch):
In NSX Manager, add a new logical switch port. Click the Logical Switches category. Under Actions, click Manage Hardware Bindings. The Manage Hardware Binding wizard appears.
Click + to add a logical port to the logical switch.
Select the logical switch that you created earlier (5000).
Select the switch port and the corresponding VLAN binding for logical switch 5000. This creates the logical switch port and also maps VLAN 16 of switch port swp2 to VNI 5000.
Click OK to save the logical switch port. Connectivity is established. Repeat this procedure for each logical switch port you want to define.
Verify the VXLAN Configuration
After configuration is complete, you can verify the VXLAN configuration using either or both of these Cumulus Linux commands in a terminal connected to the switch:
cumulus@switch:/var/lib/openvswitch$ ip -d link show vxln5000
65: vxln5000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9152 qdisc noqueue master br-vxln5000 state UNKNOWN mode DEFAULT group default
link/ether da:d1:23:44:c4:5e brd ff:ff:ff:ff:ff:ff promiscuity 1
vxlan id 5000 local 172.16.20.157 srcport 0 0 dstport 4789 ageing 300
bridge_slave state forwarding priority 8 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8006 port_no 0x6 designated_port 32774 designated_cost 0 designated_bridge 8000.16:28:56:cc:97:e5 designated_root 8000.16:28:56:cc:97:e5 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on neigh_suppress off addrgenmode eui64
cumulus@switch:/var/lib/openvswitch$ bridge fdb show
b6:fb:be:89:99:65 dev vxln5000 master br-vxln5000 permanent
00:50:56:b5:3f:d2 dev vxln5000 master br-vxln5000 static
00:00:00:00:00:00 dev vxln5000 dst 172.16.1.11 self permanent
00:50:56:b5:3f:d2 dev vxln5000 dst 172.16.1.11 self static
36:cc:7a:bc:b9:e1 dev vxln0 master br-vxln0 permanent
00:23:20:00:00:01 dev dummy0 master br-vxln0 permanent
00:23:20:00:00:01 dev dummy 5000 master br-vxln5000 permanent
7c:fe:90:0b:c5:7e dev swp2.16 master br-vxln5000 permanent
To check that the active OVSDB server is connected to the NSX controller, run the ovsdb-client dump Manager
command:
cumulus@switch:~$ sudo ovsdb-client dump Manager
Manager table
_uuid inactivity_probe is_connected max_backoff other_config status target
------------------------------------ ---------------- ------------ ----------- ------------ -------------------------------------- -------------------
e700ad21-8fd8-4f09-96dc-fa7cc6e498d8 30000 true [] {} {sec_since_connect="68 ", state=ACTIVE} "ssl:54.0.0.2:6632"