Switches running Cumulus Linux can integrate with VMware NSX Multi-Hypervisor (MH) to act as hardware VTEP gateways. The VMware NSX-MH controller provides consistent provisioning across virtual and physical server infrastructures.
Cumulus Linux also supports integration with VMware NSX in high availability mode. Refer to OVSDB Server High Availability.
Before you integrate VXLANs with NSX-MH, make sure you have a layer 2 gateway; a Broadcom Tomahawk, Trident II+, Trident II, Maverick, or Mellanox Spectrum switch running Cumulus Linux. Cumulus Linux includes OVSDB server (
ovsdb-server) and VTEPd (
ovs-vtepd), which support VLAN-aware bridges.
To integrate a VXLAN with NSX-MH, you need to:
- Configure the NSX-MH integration on the switch.
- Configure the transport and logical layers from the NSX Manager.
- Verify the VXLAN configuration.
Cumulus Linux supports security protocol version TLSv1.2 for SSL connections between the OVSDB server and the NSX controller.
The OVSDB server cannot select the loopback interface as the source IP address, causing top of rack registration to the controller to fail. To work around this issue, run the
net add bgp redistribute connected command followed by the
net commit command.
Configure the Switch for NSX-MH Integration
Before you start configuring the gateway service, logical switches, and ports that comprise the VXLAN, you need to enable and start the
openvswitch-vtep service, and configure the NSX integration on the switch, either using the script or performing the manual configuration.
Start the openvswitch-vtep Service
To enable and start the
openvswitch-vtep service, run the following command:
In previous versions of Cumulus Linux, you had to edit the
/etc/default/openvswitch-vtep file and then start the
openvswitch-vtep service. Now, you just have to enable and start the
Configure the NSX-MH Integration Using the Configuration Script
A script is available so you can configure the NSX-MH integration on the switch automatically.
In a terminal session connected to the switch, run the
vtep-bootstrap command with these options:
controller_ipis the IP address of the NSX controller (192.168.100.17 in the example command below).
The ID for the VTEP (
vtep7in the example command below).
The datapath IP address of the VTEP (
172.16.20.157in the example command below). This is the VXLAN anycast IP address.
The IP address of the management interface on the switch (
192.168.100.157in the example command below). This interface is used for control traffic.
Run the following commands in the order shown to complete the configuration process:
Configure the NSX-MH Integration Manually
You can configure the NSX-V integration manually for standalone mode only; manual configuration for OVSDB server high availability is not supported.
If you do not want to use the configuration script to configure the NSX-MH integration on the switch automatically, you can configure the integration manually, which requires you to perform the following steps:
- Generate a certificate and key pair for authentication by NSX.
- Configure the switch as a VTEP gateway.
Generate the Credentials Certificate
In Cumulus Linux, generate a certificate that the NSX controller uses for authentication.
In a terminal session connected to the switch, run the following commands:
/usr/share/openvswitch/scripts/ovs-ctl-vtepfile, make sure the lines containing private-key, certificate, and bootstrap-ca-cert point to the correct files; bootstrap-ca-cert is obtained dynamically the first time the switch talks to the controller:
If files have been moved or regenerated, restart the OVSDB server and VTEPd:
Define the NSX controller cluster IP address in OVSDB. This causes the OVSDB server to start contacting the NSX controller:
Define the local IP address on the VTEP for VXLAN tunnel termination. First, find the physical switch name as recorded in OVSDB:
Then set the tunnel source IP address of the VTEP. This is the datapath address of the VTEP, which is typically an address on a loopback interface on the switch that is reachable from the underlying layer 3 network:
After you generate the certificate, keep the terminal session active; you need to paste the certificate into NSX Manager when you configure the VTEP gateway.
Enable ovs-vtepd to Use the VLAN-aware Bridge
By default, in stand-alone mode, the ovs-vtep daemon creates traditional bridges for each VXLAN VTEP. To use the VLAN-aware bridge with the VTEPs, edit the
/usr/share/openvswitch/scripts/ovs-ctl-vtep file and uncomment the
Then restart the OVSDB server and VTEPd:
Provision VMware NSX-V
Configure the Switch as a VTEP Gateway
After you create a certificate, connect to NSX Manager in a browser to configure a Cumulus Linux switch as a VTEP gateway. In this example, the IP address of the NSX Manager is 192.168.100.12.
In NSX Manager, add a new gateway. Click the Network Components tab, then the Transport Layer category. Under Transport Node, click Add, then select Manually Enter All Fields. The Create Gateway wizard opens.
In the Create Gateway dialog, select Gateway for the Transport Node Type, then click Next.
In the Display Name field, provide a name for the gateway, then click Next.
Enable the VTEP service. Select the VTEP Enabled checkbox, then click Next.
From the terminal session connected to the switch where you generated the certificate, copy the certificate and paste it into the Security Certificate text field. Copy only the bottom portion, including the
END CERTIFICATElines. For example, copy all the highlighted text in the terminal:
Paste it into NSX Manager, then click Next:
In the Connectors dialog, click Add Connector to add a transport connector. This defines the tunnel endpoint that terminates the VXLAN tunnel and connects NSX to the physical gateway. You must choose a tunnel Transport Type of VXLAN. Choose an existing transport zone for the connector or click Create to create a new transport zone.
Define the IP address of the connector (the underlay IP address on the switch for tunnel termination).
Click OK to save the connector, then click Save to save the gateway.
After communication is established between the switch and the controller, a
controller.cacert file downloads onto the switch.
Verify that the controller and switch handshake is successful. In a terminal connected to the switch, run this command:
Configure the Transport and Logical Layers
Configure the Transport Layer
After you finish configuring the NSX-MH integration on the switch, configure the transport layer. For each host-facing switch port to be associated with a VXLAN instance, define a Gateway Service for the port.
In NSX Manager, add a new gateway service. Click the Network Components tab, then the Services category. Under Gateway Service, click Add. The Create Gateway Service wizard opens.
In the Create Gateway Service dialog, select VTEP L2 Gateway Service as the Gateway Service Type.
Provide a Display Name for the service to represent the VTEP in NSX.
Click Add Gateway to associate the service with the gateway you created earlier.
In the Transport Node field, choose the name of the gateway you created earlier.
In the Port ID field, choose the physical port on the gateway (for example, swp10) that will connect to a logical layer 2 segment and carry data traffic.
Click OK to save this gateway in the service, then click Save to save the gateway service.
The gateway service shows up as type VTEP L2 in NSX.
Next, configure the logical layer on NSX.
Configure the Logical Layer
To complete the integration with NSX, you need to configure the logical layer, which requires defining a logical switch (the VXLAN instance) and all the logical ports needed.
To define the logical switch:
In NSX Manager, add a new logical switch. Click the Network Components tab, then the Logical Layer category. Under Logical Switch, click Add. The Create Logical Switch wizard opens.
In the Display Name field, enter a name for the logical switch, then click Next.
Under Replication Mode, select Service Nodes, then click Next.
Specify the transport zone bindings for the logical switch. Click Add Binding. The Create Transport Zone Binding dialog opens.
In the Transport Type list, select VXLAN, then click OK to add the binding to the logical switch.
In the VNI field, assign the switch a VNI ID, then click OK.Do not use 0 or 16777215 as the VNI ID; these are reserved values under Cumulus Linux.
Click Save to save the logical switch configuration.
Define Logical Switch Ports
Logical switch ports can be virtual machine VIF interfaces from a registered OVS or a VTEP gateway service instance on this switch, as defined above in the Configuring the Transport Layer. You can define a VLAN binding for each VTEP gateway service associated with the particular logical switch.
To define the logical switch ports:
In NSX Manager, add a new logical switch port. Click the Network Components tab, then the Logical Layer category. Under Logical Switch Port, click Add. The Create Logical Switch Port wizard opens.
In the Logical Switch UUID list, select the logical switch you created above, then click Create.
In the Display Name field, provide a name for the port that indicates it is the port that connects the gateway, then click Next.
In the Attachment Type list, select VTEP L2 Gateway.
In the VTEP L2 Gateway Service UUID list, choose the name of the gateway service you created earlier.
In the VLAN list, you can choose a VLAN if you want to connect only traffic on a specific VLAN of the physical network. Leave it blank to handle all traffic.
Click Save to save the logical switch port. Connectivity is established. Repeat this procedure for each logical switch port you want to define.
Verify the VXLAN Configuration
After configuration is complete, verify the VXLAN configuration in a terminal connected to the switch using these Cumulus Linux commands:
ovsdb-client dump command to troubleshoot issues on the switch. This command verifies that the controller and switch handshake is successful (and works only for VXLANs integrated with NSX):