Cumulus Linux seamlessly integrates with the MidoNet OpenStack infrastructure, where the switches provide the VTEP gateway for terminating VXLAN tunnels from within MidoNet. MidoNet connects to the OVSDB server running on the Cumulus Linux switch, and exchanges information about the VTEPs and MAC addresses associated with the OpenStack Neutron networks. This provides seamless Ethernet connectivity between virtual and physical server infrastructures.
Make sure you have a layer 2 gateway; a Tomahawk, Trident II+ or Trident II switch running Cumulus Linux. Cumulus Linux includes OVSDB server (
ovsdb-server) and VTEPd (
ovs-vtepd), which support VLAN-aware bridges.
To integrate a VXLAN with MidoNet, you need to:
- Configure the MidoNet integration on the swtich
- Configure the MidoNet VTEP and port bindings
- Verify the VXLAN configuration
For more information about MidoNet, see the MidoNet Operations Guide, version 1.8 or later.
Configuring the MidoNet Integration on the Switch
Before you start to configure the MidoNet tunnel zones and VTEP binding, and connect virtual ports to the VXLAN, you need to enable and start the
openvswitch-vtep service, and configure the MidoNet integration on the switch. This creates the VTEP gateway and initializes the OVS database server.
Start the openvswitch-vtep Service
To enable and start the
openvswitch-vtep service, run the following command:
In previous versions of Cumulus Linux, you had to edit the
/etc/default/openvswitch-vtep file and then start the
openvswitch-vtep service. Now, you just have to enable and start the
Configure the MidoNet Integration Using the Configuration Script
vtep-bootstrap script is available so you can perform the configuration automatically. For information, read
man vtep-bootstrap. This script requires three parameters, in this order:
- The switch name (the name of the switch that is the VTEP gateway).
- The tunnel IP address (the datapath IP address of the VTEP).
- The management IP address (the IP address of the management interface on the switch).
Because MidoNet does not have a controller, you need to use a dummy IP address (for example, 18.104.22.168) for the controller parameter in the script. After the script completes, delete the VTEP manager, as it is not needed and will otherwise fill the logs with inconsequential error messages:
Configure the MidoNet Integration Manually
If you do not use the configuration script, you must initialize the OVS database instance manually and create the VTEP.
Perform the following commands in order (see the automated script example above for values):
Define the switch in OVSDB:
Define the VTEP tunnel IP address:
Define the management interface IP address:
Restart the OVSDB server and
The switch is now ready to connect to MidoNet. The rest of the configuration is performed from the MidoNet Manager GUI or using the MidoNet API.
Configuring MidoNet VTEP and Port Bindings
This part of the configuration sets up MidoNet and OpenStack to connect the virtualization environment to the Cumulus Linux switch. The
midonet-agent is the networking component that manages the VXLAN, while the Open Virtual Switch (OVS) client on the OpenStack controller node communicates MAC address information between the
midonet-agent and the Cumulus Linux OVS database (OVSDB) server.
You can configure the MidoNet VTEP and port bindings from the MidoNet Manager GUI or the MidoNet CLI.
From the MidoNet Manager GUI
Create a Tunnel Zone
- Click Tunnel Zones in the menu on the left side.
- Click Add.
- Give the tunnel zone a Name and select VTEP for the Type.
- Click Save.
Add Hosts to a Tunnel Zone
After you create the tunnel zone, click the name of the tunnel zone to view the hosts table.
The tunnel zone is a construct used to define the VXLAN source address used for the tunnel. The address of this host is used for the source of the VXLAN encapsulation and traffic transits into the routing domain from this point. Therefore, the host must have layer 3 reachability to the Cumulus Linux switch tunnel IP.
Next, add a host entry to the tunnel zone:
- Click Add.
- Select a host from the Host list.
- Provide the tunnel source IP Address to use on the selected host.
- Click Save.
The host list now displays the new entry:
Create the VTEP
- Click the Vteps menu on the left side.
- Click Add.
Fill out the fields using the same information you used earlier on the switch for the bootstrap procedure:
- Management IP is typically the eth0 address of the switch. This tells the OVS-client to connect to the OVSDB-server on the Cumulus Linux switch.
- Management Port Number is the PTCP port you configured in the
ovs-ctl-vtepscript earlier (the example uses 6632).
- Tunnel Zone is the name of the zone you created in the previous procedure.
- Click Save.
The new VTEP appears in the list below. MidoNet then initiates a connection between the OpenStack Controller and the Cumulus Linux switch. If the OVS client successfully connects to the OVSDB server, the VTEP entry displays the switch name and VXLAN tunnel IP address, which you specified during the bootstrapping process.
Bind Ports to the VTEP
Now that connectivity is established to the switch, you need to add a physical port binding to the VTEP on the Cumulus Linux switch:
- Click Add.
- In the Port Name list, select the port on the Cumulus Linux switch that you are using to connect to the VXLAN segment.
- Specify the VLAN ID (enter 0 for untagged).
- In the Bridge list, select the MidoNet bridge that the instances (VMs) are using in OpenStack.
- Click Save.
You see the port binding displayed in the binding table under the VTEP.
After the port is bound, this automatically configures a VXLAN bridge interface, and includes the VTEP interface and the port bound to the bridge. Now the OpenStack instances (VMs) are able to ping the hosts connected to the bound port on the Cumulus switch. The Troubleshooting section below demonstrates the verification of the VXLAN data and control planes.
From the MidoNet CLI
To get started with the MidoNet CLI, you can access the CLI prompt on the OpenStack Controller:
From the MidoNet CLI, the commands explained in this section perform the same operations depicted in the previous section with the MidoNet Manager GUI.
Create a tunnel zone with a name and type vtep:
- The tunnel zone is a construct used to define the VXLAN source address used for the tunnel. The address of this host is used for the source of the VXLAN encapsulation and traffic transits into the routing domain from this point. Therefore, the host must have layer 3 reachability to the Cumulus Linux switch tunnel IP.
- First, obtain the list of available hosts connected to the Neutron network and the MidoNet bridge.
- Next, get a listing of all the interfaces.
Finally, add a host entry to the tunnel zone ID returned in the previous step and specify which interface address to use.
Create a VTEP and assign it to the tunnel zone ID returned in the previous step. The management IP address (the destination address for the VXLAN or remote VTEP) and the port must be the same ones you configure in the
vtep-bootstrapscript or the manual bootstrapping:
In this step, MidoNet initiates a connection between the OpenStack Controller and the Cumulus Linux switch. If the OVS client successfully connects to the OVSDB server, the returned values should show the name and description matching the
switch-nameparameter specified in the bootstrap process.
Verify the connection-state as CONNECTED. If ERROR is returned, you must debug. Typically this only fails if the
management-ipand or the
management-portsettings are incorrect.
- The VTEP binding uses the information provided to MidoNet from the OVSDB server, providing a list of ports that the hardware VTEP can use for layer 2 attachment. This binding virtually connects the physical interface to the overlay switch, and joins it to the Neutron bridged network.
First, get the UUID of the Neutron network behind the MidoNet bridge:
Next, create the VTEP binding using the UUID and the switch port being bound to the VTEP on the remote end. If there is no VLAN ID, set
At this point, the VTEP is connected and the layer 2 overlay is operational. From the openstack instance (VM), you can ping a physical server connected to the port bound to the hardware switch VTEP.
Troubleshooting MidoNet and Cumulus VTEPs
As with any complex system, there is a control plane and data plane.
Troubleshooting the Control Plane
In this solution, the control plane consists of the connection between the OpenStack Controller and each Cumulus Linux switch running the
Verifying VTEP and OVSDB Services
First, it is important that the OVSDB server and
ovs-vtep daemon are running. Verify this is the case:
Verifying OVSDB-server Connections
From the OpenStack Controller host, verify that it can connect to the
ovsdb-server. Telnet to the switch IP address on port 6632:
If the connection fails, verify IP reachability from the host to the switch. If that succeeds, it is likely that the bootstrap process did not set up port 6632. Redo the bootstrapping procedures above.
Verifying the VXLAN Bridge and VTEP Interfaces
After creating the VTEP in MidoNet and adding an interface binding, you see the br-vxln and vxln interfaces on the switch. Verify that the VXLAN bridge and VTEP interface are created and UP:
Next, look at the bridging table for the VTEP and the forwarding entries. The bound interface and the VTEP are listed along with the MAC addresses of those interfaces. When the hosts attached to the bound port send data, those MACs are learned and entered into the bridging table, as well as the OVSDB.
If you have verified the control plane is correct, and you still cannot get data between the OpenStack instances and the physical nodes on the switch, there might be something wrong with the data plane. The data plane consists of the actual VXLAN encapsulated path, between one of the OpenStack nodes running the
midolman service. This is typically the compute nodes, but can include the MidoNet gateway nodes. If the OpenStack instances can ping the tenant router address but cannot ping the physical device connected to the switch (or vice versa), then something is wrong in the data plane.
Verifying IP Reachability
First, there must be IP reachability between the encapsulating node, and the address you bootstrapped as the tunnel IP on the switch. Verify the OpenStack host can ping the tunnel IP. If this does not work, check the routing design and fix the layer 3 problem first.
MidoNet VXLAN Encapsulation
If the instance (VM) cannot ping the physical server or the reply is not returning, look at the packets on the OpenStack node. Initiate a ping from the OpenStack instance, then use
tcpdump to see the VXLAN data. This example displays a successful tcpdump.
Inspecting the OVSDB
These commands show you the information installed in the OVSDB. This database is structured using the physical switch ID, with one or more logical switch IDs associated with it. The bootstrap process creates the physical switch and MidoNet creates the logical switch after the control session is established.
Listing the Physical Switch
Listing the Logical Switch
Listing Local or Remote MAC Addresses
These commands show the MAC addresses learned from the connected port bound to the logical switch or the MAC addresses advertised from MidoNet. The unknown-dst entries are installed to satisfy the ethernet flooding of unknown unicast and are important for learning.
Getting Open Vswitch Database (OVSDB) Data
ovsdb-client dump command is large but shows all of the information and tables used in communication between the OVS client and server.