Cumulus Linux seamlessly integrates with the MidoNet OpenStack infrastructure, where the switches provide the VTEP gateway for terminating VXLAN tunnels from within MidoNet. MidoNet connects to the OVSDB server running on the Cumulus Linux switch, and exchanges information about the VTEPs and MAC addresses associated with the OpenStack Neutron networks. This provides seamless Ethernet connectivity between virtual and physical server infrastructures.
Before you create VXLANs with MidoNet, make sure you have the following components:
- A switch (L2 gateway) with a Tomahawk, Trident II+ or Trident II chipset running Cumulus Linux 2.0 and later
- OVSDB server (
ovsdb-server), included in Cumulus Linux 2.0 and later
- VTEPd (
ovs-vtepd), included in Cumulus Linux 2.0 and later
Integrating a VXLAN with MidoNet involves:
- Preparing for the MidoNet integration
- Bootstrapping the OVS and VTEP
- Configuring the MidoNet VTEP binding
- Verifying the VXLAN configuration
Caveats and Errata
- There is no support for VXLAN routing in the Tomahawk, Trident II+ and Trident II chipsets; use a loopback interface or external router.
- For more information about MidoNet, see the MidoNet Operations Guide, version 1.8 or later.
Preparing for the MidoNet Integration
Before you start configuring the MidoNet tunnel zones, VTEP binding and connecting virtual ports to the VXLAN, you need to complete the bootstrap process on each switch to which you plan to build VXLAN tunnels. This creates the VTEP gateway and initializes the OVS database server. You only need to do the bootstrapping once, before you begin the MidoNet integration.
Enabling the openvswitch-vtep Package
Before you start bootstrapping the integration, you need to enable the
openvswitch-vtep package, since it is disabled by default in Cumulus Linux.
/etc/default/openvswitch-vtepfile, changing the
STARToption from no to yes. This simple
sedcommand does this, and creates a backup as well:
Start the daemon:
Bootstrapping the OVSDB Server and VTEP
Automating with the Bootstrap Script
vtep-bootstrap script is available so you can do the bootstrapping automatically. For information, read
man vtep-bootstrap. This script requires three parameters, in this order:
- Switch name: The name of the switch that is the VTEP gateway.
- Tunnel IP address: The datapath IP address of the VTEP.
- Management IP address: The IP address of the switch's management interface.
Since MidoNet does not have a controller, you need to use a dummy IP address (for example, 220.127.116.11) for the controller parameter in the bootstrap script. After the script completes, delete the VTEP manager, since it is not needed and will otherwise fill the logs with inconsequential error messages:
If you don't use the bootstrap script, then you must initialize the OVS database instance manually, and create the VTEP.
Perform the following commands in order (see the automated bootstrapping example above for values):
Define the switch in OVSDB:
Define the VTEP tunnel IP address:
Define the management interface IP address:
Restart the OVSDB server and
At this point, the switch is ready to connect to MidoNet. The rest of the configuration is performed in the MidoNet Manager GUI, or using the MidoNet API.
Configuring MidoNet VTEP and Port Bindings
This part of the configuration sets up MidoNet and OpenStack to connect the virtualization environment to the Cumulus Linux switch. The
midonet-agent is the networking component that manages the VXLAN, while the Open Virtual Switch (OVS) client on the OpenStack controller node communicates MAC address information between the
midonet-agent and the Cumulus Linux OVS database (OVSDB) server.
Using the MidoNet Manager GUI
Creating a Tunnel Zone
- Click Tunnel Zones in the menu on the left side.
- Click Add.
- Give the tunnel zone a Name and select "VTEP" for the Type.
- Click Save.
Adding Hosts to a Tunnel Zone
Once the tunnel zone is created, click the name of the tunnel zone to view the hosts table.
The tunnel zone is a construct used to define the VXLAN source address used for the tunnel. This host's address is used for the source of the VXLAN encapsulation, and traffic will transit into the routing domain from this point. Thus, the host must have layer 3 reachability to the Cumulus Linux switch tunnel IP.
Next, add a host entry to the tunnel zone:
- Click Add.
- Select a host from the Host list.
- Provide the tunnel source IP Address to use on the selected host.
- Click Save.
The host list now displays the new entry:
Creating the VTEP
- Click the Vteps menu on the left side.
- Click Add.
Fill out the fields using the same information you used earlier on the switch for the bootstrap procedure:
- Management IP is typically the eth0 address of the switch. This tells the OVS-client to connect to the OVSDB-server on the Cumulus Linux switch.
- Management Port Number is the PTCP port you configured in the
ovs-ctl-vtepscript earlier (the example uses 6632).
- Tunnel Zone is the name of the zone you created in the previous procedure.
- Click Save.
The new VTEP appears in the list below. MidoNet then initiates a connection between the OpenStack Controller and the Cumulus Linux switch. If the OVS client is successfully connected to the OVSDB server, the VTEP entry should display the switch name and VXLAN tunnel IP address, which you specified during the bootstrapping process.
Binding Ports to the VTEP
Now that connectivity is established to the switch, you need to add a physical port binding to the VTEP on the Cumulus Linux switch:
- Click Add.
- In the Port Name list, select the port on the Cumulus Linux switch that you are using to connect to the VXLAN segment.
- Specify the VLAN ID (enter 0 for untagged).
- In the Bridge list, select the MidoNet bridge that the instances (VMs) are using in OpenStack.
- Click Save.
You should see the port binding displayed in the binding table under the VTEP.
Once the port is bound, this automatically configures a VXLAN bridge interface, and includes the VTEP interface and the port bound to the bridge. Now the OpenStack instances (VMs) should be able to ping the hosts connected to the bound port on the Cumulus switch. The Troubleshooting section below demonstrates the verification of the VXLAN data and control planes.
Using the MidoNet CLI
To get started with the MidoNet CLI, you can access the CLI prompt on the OpenStack Controller:
Now from the MidoNet CLI, the commands explained in this section perform the same operations depicted in the previous section with the MidoNet Manager GUI.
Create a tunnel zone with a name and type vtep:
- The tunnel zone is a construct used to define the VXLAN source address used for the tunnel. This host's address is used for the source of the VXLAN encapsulation, and traffic will transit into the routing domain from this point. Thus, the host must have layer 3 reachability to the Cumulus Linux switch tunnel IP.
- First, get the list of available hosts connected to the Neutron network and the MidoNet bridge.
- Next, get a listing of all the interfaces.
Finally, add a host entry to the tunnel zone ID returned in the previous step, and specify which interface address to use.
Create a VTEP and assign it to the tunnel zone ID returned in the previous step. The management IP address (the destination address for the VXLAN/remote VTEP) and the port must be the same ones you configured in the
vtep-bootstrapscript or the manual bootstrapping:
In this step, MidoNet initiates a connection between the OpenStack Controller and the Cumulus Linux switch. If the OVS client is successfully connected to the OVSDB server, the returned values should show the name and description matching the
switch-nameparameter specified in the bootstrap process.
Verify the connection-state as CONNECTED, otherwise if ERROR is returned, you must debug. Typically this only fails if the
management-portsettings are wrong.
- The VTEP binding uses the information provided to MidoNet from the OVSDB server, providing a list of ports that the hardware VTEP can use for layer 2 attachment. This binding virtually connects the physical interface to the overlay switch, and joins it to the Neutron bridged network.
First, get the UUID of the Neutron network behind the MidoNet bridge:
Next, create the VTEP binding, using the UUID and the switch port being bound to the VTEP on the remote end. If there is no VLAN ID, set
At this point, the VTEP should be connected, and the layer 2 overlay should be operational. From the openstack instance (VM), you should be able to ping a physical server connected to the port bound to the hardware switch VTEP.
Troubleshooting MidoNet and Cumulus VTEPs
As with any complex system, there is a control plane and data plane.
Troubleshooting the Control Plane
In this solution, the control plane consists of the connection between the OpenStack Controller, and each Cumulus Linux switch running the
Verifying VTEP and OVSDB Services
First, it is important that the OVSDB server and
ovs-vtep daemon are running. Verify this is the case:
Verifying OVSDB-server Connections
From the OpenStack Controller host, verify that it can connect to the
ovsdb-server. Telnet to the switch IP address on port 6632:
If the connection fails, verify IP reachability from the host to the switch. If that succeeds, it is likely the bootstrap process did not set up port 6632. Redo the bootstrapping procedures above.
Verifying the VXLAN Bridge and VTEP Interfaces
After creating the VTEP in MidoNet and adding an interface binding, you should see br-vxln and vxln interfaces on the switch. You can verify that the VXLAN bridge and VTEP interface are created and UP:
Next, look at the bridging table for the VTEP and the forwarding entries. The bound interface and the VTEP should be listed along with the MAC addresses of those interfaces. When the hosts attached to the bound port send data, those MACs are learned, and entered into the bridging table, as well as the OVSDB.
If you have verified the control plane is correct, and you still cannot get data between the OpenStack instances and the physical nodes on the switch, there may be something wrong with the data plane. The data plane consists of the actual VXLAN encapsulated path, between one of the OpenStack nodes running the
midolman service. This is typically the compute nodes, but can include the MidoNet gateway nodes. If the OpenStack instances can ping the tenant router address but cannot ping the physical device connected to the switch (or vice versa), then something is wrong in the data plane.
Verifying IP Reachability
First, there must be IP reachability between the encapsulating node, and the address you bootstrapped as the tunnel IP on the switch. Verify the OpenStack host can ping the tunnel IP. If this doesn't work, check the routing design, and fix the layer 3 problem first.
MidoNet VXLAN Encapsulation
If the instance (VM) cannot ping the physical server, or the reply is not returning, look at the packets on the OpenStack node. Initiate a ping from the OpenStack instance, then using
tcpdump, hopefully you can see the VXLAN data. This example displays what it looks like when it is working.
Inspecting the OVSDB
These commands show you the information installed in the OVSDB. This database is structured using the physical switch ID, with one or more logical switch IDs associated with it. The bootstrap process creates the physical switch, and MidoNet creates the logical switch after the control session is established.
Listing the Physical Switch
Listing the Logical Switch
Listing Local or Remote MAC Addresses
These commands show the MAC addresses learned from the connected port bound to the logical switch, or the MAC addresses advertised from MidoNet. The unknown-dst entries are installed to satisfy the ethernet flooding of unknown unicast, and important for learning.
Getting Open Vswitch Database (OVSDB) Data
ovsdb-client dump command is large, but shows all of the information and tables that are used in communication between the OVS client and server.