The Cumulus Hyperconverged Solution (HCS) in Cumulus Linux supports automated integration with the Nutanix Prism Management solution and the Nutanix AHV hypervisor. Cumulus HCS automatically configures ports attached to Nutanix nodes, provisions networking and manages VLANs with Nutanix Prism and Nutanix AHV.
In addition, you can augment the deployment with:
- Cumulus on a Stick for zero touch provisioning Nutanix and Cumulus HCS without any user interaction or additional equipment.
- Cumulus NetQ for network telemetry and unprecedented real-time and historic visibility into dynamic changes in both the network and virtual machines.
- Out-of-band management and IPMI access using Cumulus RMP or a generic Cumulus Linux switch, enabling the full provisioning of a zero-touch data and management network, eliminating any network deployment delays when standing up a Nutanix cluster.
Cumulus HCS has two major components:
- Nutanix LLDP Switch Agent. When enabled, the agent listens for directly connected Nutanix servers via LLDP and enables MLAG bonding on the relevant ports.
- Nutanix Webhook VLAN Provisioner. Cumulus Linux switches register with the Nutanix CVM and wait to receive Nutanix webhooks. When a new VM is deployed on a server in the cluster, the CVM sends a message to the Cumulus Linux switch with the physical server name and relevant VLANs. The switch then dynamically provisions the configuration on the ports of the specific physical server.
Cumulus HCS periodically polls Nutanix Prism for information about VMs in the cluster. When a new VM is discovered, the service automatically identifies the physical Nutanix server hosting the VM and discovers any VLANs required for the VM. The service then automatically adds these VLANs to the default VLAN-aware bridge, the MLAG peer link and the automatically created bond to the Nutanix node. When a VM is powered off, removed or moved, and the associated VLAN has no other VMs, the VLAN is automatically removed from the bridge, peer link and dynamic bond.
- 2 Cumulus Networks-compatible switches running Cumulus Linux 3.7.3 or later
- Nutanix AOS 5.5.8 or later
- Nutanix AHV 20170830.185 or later
- LLDP enabled on Nutanix (which is the default in 5.5.8 and later)
- IP connectivity between the Cumulus Linux switches and the Nutanix controller VMs (CVMs)
- MLAG enabled on the Cumulus Linux switches
Cumulus HCS runs on any platform. However, this chapter assumes a typical Nutanix deployment with the following configuration:
- Leaf switches with 48 x 10G or 25G ports
- Four or more 40G or 100G uplinks
- Nutanix servers are attached to any of the 10G or 25G ports
- MLAG peer link is on the first two uplink ports: swp49 and swp50
- Connections to other infrastructure are on ports swp51 and above
- The eth0 management interface is configured for management VRF via DHCP
- For automatic configuration, the gateway IP addresses for all VMs, including the CVM, do not exist on the Cumulus Linux switches.
The example configuration utilizes the following topology. All configuration focuses on the leaf01 and leaf02 switches. Configurations for spine01 and spine02 are not included.
Configure Cumulus HCS and Nutanix
The method you choose for configuring Cumulus HCS and Nutanix depends upon whether or not you already have Cumulus Linux installed on your switches, which are named leaf01 and leaf02 in the example configuration above.
- If you have bare-metal switches without Cumulus Linux installed, follow the steps below for configuring a bare-metal switch with ZTP.
- If Cumulus Linux is already installed on your switches, follow the steps below for manually configuring an existing Cumulus Linux switch.
Configure the Service with ZTP
The following steps describe how to use zero touch provisioning to install Cumulus Linux and fully configure Cumulus HCS and Nutanix on your network.
To do this, you need a Cumulus on a Stick disk image and a USB stick with at least 1GB of storage.
- Insert the USB stick into your computer and copy the Cumulus on a Stick files onto it.
On the USB stick, open the
ztp_config.txtfile in a text editor and set your Nutanix username and password and the server IP address, then save and close the file.
- Place the USB stick into the Cumulus Linux switch (leaf01) and power on the switch. Cumulus Linux is automatically installed, including the license and a baseline configuration. The switch reboots multiple times during this process. Depending on your specific hardware platform, this process can take up to 20 minutes. After the installation completes, the LEDs corresponding to the ports connected to the Nutanix nodes illuminate in green.
- When the installation completes, remove the USB stick and repeat this procedure on the other Cumulus Linux switch (leaf02).
Configure the Service Manually
If Cumulus Linux is already installed on your switches, follow the steps below to configure Cumulus Linux, Nutanix and Cumulus HCS.
Configure MLAG on both the leaf01 and leaf02 nodes. The
sys-macis a MAC address from the Cumulus Networks reserved MAC address space and must be the same on both MLAG peers. If you are deploying more than one pair of switches with MLAG, the
sys-macmust be unique for each pair of MLAG-configured switches.
Configure the default layer 2 bridge. Add a unique IP address to each leaf in the same subnet as the CVM.
In both configurations the
pvidvalue of 1 indicates the native VLAN ID. If you don't know the value for the native VLAN ID, use 1.
/etc/default/cumulus-hyperconvergedfile and set the Nutanix username, password and server IP address. Do this on both switches (leaf01 and leaf02). Cumulus Linux uses the settings in this file to authenticate and communicate with the Nutanix cluster.
These settings are defined below.
The server IP address may be a specific Nutanix CVM address or the virtual cluster IP address.
Enable and start Cumulus HCS on leaf01 and leaf02.
Verify that the service is running on leaf01 and leaf02.
If the service fails to start, you may find more information in the service's log file. View the log with
sudo journalctl -u cumulus-hyperconverged.
Enable the server-facing ports to accept inbound LLDP frames and configure jumbo MTU on both leaf01 and leaf02.
At this point, the service is fully configured. It may take up to 60 seconds for LLDP frames to be received to trigger Cumulus HCS.
Cumulus HCS Configuration Settings
Some of the settings you can configure include:
- HOOK_SERVER: the source IP the switch uses when communicating with the Nutanix API. By default, it follows the routing table.
- HOOK_PORT: the port on which the Nutanix CVM is running. The default is 9440.
- SOCKET_TIMEOUT: the amount of time to wait for a timeout when attempting to communicate with the Nutanix API. The default is 10 seconds.
- VXLAN_CONFIG: when set to TRUE, Cumulus HCS automatically provisions VXLAN VNIs as well as VLANs.
- LOGLEVEL: describes the logging level. Verbose and Debug are acceptable values. Verbose provides information about bond and VLAN creation while Debug helps in troubleshooting by providing more information from sources like LLDP and the Nutanix webhook.
- PERIODIC_SYNC_TIMEOUT: how long before Cumulus HCS times out dynamic configurations without contacting the Nutanix API. The default is 60 seconds.
How you configure uplinks depends upon whether you configured Cumulus HCS with ZTP or manually.
If you used ZTP, you can edit the ZTP settings file to define the uplink ports and the VLANs assigned to those uplinks.
If you manually configured the service, you need to enable the uplinks and define the associated VLANs, as shown below. You need to configure both leaf01 and leaf02.
In this example, all VLANs are allowed on the uplink ports. Configuring any set of VLANs is allowed. Be aware that VLANs 3000-3999 are reserved on Cumulus Linux. This example assumes the untagged or native VLAN is VLAN ID (
pvid) 1. Change the VLAN ID as needed.
Add Local Default Gateways
You can add one or more local default gateways on both switches to provide a redundant solution, as shown below. It does not matter whether you configured Cumulus HCS with ZTP or manually. ZTP does not add any gateway configuration.
To provide redundant gateways for the dual-attached Nutanix servers, Cumulus Linux relies on Virtual Router Redundancy (VRR). VRR enables hosts to communicate with any redundant router without reconfiguration, running dynamic routing protocols, or running router redundancy protocols. This means that redundant routers will respond to Address Resolution Protocol (ARP) requests from hosts. Routers are configured to respond in an identical manner, but if one fails, the other redundant routers will continue to respond, leaving the hosts with the impression that nothing has changed.
The first configuration line defines the IP address assigned to each switch, which is required and must be unique. On leaf01, this IP address is 10.1.1.11/24; on leaf02, it is 10.1.1.12/24.
The second line defines the virtual IP address that is used as the default gateway address for any hosts in this VLAN. On both leaf01 and leaf02 this IP address is 10.1.1.1/24. The address-virtual MAC address is assigned from a reserved pool of Cumulus Networks MAC addresses. The address must start with 00:00:05:00:01: and end with any hex value between 00 and ff. Both leaf01 and leaf02 must have the same MAC address. Outside of this switch pair, this MAC address must be unique and only be assigned to a single switch pair in your network.
You can configure out-of-band management in one of two ways:
- Using Cumulus RMP, which is the recommended way.
- Running Cumulus Linux on a supported 1G non-Cumulus RMP switch.
Cumulus RMP is a ready-to-deploy solution that enables out-of-band management for web-scale networks. With Cumulus RMP, you can directly manage and support Nutanix systems in the rack without relying on the rest of the network.
To deploy Nutanix with Cumulus RMP, connect the Nutanix 1G IPMI, 1G Shared IPMI and 1G ports to the Cumulus RMP switch. No additional configuration is required.
Cumulus RMP does not support MLAG or active/active connections across Cumulus RMP switches. Connections across more than one Cumulus RMP switch rely on traditional spanning tree protocol for redundancy.
Other Cumulus Linux 1G Switches
If you want to use a non-Cumulus RMP 1G switch that supports Cumulus Linux for out-of-band management, you must manually install the Cumulus Linux software and license and set up the baseline configuration. The default Cumulus on a Stick image has this information.
Once you install the software, you can use the following command to configure all ports for a single, untagged management VLAN, including any uplinks.
You can assign a management IP address to this same untagged bridge interface. Use an appropriate IP address for your infrastructure.
Apply the configuration:
In both configurations the value of 1 indicates the native or untagged VLAN ID. If you want to use a different VLAN ID, just replace the 1 in both commands with the desired VLAN ID.
Troubleshoot Cumulus HCS
Some ways you can troubleshoot Cumulus HCS include:
- Checking that bonds are being dynamically created.
- Ensuring LLDP messages are being received.
- Verifying the Cumulus HCS configuration.
Verify Dynamic Bonds Are Being Created
net show interface bonds command to verify that bonds are being dynamically created. The following example shows that three bonds, bond_swp1, bond_swp2 and bond_swp3 are created automatically, which means that Cumulus HCS is operating correctly. The name of every dynamically created bond begins with bond_ and ends with the interface name.
Verify LLDP Messages Are Being Received
If bonds are not being created, then LLDP messages may not be getting through. You can check for this possibility using the
net show lldp command:
View Detailed Nutanix LLDP Information
Cumulus HCS replies on the LLDP
SysDescr field to identify a Nutanix host. Run the
net show lldp <swp> command to view the complete LLDP details of the Nutanix node and verify the
- Reloading Cumulus HCS causes the bond interfaces to rebuild. For the stability of the Nutanix cluster, do not reload the service on both leaf switches simultaneously.
- Hyperconverged infrastructure site on the Cumulus Networks website