This chapter discusses the various network interfaces on a switch running Cumulus Linux, how to configure various interface-level settings (if needed) and some troubleshooting commands.
Cumulus Linux exposes network interfaces for several types of physical and logical devices:
- lo, network loopback device
- ethN, switch management port(s), for out of band management only
- swpN, switch front panel ports
- (optional) brN, bridges (IEEE 802.1Q VLANs)
- (optional) bondN, bonds (IEEE 802.3ad link aggregation trunks, or port channels)
Each physical network interface has a number of configurable settings:
Almost all of these settings are configured automatically for you, depending upon your switch ASIC, although you must always set MTU manually.
You can only set MTU for logical interfaces. If you try to set auto-negotiation, duplex mode or link speed for a logical interface, an unsupported error gets returned.
Differences between Broadcom-based and Mellanox-based Switches
On a Broadcom-based switch, all you need to do is enable auto-negotiation. Once enabled, Cumulus Linux automatically configures the link speed, duplex mode and forward error correction (FEC, if the cable or optic requires it) for every switch port, based on the port type and cable or optic used on the port, as listed in the table below.
Ports are always automatically configured on a Mellanox-based switch, with one exception — you only need to configure is MTU. You don't even need to enable auto-negotation, as the Mellanox firmware configures everything for you.
To configure auto-negotiation for a Broadcom-based switch, set
link-autoneg to on for all the switch ports. For example, to enable auto-negotiation for swp1 through swp52:
Any time you enable auto-negotiation, Cumulus Linux restores the default configuration settings specified in the table below.
By default on a Broadcom-based switch, auto-negotiation is disabled — except on 10G and 1000BASE-T switch ports, where it's required for links to work at all. And for RJ-45 SFP adapters, you need to manually configure the settings as described in the default settings table below.
If you disable it later or never enable it, then you have to configure the duplex, FEC and link speed settings manually using NCLU — see the relevant sections below. The default speed if you disable auto-negotiation depends on the type of connector used with the port. For example, a QSFP28 optic defaults to 100G, while a QSFP+ optic defaults to 40G and SFP+ defaults to 10G.
You should keep auto-negotiation enabled at all times. If you do decide to disable it, keep in mind the following:
- You must manually set link speed, duplex, pause and FEC.
- Disabling auto-negotiation on a copper cable of any kind prevents the port from optimizing the link through link training.
- Disabling auto-negotiation on a 1G optical cable prevents detection of single fiber breaks.
- You cannot disable auto-negotiation for 1GT or 10GT cables.
However, 10/100/1000BASE-T RJ-45 SFP adapters do not work with auto-negotiation enabled. You must manually configure these ports using the settings below (link-autoneg=off, link-speed=1000|100|10, link-duplex=full|half).
Depending upon the connector used for a port, enabling auto-negotiation also enables forward error correction (FEC), if the cable requires it (see the table below). FEC always adjusts for the speed of the cable. However, you cannot disable FEC separately using NCLU.
Default Interface Configuration Settings
The default configuration for each type of interface is described in the following table. Except as noted below, the settings for both sides of the link are expected to be the same.
If the other side of the link is running a version of Cumulus Linux earlier than 3.2, depending up on the interface type, auto-negotiation may not work on that switch. Cumulus Networks recommends you use the default settings on this switch in this case.
|Speed||Auto-negotiation||FEC Setting||Manual Configuration Steps|
|On||N/A (does not apply at this speed)|
|10/100BASE-T on a 1G fixed copper port||On||N/A|
(RJ-45 SFP adapter)
|1000BASE-T on a 1G fixed copper port||On||N/A|
|1000BASE-T on a 10G fixed copper port||On||N/A|
|10GBASE-T fixed copper port||On||N/A|
|40GBASE-CR4||Recommended On||Disable it|
Port Speed and Duplexing
Cumulus Linux supports both half- and full-duplex configurations. Supported port speeds include 100M, 1G, 10G, 25G, 40G, 50G and 100G. If you need to manually set the speed on a Broadcom-based switch, set it in terms of Mbps, where the setting for 1G is 1000, 40G is 40000 and 100G is 100000, for example.
The duplex mode setting defaults to full. You only need to specify
link duplex if you want half-duplex mode.
Example Port Speed and Duplexing Configuration
The following NCLU commands configure the port speed for the swp1 interface:
The above commands create the following
/etc/network/interfaces code snippet:
Port Speed Limitations
Ports can be configured to one speed less than their maximum speed.
|Switch Port Type||Lowest Configurable Speed|
|10G||1 Gigabit (1000 Mb)|
|100G||50G & 40G (with or without breakout port), 25G*, 10G*|
*Requires the port to be converted into a breakout port. See below.
Interface MTU (maximum transmission unit) applies to traffic traversing the management port, front panel/switch ports, bridge, VLAN subinterfaces and bonds — in other words, both physical and logical interfaces.
MTU is the only interface setting that must be set manually.
In Cumulus Linux,
ifupdown2 assigns 1500 as the default MTU setting. To change the setting, run:
Some switches may not support the same maximum MTU setting in hardware for both the management interface (eth0) and the data plane ports.
MTU for a Bridge
The MTU setting is the lowest MTU setting of any interface that is a member of that bridge (that is, every interface specified in
bridge-ports in the bridge configuration in the
interfaces file), even if another bridge member has a higher MTU value. There is no need to specify an MTU on the bridge. Consider this bridge configuration:
In order for bridge to have an MTU of 9000, set the MTU for each of the member interfaces (bond1 to bond 4, and peer5), to 9000 at minimum.
Use MTU 9216 for a bridge
Two common MTUs for jumbo frames are 9216 and 9000 bytes. The corresponding MTUs for the VNIs would be 9166 and 8950.
When configuring MTU for a bond, configure the MTU value directly under the bond interface; the configured value is inherited by member links/slave interfaces. If you need a different MTU on the bond, set it on the bond interface, as this ensures the slave interfaces pick it up. There is no need to specify MTU on the slave interfaces.
VLAN interfaces inherit their MTU settings from their physical devices or their lower interface; for example, swp1.100 inherits its MTU setting from swp1. Hence, specifying an MTU on swp1 ensures that swp1.100 inherits swp1's MTU setting.
VXLANs, the MTU for a virtual network interface (VNI) must be 50 bytes smaller than the MTU of the physical interfaces on the switch, as those 50 bytes are required for various headers and other data. You should also consider setting the MTU much higher than the default 1500.If you are working with
Example MTU Configuration
In general, the policy file specified above handles default MTU settings for all interfaces on the switch. If you need to configure a different MTU setting for a subset of interfaces, use NCLU.
The following commands configure an MTU minimum value of 9000 on swp1:
These commands create the following code snippet:
You must take care to ensure there are no MTU mismatches in the conversation path. MTU mismatches will result in dropped or truncated packets, degrading or blocking network performance.
The MTU for an SVI interface, such as vlan100, is derived from the bridge. When you use NCLU to change the MTU for an SVI and the MTU setting is higher than it is for the other bridge member interfaces, the MTU for all bridge member interfaces changes to the new setting. If you need to use a mixed MTU configuration for SVIs, for example, if some SVIs have a higher MTU and some lower, then set the MTU for all member interfaces to the maximum value, then set the MTU on the specific SVIs that need to run at a lower MTU.
To view the MTU setting, use
net show interface <interface>:
Setting a Policy for Global System MTU
For a global policy to set MTU, create a policy document (called
mtu.json here) like the following:
If your platform does not support a high MTU on eth0, you can set a lower MTU with the following command:
The policies and attributes in any file in
/etc/network/ifupdown2/policy.d/ override the default policies and attributes in
Creating a Default Policy for Various Interface Settings
Instead of configuring these settings for each individual interface, you can specify a policy for all interfaces on a switch, or tailor custom settings for each interface. Create a file in
/etc/network/ifupdown2/policy.d/, like in the following example (called
address.json), and populate the settings accordingly:
Configuring Breakout Ports
Cumulus Linux has the ability to:
- Break out 100G switch ports into the following with breakout cables:
- 2x50G, 4x25G, 4x10G
- Break out 40G switch ports into four separate 10G ports for use with breakout cables.
- Combine (also called aggregating or ganging) four 10G switch ports into one 40G port for use with a breakout cable (not to be confused with a bond).
To configure a 4x25G breakout port, first configure the port to break out then set the link speed:
On Dell switches with Maverick ASICs, you configure breakout ports on the 100G uplink ports by manually editing the
/etc/cumulus/ports.conf file. You need to specify either 4x10 or 4x25 for the port speed. For example, on a Dell S4148F-ON switch, to break out swp26 into 4 25G ports, you would modify the line starting with "26=" in
ports.conf as follows:
Then you need to configure the breakout ports in the
You cannot use NCLU to break out the uplink ports.
On Mellanox switches, you need to disable the next port (see below). In this example, you would also run the following before committing the update:
These commands create 4 interfaces in the
/etc/network/interfaces file named as follows:
When you commit your change configuring the breakout ports,
switchd restarts to apply the changes. The restart interrupts network services.
The breakout port configuration is stored in the
/etc/cumulus/ports.conf varies across different hardware platforms. Check the current list of supported platforms on the hardware compatibility list.
A snippet from the
/etc/cumulus/ports.conf on a Dell S6000 switch (with a Trident II+ ASIC) where swp3 is broken out as above looks like this:
Notice that you can break out any of the 100G ports into a variety of options: four 10G ports, four 25G ports or two 50G ports. Keep in mind that you cannot have more than 128 total logical ports on a Broadcom switch.
The Mellanox SN2700, SN2700B, SN2410, and SN2410B switches both have a limit of 64 logical ports in total. However, if you want to break out to 4x25G or 4x10G, you must configure the logical ports as follows:
- You can only break out odd-numbered ports into 4 logical ports.
- You must disable the next even-numbered port.
These restrictions do not apply to a 2x50G breakout configuration.
For example, if you have a 100G Mellanox SN2700 switch and break out port 11 into 4 logical ports, you must disable port 12 by running
net add interface swp12 breakout disabled, which results in this configuration in
There is no limitation on any port if interfaces are configured in 2x50G mode.
Here is an example showing how to configure breakout cables for the Mellanox Spectrum SN2700.
Removing a Breakout Port
To remove a breakout port, run
net add interface swp# breakout 1x. Continuing with the previous example:
Combining Four 10G Ports into One 40G Port
You can gang (aggregate) four 10G ports into one 40G port for use with a breakout cable, provided you follow these requirements:
- You must gang four 10G ports in sequential order. For example, you cannot gang swp1, swp10, swp20 and swp40 together.
- The ports must be in increments of four, with the starting port being swp1 (or swp5, swp9, or so forth); so you cannot gang swp2, swp3, swp4 and swp5 together.
For example, to gang swp1 through swp4 into a 40G port, run:
These commands create the following configuration snippet in the
Logical Switch Port Limitations
100G and 40G switches can support a certain number of logical ports, depending upon the manufacturer; these include:
- Mellanox SN2700 and SN2700B switches
- Switches with Broadcom Tomahawk, Trident II and Trident II+ chipsets (check the HCL)
Before you configure any logical/unganged ports on a switch, check the limitations listed in
/etc/cumulus/ports.conf; this file is specific to each manufacturer.
For example, the Dell S6000
ports.conf file indicates the logical port limitation like this:
The means the maximum number of ports for this Dell S6000 is 104.
Mellanox SN2700 and SN2700B switches have a limit of 64 logical ports in total. However, the logical ports must be configured in a specific way. See the note above.
Using ethtool to Configure Interfaces
The Cumulus Linux
ethtool command is an alternative for configuring interfaces as well as viewing and troubleshooting them.
For example, to manually set link speed, auto-negotiation, duplex mode and FEC on swp1, run:
To view the FEC setting on an interface, run:
Verification and Troubleshooting Commands
High-level interface statistics are available with the
net show interface command:
Low-level interface statistics are available with
Querying SFP Port Information
You can verify SFP settings using
ethtool -m . The following example shows the output for 1G and 10G modules:
Caveats and Errata
Timeout Error on Quanta LY8 and LY9 Switches
On Quanta T5048-LY8 and T3048-LY9 switches, an "Operation timed out" error occurs while removing and reinserting QSFP module.
The QSFPx2 module cannot be removed while the switch is powered on, as it is not hot-swappable. However, if this occurs, you can get the link to come up; however, this involves restarting
switchd , which disrupts your network.
On the T3048-LY9, run the following commands:
On the T5048-LY8, run the following commands:
swp33 and swp34 Disabled on Some Switches
The front SFP+ ports (swp33 and swp34) are disabled in Cumulus Linux on the following switches:
- Dell Z9100-ON
- Penguin Arctica 3200-series switches (the 3200C, 3200XL and 3200XLP)
- Supermicro SSE-C3632S
These ports appear as disabled in the
ethtool Shows Incorrect Port Speed on 100G Mellanox Switches
After setting interface speed to 40G by editing the
ports.conf file on a Mellanox switch,
ethtool still shows the speed as 100G.
This is a known issue whereby
ethtool does not update after restarting
switchd, so it continues to display the outdated port speed.
To correctly set the port speed, use NCLU or
ethtool to set the speed instead of hand editing the
For example, to set the speed to 40G using NCLU: