This documentation is for the extended support release (ESR) version of Cumulus Linux. We will continue to keep this content up to date until 21 February, 2023, when ESR support ends. For more information about ESR, please read this knowledge base article.

If you are using the current version of Cumulus Linux, the content on this page may not be up to date. The current version of the documentation is available here. If you are redirected to the main page of the user guide, then this page may have been renamed; please search for it there.

Monitoring Interfaces and Transceivers Using ethtool - ethtool Counter Definitions

The ethtool command enables you to query or control the network driver and hardware settings. It takes the device name (like swp1) as an argument. When the device name is the only argument to ethtool, it prints the current settings of the network device. See man ethtool(8) for details. Not all options are currently supported on switch port interfaces.

Monitor Interface Status Using ethtool

To check the status of an interface using ethtool:

cumulus@switch:~$ ethtool swp1
Settings for swp1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: off
        Current message level: 0x00000000 (0)
 
        Link detected: yes

To query interface statistics:

cumulus@switch:~$ sudo ethtool -S swp1
NIC statistics:
        HwIfInOctets: 1435339
        HwIfInUcastPkts: 11795
        HwIfInBcastPkts: 3
        HwIfInMcastPkts: 4578
        HwIfOutOctets: 14866246
        HwIfOutUcastPkts: 11791
        HwIfOutMcastPkts: 136493
        HwIfOutBcastPkts: 0
        HwIfInDiscards: 0
        HwIfInL3Drops: 0
        HwIfInBufferDrops: 0
        HwIfInAclDrops: 28
        HwIfInDot3LengthErrors: 0
        HwIfInErrors: 0
        SoftInErrors: 0
        SoftInDrops: 0
        SoftInFrameErrors: 0
        HwIfOutDiscards: 0
        HwIfOutErrors: 0
        HwIfOutQDrops: 0
        HwIfOutNonQDrops: 0
        SoftOutErrors: 0
        SoftOutDrops: 0
        SoftOutTxFifoFull: 0
        HwIfOutQLen: 0

View and Clear Interface Counters

Interface counters contain information about an interface. You can view this information when you run cl-netstat, ifconfig, or cat /proc/net/dev. You can also use cl-netstat to save or clear this information:

cumulus@switch:~$ sudo cl-netstat
Kernel Interface table
Iface   MTU Met        RX_OK RX_ERR RX_DRP RX_OVR        TX_OK TX_ERR TX_DRP TX_OVR    Flg
---------------------------------------------------------------------------------------------
eth0   1500   0          611      0      0      0          487      0      0      0   BMRU
lo    16436   0            0      0      0      0            0      0      0      0    LRU
swp1   1500   0            0      0      0      0            0      0      0      0    BMU
 
cumulus@switch:~$ sudo cl-netstat -c
Cleared counters

Option

Description

-c

Copies and clears statistics. It does not clear counters in the kernel or hardware.

The -c argument is applied per user ID by default. You can override it by using the -t argument to save statistics to a different directory.

-d

Deletes saved statistics, either the uid or the specified tag.

The -d argument is applied per user ID by default. You can override it by using the -t argument to save statistics to a different directory.

-D

Deletes all saved statistics.

-l

Lists saved tags.

-r

Displays raw statistics (unmodified output of cl-netstat).

-t <tag name>

Saves statistics with <tag name>.

-v

Prints cl-netstat version and exits.

Monitor Switch Port SFP/QSFP Hardware Information Using ethtool

To see hardware capabilities and measurement information on the SFP or QSFP module installed in a particular port, use the ethtool -m command. If the SFP/QSFP supports Digital Optical Monitoring (that is, the Optical diagnostics support field in the output below is set to Yes), the optical power levels and thresholds are also printed below the standard hardware details.

In the sample output below, you can see that this module is a 1000BASE-SX short-range optical module, manufactured by JDSU, part number PLRXPL-VI-S24-22. The second half of the output displays the current readings of the Tx power levels (Laser output power) and Rx power (Receiver signal average optical power), temperature, voltage and alarm threshold settings.

cumulus@switch$ sudo ethtool -m swp3
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x00 0x00 0x00 0x01 0x20 0x40 0x0c 0x05
        Transceiver type                          : Ethernet: 1000BASE-SX
        Transceiver type                          : FC: intermediate distance (I)
        Transceiver type                          : FC: Shortwave laser w/o OFC (SN)
        Transceiver type                          : FC: Multimode, 62.5um (M6)
        Transceiver type                          : FC: Multimode, 50um (M5)
        Transceiver type                          : FC: 200 MBytes/sec
        Transceiver type                          : FC: 100 MBytes/sec
        Encoding                                  : 0x01 (8B/10B)
        BR, Nominal                               : 2100MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 300m
        Length (62.5um)                           : 150m
        Length (Copper)                           : 0m
        Length (OM3)                              : 0m
        Laser wavelength                          : 850nm
        Vendor name                               : JDSU            
        Vendor OUI                                : 00:01:9c
        Vendor PN                                 : PLRXPL-VI-S24-22
        Vendor rev                                : 1   
        Optical diagnostics support               : Yes
        Laser bias current                        : 21.348 mA
        Laser output power                        : 0.3186 mW / -4.97 dBm
        Receiver signal average optical power     : 0.3195 mW / -4.96 dBm
        Module temperature                        : 41.70 degrees C / 107.05 degrees F
        Module voltage                            : 3.2947 V
        Alarm/warning flags implemented           : Yes
        Laser bias current high alarm             : Off
        Laser bias current low alarm              : Off
        Laser bias current high warning           : Off
        Laser bias current low warning            : Off
        Laser output power high alarm             : Off
        Laser output power low alarm              : Off
        Laser output power high warning           : Off
        Laser output power low warning            : Off
        Module temperature high alarm             : Off
        Module temperature low alarm              : Off
        Module temperature high warning           : Off
        Module temperature low warning            : Off
        Module voltage high alarm                 : Off
        Module voltage low alarm                  : Off
        Module voltage high warning               : Off
        Module voltage low warning                : Off
        Laser rx power high alarm                 : Off
        Laser rx power low alarm                  : Off
        Laser rx power high warning               : Off
        Laser rx power low warning                : Off
        Laser bias current high alarm threshold   : 10.000 mA
        Laser bias current low alarm threshold    : 1.000 mA
        Laser bias current high warning threshold : 9.000 mA
        Laser bias current low warning threshold  : 2.000 mA
        Laser output power high alarm threshold   : 0.8000 mW / -0.97 dBm
        Laser output power low alarm threshold    : 0.1000 mW / -10.00 dBm
        Laser output power high warning threshold : 0.6000 mW / -2.22 dBm
        Laser output power low warning threshold  : 0.2000 mW / -6.99 dBm
        Module temperature high alarm threshold   : 90.00 degrees C / 194.00 degrees F
        Module temperature low alarm threshold    : -40.00 degrees C / -40.00 degrees F
        Module temperature high warning threshold : 85.00 degrees C / 185.00 degrees F
        Module temperature low warning threshold  : -40.00 degrees C / -40.00 degrees F
        Module voltage high alarm threshold       : 4.0000 V
        Module voltage low alarm threshold        : 0.0000 V
        Module voltage high warning threshold     : 3.6450 V
        Module voltage low warning threshold      : 2.9550 V
        Laser rx power high alarm threshold       : 1.6000 mW / 2.04 dBm
        Laser rx power low alarm threshold        : 0.0100 mW / -20.00 dBm
        Laser rx power high warning threshold     : 1.0000 mW / 0.00 dBm
        Laser rx power low warning threshold      : 0.0200 mW / -16.99 dBm

ethtool Counter Definitions

Counter

Definition

HwIfInOctets

The total number of octets received on the interface, including framing characters.

HwIfInUcastPkts

The number of packets delivered by this sub-layer to a higher (sub-)layer that were not addressed to a multicast or broadcast address at this sub-layer.

HwIfInBcastPkts

The number of packets delivered by this sub-layer to a higher (sub-)layer that were addressed to a broadcast address at this sub-layer.

HwIfInMcastPkts

The number of packets delivered by this sub-layer to a higher (sub-)layer that were addressed to a multicast address at this sub-layer. For a MAC layer protocol, this includes both group and functional addresses.

HwIfOutOctets

The total number of octets transmitted out of the interface, including framing characters.

HwIfOutUcastPkts

The total number of packets that higher-level protocols requested be transmitted, and which were not addressed to a multicast or broadcast address at this sub-layer, including those that were discarded or not sent.

HwIfOutMcastPkts

The total number of packets that higher-level protocols requested to be transmitted, and which were addressed to a multicast address at this sub-layer, including those that were discarded or not sent. For a MAC layer protocol, this includes both group and functional addresses.

HwIfOutBcastPkts

The total number of packets that higher-level protocols requested be transmitted, and which were addressed to a broadcast address at this sub-layer, including those that were discarded or not sent.

HwIfInDiscards

The number of inbound packets that were chosen to be discarded even though no errors had been detected to prevent their being deliverable to a higher-layer protocol. One possible reason for discarding such a packet could be to free up buffer space.

The sum of all Rx discards on an interface including all of the more specific itemized ethtool counters. It also accounts for all other drops that do not have a more specific ethtool drop reason when a frame arrives that doesn't result in a valid forwarding decision - STP discarding, IGMP snooping drop, VLAN tag not configured

HwIfInL3Drops

All layer 3 packets that were discarded.

HwIfInBufferDrops

All ingress buffer congestion discards.

These are ingress buffer drops that are commonly seen during bursty congestion. Broadcom platforms have a buffer pool that is shared across all interfaces rather than a per-interface queue. When the global buffer pool is congested, InBufferDrops will accrue.

HwIfInAclDrops

All packets that were intentionally dropped.

These are common ACL drops for control plane policing or otherwise. cl-acltool -L all shows the current ACLs installed on the system.

HwIfInBlackholeDrops

All packets that were unintentionally dropped.

HwIfInDot3LengthErrors

A count of frames received on a particular interface with a length field value that falls between the minimum unpadded LLC data size and the maximum allowed LLC data size inclusive and that does not match the number of LLC data octets received. The count represented by an instance of this object also includes frames for which the length field value is less than the minimum unpadded LLC data size.

This counter accrues when the value of the length field in a frame does not match the number of octets received in the frame or it has incorrect padding - we've also seen incorrect padding or length field concerns in some legacy proprietary protocols used by other vendors such as CGMP. These frames are still forwarded.

HwIfInDot3FrameErrors

A count of frames received on a particular interface that are an integral number of octets in length but do not pass the FCS check. The count represented by an instance of this object is incremented when the frameCheckError status is returned by the MAC service to the LLC (or other MAC user). Received frames for which multiple error conditions obtain are, according to the conventions of [9], counted exclusively according to the error status presented to the LLC.

HwIfInErrors

For packet-oriented interfaces, the number of inbound packets that contained errors preventing them from being deliverable to a higher-layer protocol. For character-oriented or fixed-length interfaces, the number of inbound transmission units that contained errors preventing them from being deliverable to a higher-layer protocol.

This is the total of all "in " or Rx errors such as frame/FCS errors as outlined below

SoftInErrors

SoftInDrops

SoftInFrameErrors

HwIfOutDiscards

The number of outbound packets which were chosen to be discarded even though no errors had been detected to prevent their being transmitted. One possible reason for discarding such a packet could be to free up buffer space.

HwIfOutErrors

For packet-oriented interfaces, the number of outbound packets that could not be transmitted because of errors. For character-oriented or fixed-length interfaces, the number of outbound transmission units that could not be transmitted because of errors.

HwIfOutQDrops

HwIfOutNonQDrops

SoftOutErrors

SoftOutDrops

SoftOutTxFifoFull

HwIfOutQLen

The length of the output packet queue in packets.

HwIfInPausePkt

A count of MAC control frames received on this interface with an opcode indicating the PAUSE operation.

This counter does not increment when the interface is operating in half-duplex mode.

For interfaces operating at 10 Gb/s, this counter can roll over in less than 5 minutes if it is incrementing at its maximum rate. Since that amount of time could be less than a management station's poll cycle time, in order to avoid a loss of information, a management station is advised to poll the HwIfInPausePkt object for 10 Gb/s or faster interfaces.

HwIfOutPausePkt

A count of MAC control frames transmitted on this interface with an opcode indicating the PAUSE operation.

This counter does not increment when the interface is operating in half-duplex mode.

For interfaces operating at 10 Gb/s, this counter can roll over in less than 5 minutes if it is incrementing at its maximum rate. Since that amount of time could be less than a management station's poll cycle time, in order to avoid a loss of information, a management station is advised to poll the HwIfOutPausePkt object for 10 Gb/s or faster interfaces.

HwIfInPfc0Pkt

HwIfOutPfc0Pkt

HwIfInPfc1Pkt

HwIfOutPfc1Pkt

HwIfInPfc2Pkt

HwIfOutPfc2Pkt

HwIfInPfc3Pkt

HwIfOutPfc3Pkt

HwIfInPfc4Pkt

HwIfOutPfc4Pkt

HwIfInPfc5Pkt

HwIfOutPfc5Pkt

HwIfInPfc6Pkt

HwIfOutPfc6Pkt

HwIfInPfc7Pkt

HwIfOutPfc7Pkt

HwIfOutWredDrops

HwIfOutQ0WredDrops

HwIfOutQ1WredDrops

HwIfOutQ2WredDrops

HwIfOutQ3WredDrops

HwIfOutQ4WredDrops

HwIfOutQ5WredDrops

HwIfOutQ6WredDrops

HwIfOutQ7WredDrops

HwIfOutQ8WredDrops

HwIfOutQ9WredDrops