This documentation is for an older version of the software. If you are using the current version of Cumulus Linux, this content may not be up to date. The current version of the documentation is available here. If you are redirected to the main page of the user guide, then this page may have been renamed; please search for it there.

Monitoring Best Practices

The following monitoring processes are considered best practices for reviewing and troubleshooting potential issues with Cumulus RMP environments. In addition, several of the more common issues have been listed, with potential solutions included.

Overview

This document aims to provide two sets of outputs:

  1. Metrics that can be polled from Cumulus RMP and used in trend analysis

  2. Critical log messages that can be monitored for triggered alerts

Trend Analysis via Metrics

A metric is a quantifiable measure that is used to track and assess the status of a specific infrastructure component. It is a check collected over time. Examples of metrics include bytes on an interface, CPU utilization and total number of routes.

Metrics are more valuable when used for trend analysis.

Alerting via Triggered Logging

Triggered issues are normally sent to syslog, but could go to another log file depending on the feature. On Cumulus RMP, rsyslog handles all logging including local and remote logging. Logs are the best method to use for generating alerts when the system transitions from a stable steady state.

Sending logs to a centralized collector, then creating an alerts based on critical logs is optimal solution for alerting.

Hardware

The smond process provides monitoring functionality for various switch hardware elements. Mininmum/maximum values are output, depending on the flags applied to the basic command. The hardware elements and applicable commands/flags are listed in the table below:

Hardware Element

Monitoring Command/s

Interval Poll

Temperature

cumulus@switch:~$ smonctl -j
cumulus@switch:~$ smonctl -j -s TEMP[X]

600 seconds

Fan

cumulus@switch:~$ smonctl -j
cumulus@switch:~$ smonctl -j -s FAN[X]

600 seconds

PSU

cumulus@switch:~$ smonctl -j
cumulus@switch:~$ smonctl -j -s PSU[X]

600 seconds

PSU Fan

cumulus@switch:~$ smonctl -j
cumulus@switch:~$ smonctl -j -s PSU[X]Fan[X]

600 seconds

PSU Temperature

cumulus@switch:~$ smonctl -j
cumulus@switch:~$ smonctl -j -s PSU[X]Temp[X]

600 seconds

Voltage

cumulus@switch:~$ smonctl -j
cumulus@switch:~$ smonctl -j -s Volt[X]

600 seconds

Front Panel LED

cumulus@switch:~$ ledmgrd -d
cumulus@switch:~$ ledmgrd -j

600 seconds

Hardware Logs

Log Location

Log Entries

High temperature

/var/log/syslog
/usr/sbin/smond : : Temp2(Near the CPU (Right)): Temperature(51 C) is HIGH for last 71240 secs
/usr/sbin/smond : : Temp3(Top right corner): Temperature(43 C) is HIGH for last 71240 secs
/usr/sbin/smond : : Temp4:  Avg state is CRITICAL for last 60 secs. Last 10 values: [85.0, 85.5, 86.125, 86.625, 87.0, 87.625, 88.0, 88.625, 88.875, 89.25]. Generating cl-support and shutting down system...

Fan speed issues

/var/log/syslog
/usr/sbin/smond : : Fan5(Fan Tray 3): state changed from OK to HIGH
/usr/sbin/smond : : Fan5(Fan Tray 3): Fan speed 28912 RPM greater than 19000 RPM
/usr/sbin/smond : : Fan5(Fan Tray 3): state changed from HIGH to OK
...
/usr/sbin/smond : : Fan1: state changed from OK to ABSENT
/usr/sbin/smond : : Fan1:  Fan speed is at 0 RPM (not working or absent)

PSU failure

/var/log/syslog
/usr/sbin/smond : : PSU1: state changed from UNKNOWN to BAD

System Data

Cumulus RMP includes a number of ways to monitor various aspects of system data. In addition, alerts are issued in high risk situations.

CPU Idle Time

When a CPU reports five high CPU alerts within a span of 5 minutes, an alert is logged.

Short High CPU Bursts

Short bursts of high CPU can occur during switchd churn or routing protocol startup. Do not set alerts for these short bursts.

System Element

Monitoring Command/s

Interval Poll

CPU utilization

cumulus@switch:~$ cat /proc/stat
cumulus@switch:~$ top -b -n 1

30 seconds

CPU Logs

Log Location

Log Entries

High CPU

/var/log/syslog
sysmonitor: High CPU use: 91%
sysmonitor: CPU use no longer high: 58%
sysmonitor: Critically high load average: 1.31 (1.31)
sysmonitor: High load average: 1.24 (1.24)
sysmonitor: Load Average no longer high: 0.88 (0.88)

Cumulus RMP 3.0 and later monitors CPU, memory and disk space via sysmonitor. The configurations for the thresholds are stored in /etc/cumulus/sysmonitor.conf. More information is available via man sysmonitor.

CPU measureThresholds
UseAlert: 90% Crit: 95%
Process LoadAlarm: 95% Crit: 125%
Click here to see differences between Cumulus RMP 2.5 ESR and 3.0 and later...

CPU Logs

Log Location

Log Entries

High CPU

/var/log/syslog
jdoo[2803]: 'localhost' cpu system usage of 41.1% matches resource limit [cpu system usage>30.0%]
jdoo[4727]: 'localhost' sysloadavg(15min) of 111.0 matches resource limit [sysloadavg(15min)>110.0]

In Cumulus RMP 2.5, CPU logs are created with each unique threshold:

CPU measure< 2.5 Threshold
User70%
System30%
Wait20%

Cumulus RMP 2.5, CPU and Memory warnings are generated via jdoo. The configuration for the thresholds are stored in /etc/jdoo/jdoorc.d/cl-utilities.rc.

Memory Usage

When the memory utilization exceeds 90% a warning is logged and a cl-support is generated.

System Element

Monitoring Command/s

Interval Poll

Memory utilization

cumulus@switch:~$ cat     
/proc/meminfo     
cumulus@switch:~$ cat     
/usr/bin/free    

30 seconds

Disk Usage

When monitoring disk utilization tmpfs can be excluded from monitoring.

System Element

Monitoring Command/s

Interval Poll

Disk utilization

cumulus@switch:~$ /bin/df -x tmpfs

300 seconds

Process Restart

In Cumulus RMP 3.0 and later, systemd is responsible for monitoring and restarting processes.

Process Element

Monitoring Command/s

View processes monitored by systemd

cumulus@switch:~$     
systemctl status    
Click here to changes from Cumulus Linux 2.5 ESR to 3.0 and later...

Cumulus RMP 2.5.4 through 2.5 ESR uses a forked version of monit called jdoo to monitor processes. If the process ever fails, jdoo then invokes init.d to restart the process.

Process Element

Monitoring Command/s

View processes monitored by jdoo

cumulus@switch:~$ jdoo summary

View process restarts

cumulus@switch:~$ sudo cat /var/log/syslog

View current process state

cumulus@switch:~$ ps -aux

Layer 1 Protocols and Interfaces

Link and port state interface transitions are logged to /var/log/syslog and /var/log/switchd.log.

Interface Element

Monitoring Command/s

Link state

    
cumulus@switch:~$ cat /sys/class/net/[iface]/operstate     

cumulus@switch:~$ net show interface all json

Link speed

    
cumulus@switch:~$ cat /sys/class/net/[iface]/speed     

cumulus@switch:~$ net show interface all json

Port state

cumulus@switch:~$     
ip link show

cumulus@switch:~$ net show interface all json

Bond state

cumulus@switch:~$     
cat /proc/net/bonding/[bond]

cumulus@switch:~$ net show interface all json

Interface counters are obtained from either querying the hardware or the Linux kernel. The two outputs should align, but the Linux kernel aggregates the output from the hardware.

Interface Counter Element

Monitoring Command/s

Interval Poll

Interface counters

    
cumulus@switch:~$ cat /sys/class/net/[iface]/statistics/[stat_name]    

cumulus@switch:~$ net show counters json cumulus@switch:~$ cl-netstat -j cumulus@switch:~$ ethtool -S [iface]

10 seconds

Layer 1 Logs

Log Location

Log Entries

Link failure/Link flap

/var/log/switchd.log
switchd[5692]: nic.c:213 nic_set_carrier: swp17: setting kernel carrier: down
switchd[5692]: netlink.c:291 libnl: swp1, family 0, ifi 20, oper down
...
switchd[5692]: nic.c:213 nic_set_carrier: swp1: setting kernel carrier: up
switchd[5692]: netlink.c:291 libnl: swp17, family 0, ifi 20, oper up

Unidirectional link

/var/log/switchd.log
/var/log/ptm.log
ptmd[7146]: ptm_bfd.c:2471 Created new session 0x1 with peer 10.255.255.11 port swp1
ptmd[7146]: ptm_bfd.c:2471 Created new session 0x2 with peer fe80::4638:39ff:fe00:5b port swp1
...
ptmd[7146]: ptm_bfd.c:2471 Session 0x1 down to peer 10.255.255.11, Reason 8
ptmd[7146]: ptm_bfd.c:2471 Detect timeout on session 0x1 with peer 10.255.255.11, in state 1

Bond Negotiation

  • Working

/var/log/syslog
kernel: [73946.052292] bonding: Bond1 is being created...
kernel: [73946.062957] Bond1: Enslaving swp49 as a backup interface with an up link
kernel: [73946.081609] Bond1: Enslaving swp50 as a backup interface with an up link
kernel: [73946.090636] IPv6: ADDRCONF(NETDEV_UP): Bond1: link is not ready
kernel: [74590.925353] IPv6: ADDRCONF(NETDEV_CHANGE): Bond1: link becomes ready

Bond Negotiation

  • Failing

/var/log/syslog
kernel: [73946.052292] bonding: Bond1 is being created...
kernel: [73946.062957] Bond1: Enslaving swp49 as a backup interface with an up link
kernel: [73946.081609] Bond1: Enslaving swp50 as a backup interface with an up link
kernel: [73946.090636] IPv6: ADDRCONF(NETDEV_UP): Bond1: link is not ready 

Prescriptive Topology Manager (PTM) uses LLDP information to compare against a topology.dot file that describes the network. It has built in alerting capabilities, so it is preferable to use PTM on box rather than polling LLDP information regularly. The PTM code is available on the Cumulus Networks github repository. Additional PTM, BFD and associated logs are documented in the code.

Peering information should be tracked through PTM. For more information, refer to the Prescriptive Topology Manager documentation.

Neighbor Element

Monitoring Command/s

Interval Poll

LLDP Neighbor

cumulus@switch:~$ lldpctl -f json

300 seconds

Prescriptive Topology Manager

cumulus@switch:~$ ptmctl -j [-d]

Triggered

Layer 2 Protocols

Spanning tree is a protocol that prevents loops in a layer 2 infrastructure. In a stable state, the spanning tree protocol should stably converge. Monitoring the Topology Change Notifications (TCN) in STP helps identify when new BPDUs were received.

Interface Counter Element

Monitoring Command/s

Interval Poll

STP TCN Transitions

cumulus@switch:~$ mstpctl showbridge json
cumulus@switch:~$ mstpctl showport json

60 seconds

Layer 2 Logs

Log Location

Log Entries

Spanning Tree Working

/var/log/syslog
kernel: [1653877.190724] device swp1 entered promiscuous mode
kernel: [1653877.190796] device swp2 entered promiscuous mode
mstpd: create_br: Add bridge bridge
mstpd: clag_set_sys_mac_br: set bridge mac 00:00:00:00:00:00
mstpd: create_if: Add iface swp1 as port#2 to bridge bridge
mstpd: set_if_up: Port swp1 : up
mstpd: create_if: Add iface swp2 as port#1 to bridge bridge
mstpd: set_if_up: Port swp2 : up
mstpd: set_br_up: Set bridge bridge up
mstpd: MSTP_OUT_set_state: bridge:swp1:0 entering blocking state(Disabled)
mstpd: MSTP_OUT_set_state: bridge:swp2:0 entering blocking state(Disabled)
mstpd: MSTP_OUT_flush_all_fids: bridge:swp1:0 Flushing forwarding database
mstpd: MSTP_OUT_flush_all_fids: bridge:swp2:0 Flushing forwarding database
mstpd: MSTP_OUT_set_state: bridge:swp1:0 entering learning state(Designated)
mstpd: MSTP_OUT_set_state: bridge:swp2:0 entering learning state(Designated)
sudo: pam_unix(sudo:session): session closed for user root
mstpd: MSTP_OUT_set_state: bridge:swp1:0 entering forwarding state(Designated)
mstpd: MSTP_OUT_set_state: bridge:swp2:0 entering forwarding state(Designated)
mstpd: MSTP_OUT_flush_all_fids: bridge:swp2:0 Flushing forwarding database
mstpd: MSTP_OUT_flush_all_fids: bridge:swp1:0 Flushing forwarding database

Spanning Tree Blocking

/var/log/syslog
mstpd: MSTP_OUT_set_state: bridge:swp2:0 entering blocking state(Designated)
mstpd: MSTP_OUT_set_state: bridge:swp2:0 entering learning state(Designated)
mstpd: MSTP_OUT_set_state: bridge:swp2:0 entering forwarding state(Designated)
mstpd: MSTP_OUT_flush_all_fids: bridge:swp2:0 Flushing forwarding database
mstpd: MSTP_OUT_flush_all_fids: bridge:swp2:0 Flushing forwarding database
mstpd: MSTP_OUT_set_state: bridge:swp2:0 entering blocking state(Alternate)
mstpd: MSTP_OUT_flush_all_fids: bridge:swp2:0 Flushing forwarding database

Layer 3 Protocols

Routing Logs

Layer 3 Logs

Log Location

Log Entries

Routing protocol process crash

/var/log/syslog
watchquagga[3044]: bgpd state -> down : read returned EOF
cumulus-core: Running cl-support for core files bgpd.3030.1470341944.core.core_helper
core_check.sh[4992]: Please send /var/support/cl_support__spine01_20160804_201905.tar.xz to Cumulus support.
watchquagga[5241]: Forked background command [pid 6665]: /usr/sbin/service quagga restart bgpd
watchquagga[7719]: watchquagga 0.99.24+cl3u2 watching [zebra bgpd ospfd], mode [phased zebra restart]
watchquagga[7719]: zebra state -> up : connect succeeded
watchquagga[7719]: bgpd state -> up : connect succeeded
watchquagga[7719]: ospfd state -> up : connect succeeded

Logging

The table below covers the various log files, and what they should be used for:

OSPF Element

Monitoring Command/s

Log Location

Syslog

Catch all log file. Identifies memory leaks and CPU spikes.

/var/log/syslog

Switchd functionality

Hardware Abstraction Layer (HAL).

/var/log/switchd.log

Protocols and Services

NTP

Run the following command to confirm the NTP process is working correctly, and that the switch clock is synced with NTP:

cumulus@switch:~$ /usr/bin/ntpq -p

Device Management

Device Access Logs

Access Logs

Log Location

Log Entries

User Authentication and Remote Login

/var/log/syslog
sshd[31830]: Accepted publickey for cumulus from 192.168.0.254 port 45582 ssh2: RSA 38:e6:3b:cc:04:ac:41:5e:c9:e3:93:9d:cc:9e:48:25
sshd[31830]: pam_unix(sshd:session): session opened for user cumulus by (uid=0)

Device Super User Command Logs

Super User Command Logs

Log Location

Log Entries

Executing commands using sudo

/var/log/syslog
sudo:  cumulus : TTY=pts/2 ; PWD=/home/cumulus ; USER=root ; COMMAND=/bin/uname -a
sudo: pam_unix(sudo:session): session opened for user root by cumulus(uid=0)
sudo: pam_unix(sudo:session): session closed for user root