This chapter introduces monitoring and troubleshooting Cumulus Linux.
Using the Serial Console
The serial console can be a useful tool for debugging issues, especially when you find yourself rebooting the switch often or if you don’t have a reliable network connection.
The default serial console baud rate is 115200, which is the baud rate ONIE uses.
Configuring the Serial Console on ARM Switches
On ARM switches, the U-Boot environment variable
baudrate identifies the baud rate of the serial console. To change the
baudrate variable, use the
You must reboot the switch for the
baudrate change to take effect.
The valid values for
Configuring the Serial Console on x86 Switches
On x86 switches, you configure serial console baud rate by editing
grubcan cause the switch to be inaccessible via the console. Grub changes should be carefully reviewed before implementation.
The valid values for the baud rate are:
To change the serial console baud rate:
/etc/default/grub. The two relevant lines in
/etc/default/grubare as follows; replace the 115200 value with a valid value specified above in the
--speedvariable in the first line and in the
consolevariable in the second line:
After you save your changes to the grub configuration, type the following at the command prompt:
If you plan on accessing your switch's BIOS over the serial console, you need to update the baud rate in the switch BIOS. For more information, see this knowledge base article.
Reboot the switch.
Getting General System Information
Two commands are helpful for getting general information about the switch and the version of Cumulus Linux you are running. These are helpful with system diagnostics and if you need to submit a support request to Cumulus Networks.
For information about the version of Cumulus Linux running on the switch, run
net show version, which displays the contents of
For general information about the switch, run
net show system, which gathers information about the switch from a number of files in the system:
Diagnostics Using cl-support
You can use
cl-support to generate a single export file that contains various details and the configuration from a switch. This is useful for remote debugging and troubleshooting.
You should run
cl-support before you submit a support request to Cumulus Networks as this file helps in the investigation of issues:
The directory structure is compressed using LZMA2 compression and can be extracted using the
The directory contains the following elements:
|core||Contains the core files on the switch, including those generated from |
|etc||Is a replica of the switch’s |
|log||Is a replica of the switch's |
|proc||Is a replica of the switch’s |
|support||Is a set of files containing further system information, which is obtained by |
cl-support, when untarred, contains a
cmdline.args file. This file indicates what reason triggered it. When contacting Cumulus Networks technical support, please attach the
cl-support file if possible. For more information about
cl-support, read Understanding and Decoding the cl-support Output File.
If you have issues extracting the script with the
tar command, like an error saying the file does not look like tar archive, try using the
unxz command first:
cumulus@switch:~$ sudo unxz cl_support_20130729_140040.tar.xz
You can save a lot of disk space and perhaps some time if you do not run
unxz on the tar file.
Sending Log Files to a syslog Server
All logging on Cumulus Linux is done with rsyslog.
rsyslog provides both local logging to the
syslog file as well as the ability to export logs to an external
syslog server. High precision timestamps are enabled for all
rsyslog log files; here's an example:
Local logging: Most logs within Cumulus Linux are sent to files in the
/var/log directory. Most relevant information is placed within the
/var/log/syslog file. For more information on specific log files, see Troubleshooting Log Files.
Export logging: To send
syslog files to an external
syslog server, add a rule specifying to copy all messages (*.*) to the IP address and switch port of your
syslog server in the
rsyslog configuration files as described below.
In the following example, 192.168.1.2 is the remote
syslog server and 514 is the port number. For UDP-based syslog, use a single @ before the IP address: @192.168.1.2:514. For TCP-based syslog, use two @@ before the IP address: @@192.168.1.2:514.
Create a file called something like
/etc/rsyslog.d/90-remotesyslog.conf. Make sure it starts with a number lower than 99 so that it executes before
99-syslog.conf. Add content like the following:
All Cumulus Linux rules are stored in separate files in
/etc/rsyslog.d/, which are called at the end of the
GLOBAL DIRECTIVES section of
/etc/rsyslog.conf. As a result, the
RULES section at the end of
rsyslog.conf is ignored because the messages have to be processed by the rules in
/etc/rsyslog.d and then dropped by the last line in
In the case of the
switchd rules file, the file must be numbered lower than 25. For example,
If you need to send other log files (e.g. switchd logs) to a
syslog server, configure a new file in
/etc/rsyslog.d, as described above, and add lines similar to the following lines:
In the above configuration, each setting is defined as follows:
|$ModLoad imfile||Enables the |
|$InputFileName||The file to be sent to the |
|$InputFileStateFile||This is used by |
|$InputFileTag||Defines the |
|$InputFileSeverity||Defines the logging severity level sent to the |
|$InputFileFacility||Defines the logging format. local7 is common.|
|$InputFilePollInterval||Defines how frequently in seconds |
|$InputRunFileMonitor||Enables the file monitor module with the configured settings.|
In most cases, the settings to customize include:
|$InputFileName||The file to stream to the |
|$InputFileStateFile||A unique name for each file being watched.|
|$InputFileTag||A prefix to the log message on the server.|
if $programname line is what sends the log files to the
syslog server. It follows the same syntax as the
/var/log/syslog file, where @ indicates UDP, 192.168.1.2 is the IP address of the
syslog server, and 514 is the UDP port. The value switchd must match the value in
Harmless syslog Error: Failed to reset devices.list
The following message gets logged to
/var/log/syslog when you run
systemctl daemon-reload and during system boot:
This message is harmless, and can be ignored. It is logged when
systemd attempts to change cgroup attributes that are read only. The upstream version of systemd has been modified to not log this message by default.
systemctl daemon-reload command is often issued when Debian packages are installed, so the message may be seen multiple times when upgrading packages.
The links below discuss more specific monitoring topics.
- Single User Mode - Boot Recovery
- Resource Diagnostics Using cl-resource-query
- Monitoring System Hardware
- Monitoring Virtual Device Counters
- Understanding and Decoding the cl-support Output File
- Troubleshooting Network Interfaces
- Network Troubleshooting
- SNMP Monitoring
- Monitoring Best Practices