This chapter introduces monitoring and troubleshooting Cumulus Linux.
Using the Serial Console
The serial console can be a useful tool for debugging issues, especially when you find yourself rebooting the switch often or if you don’t have a reliable network connection.
The default serial console baud rate is 115200, which is the baud rate ONIE uses.
Configuring the Serial Console on PowerPC or ARM Switches
On PowerPC switches, the U-Boot environment variable
baudrate identifies the baud rate of the serial console. To change the
baudrate variable, use the
cumulus@switch:~$ sudo fw_setenv baudrate 9600 Updating environment variable: `baudrate' Proceed with update [N/y]? y
You must reboot the switch for the
baudrate change to take effect.
The valid values for
Configuring the Serial Console on x86 Switches
On x86 switches, you configure serial console baud rate by editing
grubcan cause the switch to be inaccessible via the console. Grub changes should be carefully reviewed before implementation.
The valid values for the baud rate are:
To change the serial console baud rate:
/etc/default/grub. The two relevant lines in
/etc/default/grubare as follows; replace the 115200 value with a valid value specified above in the
--speedvariable in the first line and in the
consolevariable in the second line:
GRUB_SERIAL_COMMAND="serial --port=0x2f8 --speed=115200 --word=8 --parity=no --stop=1" GRUB_CMDLINE_LINUX="console=ttyS1,115200n8 cl_platform=accton_as5712_54x"
After you save your changes to the grub configuration, type the following at the command prompt:
If you plan on accessing your switch's BIOS over the serial console, you need to update the baud rate in the switch BIOS. For more information, see this knowledge base article.
Reboot the switch.
Diagnostics Using cl-support
You can use
cl-support to generate a single export file that contains various details and the configuration from a switch. This is useful for remote debugging and troubleshooting.
You should run
cl-support before you submit a support request to Cumulus Networks as this file helps in the investigation of issues:
cumulus@switch:~$ sudo cl-support -h Usage: cl-support [-h] [reason]... Args: [reason]: Optional reason to give for invoking cl-support. Saved into tarball's reason.txt file. Options: -h: Print this usage statement
cumulus@switch:~$ ls /var/support cl_support_20130806_032720.tar.xz
The directory structure is compressed using LZMA2 compression and can be extracted using the
cumulus@switch:~$ cd /var/support cumulus@switch:~$ sudo unxz cl_support_20130729_140040.tar.xz cumulus@switch:~$ sudo tar xf cl_support_20130729_140040.tar cumulus@switch:~$ ls -l cl_support_20130729_140040/ -rwxr-xr-x 1 root root 7724 Jul 29 14:00 cl-support -rw-r--r-- 1 root root 52 Jul 29 14:00 cmdline.args drwxr-xr-x 2 root root 4096 Jul 29 14:00 core drwxr-xr-x 64 root root 4096 Jul 29 13:51 etc drwxr-xr-x 4 root root 4096 Jul 29 14:00 proc drwxr-xr-x 2 root root 4096 Jul 29 14:01 support drwxr-xr-x 3 root root 4096 Jul 29 14:00 sys drwxr-xr-x 3 root root 4096 Aug 8 15:22 var
The directory contains the following elements:
|core||Contains the core files generated from Cumulus Linux HAL process, |
|etc||Is a replica of the switch’s |
|log||Is a replica of the switch's |
|proc||Is a replica of the switch’s |
|support||Is a set of files containing further system information, which is obtained by |
cl-support, when untarred, contains a
reason.txt file. This file indicates what reason triggered it. When contacting Cumulus Networks technical support, please attach the
cl-support file if possible. For more information about
cl-support, read Understanding and Decoding the cl-support Output File.
Sending Log Files to a syslog Server
All logging on Cumulus Linux is done with rsyslog.
rsyslog provides both local logging to the
syslog file as well as the ability to export logs to an external
syslog server. High precision timestamps are enabled for all
rsyslog log files; here's an example:
There are applications in Cumulus Linux that write directly to a log file without going through
rsyslog. These files are typically located in
Starting with Cumulus Linux 2.5.4, all Cumulus Linux rules have been moved from
/etc/rsyslog.conf into separate files in
/etc/rsyslog.d/, which are called at the end of the
GLOBAL DIRECTIVES section of
/etc/rsyslog.conf. As a result, the
RULES section at the end of
rsyslog.conf is ignored because the messages have to be processed by the rules in
/etc/rsyslog.d and then dropped by the last line in
Most logs within Cumulus Linux are sent through
rsyslog, which then writes them to files in the
/var/log directory. There are default rules in the
/etc/rsyslog.d/ directory that define where the logs are written:
|Sets defaults for log messages, include log format and log rate limits.|
|15-crit.conf||Logs crit, alert or emerg log messages to |
|99-syslog.conf||All remaining processes that use |
Log files that are rotated are compressed into an archive. Processes that do not use
rsyslog write to their own log files within the
/var/log directory. For more information on specific log files, see Troubleshooting Log Files.
Enabling Remote syslog
If you need to send other log files — such as
switchd logs — to a
syslog server, do the following:
Create a file in
/etc/rsyslog.d/. Make sure it starts with a number lower than 99 so that it executes before log messages are dropped in, such as
25-switchd.conf. Our example file is called
/etc/rsyslog.d/11-remotesyslog.conf. Add content similar to the following:
This configuration sends log messages to a remote
syslogserver for the following processes:
if $programnameline sends the log files to the
syslogserver. It follows the same syntax as the
/var/log/syslogfile, where @ indicates UDP, 192.168.1.2 is the IP address of the
syslogserver, and 514 is the UDP port.
For TCP-based syslog, use two @@ before the IP address: @@192.168.1.2:514.
syslogover TCP places a burden on the switch to queue packets in the
syslogbuffer. This may cause detrimental effects if the remote
syslogserver becomes unavailable.
The numbering of the files in
/etc/rsyslog.d/dictates how the rules are installed into
rsyslog.d. If you want to remotely log only the messages in
/var/syslog, and not those in
/var/log/switchd.log, for instance, then name the file
98-remotesyslog.conf, since it's lower than the
In the above configuration, each setting is defined as follows:
|$ModLoad imfile||Enables the |
|$InputFileName||The file to be sent to the |
|$InputFileStateFile||This is used by |
|$InputFileTag||Defines the |
|$InputFileSeverity||Defines the logging severity level sent to the |
|$InputFileFacility||Defines the logging format. local7 is common.|
|$InputFilePollInterval||Defines how frequently in seconds |
|$InputRunFileMonitor||Enables the file monitor module with the configured settings.|
In most cases, the settings to customize include:
|$InputFileName||The file to stream to the |
|$InputFileStateFile||A unique name for each file being watched.|
|$InputFileTag||A prefix to the log message on the server.|
The links below discuss more specific monitoring topics.
- Single User Mode - Boot Recovery
- Using netshow to Troubleshoot Your Network Configuration
- Monitoring Interfaces and Transceivers Using ethtool
- Resource Diagnostics Using cl-resource-query
- Monitoring System Hardware
- Monitoring System Statistics and Network Traffic with sFlow
- Monitoring Virtual Device Counters
- Understanding and Decoding the cl-support Output File
- Managing Application Daemons
- Troubleshooting Network Interfaces
- Network Troubleshooting
- SNMP Monitoring