This topic describes how to upgrade Cumulus Linux on your switches to a more recent release.

Cumulus Networks recommends that you deploy, provision, configure, and upgrade switches using automation, even with small networks or test labs. During the upgrade process, you can quickly upgrade dozens of devices in a repeatable manner. Using tools like Ansible, Chef, or Puppet for configuration management greatly increases the speed and accuracy of the next major upgrade; these tools also enable the quick swap of failed switch hardware.

Contents

 This topic describes ...

Before You Upgrade Cumulus Linux

Be sure to read the knowledge base article Upgrades: Network Device and Linux Host Worldview Comparison, which provides a detailed comparison between the network device and Linux host worldview of upgrade and installation.

Understanding the location of configuration data is required for successful upgrades, migrations, and backup. As with other Linux distributions, the /etc directory is the primary location for all configuration data in Cumulus Linux. The following list is a likely set of files that you need to back up and migrate to a new release. Make sure you examine any file that has been changed. Cumulus Networks recommends you consider making the following files and directories part of a backup strategy.

 Network Configuration Files
File Name and LocationExplanationCumulus Linux DocumentationDebian Documentation
/etc/network/Network configuration files, most notably /etc/network/interfaces and /etc/network/interfaces.d/Switch Port AttributesN/A
/etc/resolv.confDNS resolutionNot unique to Cumulus Linux: wiki.debian.org/NetworkConfigurationwww.debian.org/doc/manuals/debian-reference/ch05.en.html
/etc/frr/Routing application (responsible for BGP and OSPF)FRRouting OverviewN/A
/etc/hostnameConfiguration file for the hostname of the switchQuick Start Guidewiki.debian.org/HowTo/ChangeHostname
/etc/cumulus/acl/*Netfilter configurationNetfilter - ACLsN/A
/etc/cumulus/ports.confBreakout cable configuration fileSwitch Port AttributesN/A; please read the guide on breakout cables
/etc/cumulus/switchd.confSwitchd configurationConfiguring switchdN/A; please read the guide on switchd configuration

If you are using the root user account, consider including /root/.

If you have custom user accounts, consider including /home/<username>/.

 Additional Commonly Used Files
File Name and LocationExplanationCumulus Linux DocumentationDebian Documentation
/etc/motdMessage of the dayNot unique to Cumulus Linuxwiki.debian.org/motd
/etc/passwdUser account informationNot unique to Cumulus Linuxwww.debian.org/doc/manuals/debian-reference/ch04.en.html
/etc/shadowSecure user account informationNot unique to Cumulus Linuxwww.debian.org/doc/manuals/debian-reference/ch04.en.html
/etc/groupDefines user groups on the switchNot unique to Cumulus Linuxwww.debian.org/doc/manuals/debian-reference/ch04.en.html
/etc/lldpd.confLink Layer Discover Protocol (LLDP) daemon configurationLink Layer Discovery Protocolpackages.debian.org/wheezy/lldpd
/etc/lldpd.d/Configuration directory for lldpdLink Layer Discovery Protocolpackages.debian.org/wheezy/lldpd
/etc/nsswitch.confName Service Switch (NSS) configuration fileTACACS PlusN/A
/etc/ssh/SSH configuration filesSSH for Remote Accesswiki.debian.org/SSH

/etc/sudoers

/etc/sudoers.d

Best practice is to place changes in /etc/sudoers.d/ instead of /etc/sudoers; changes in the /etc/sudoers.d/ directory are not lost during upgrade. If you are upgrading from a release prior to 3.2 (such as 3.1.2) to a 3.2 or later release, be aware that the sudoers file changed in Cumulus Linux 3.2. Using sudo to Delegate Privileges

If you are using the root user account, consider including /root/.

If you have custom user accounts, consider including /home/<username>/.

 Files to Never Migrate between Versions or Switches
File Name and LocationExplanation
/etc/adjtimeSystem clock adjustment data. NTP manages this automatically. It is incorrect when the switch hardware is replaced. Do not copy.
/etc/bcm.d/Per-platform hardware configuration directory, created on first boot. Do not copy.
/etc/mlx/Per-platform hardware configuration directory, created on first boot. Do not copy.
/etc/blkid.tabPartition table. Do not modify manually. Do not copy.
/etc/blkid.tab.oldA previous partition table. Do not modify manually. Do not copy.
/etc/cumulus/initPlatform hardware-specific files. Do not copy.
/etc/default/clagdCreated and managed by ifupdown2. Do not copy.
/etc/default/grubGrub init table. Do not modify manually.
/etc/default/hwclockPlatform hardware-specific file. Created during first boot. Do not copy.
/etc/initPlatform initialization files. Do not copy.
/etc/init.d/Platform initialization files. Do not copy.
/etc/fstabStatic info on filesystem. Do not copy.
/etc/image-releaseSystem version data. Do not copy.
/etc/os-releaseSystem version data. Do not copy.
/etc/lsb-releaseSystem version data. Do not copy.
/etc/lvm/archiveFilesystem files. Do not copy.
/etc/lvm/backupFilesystem files. Do not copy.
/etc/modulesCreated during first boot. Do not copy.
/etc/modules-load.d/Created during first boot. Do not copy.
/etc/sensors.dPlatform-specific sensor data. Created during first boot. Do not copy.
/root/.ansibleAnsible tmp files. Do not copy.
/home/cumulus/.ansibleAnsible tmp files. Do not copy.

If you are using certain forms of network virtualization, including VMware NSX-V or Midokura MidoNet, you might have updated the /usr/share/openvswitch/scripts/ovs-ctl-vtep file. This file is not marked as a configuration file; therefore, if the file contents change in a newer version of Cumulus Linux, they overwrite any changes you made to the file. Cumulus Networks recommends you back up this file before upgrading.  

Upgrade Cumulus Linux

You can upgrade Cumulus Linux in one of two ways:

  • Perform a binary (full image) install of the new version, using ONIE. This is the preferred method.
  • Upgrade only the changed packages using the sudo -E apt-get update and sudo -E apt-get upgrade command. 

Upgrading an MLAG pair requires additional steps. If you are using MLAG to dual connect two Cumulus Linux switches in your environment, follow the steps in Upgrade Switches in an MLAG Pair, below to ensure a smooth upgrade.

Binary Install (ONIE)

You can use the binary install (ONIE) to upgrade your switch. ONIE is an open source project (equivalent to PXE on servers) that enables the installation of network operating systems (NOS) on a bare metal switch. This upgrade method enables you to choose the exact version to which you want to upgrade and is the only method available to upgrade your switch to a new release train (for example, an upgrade from 2.5.6 to 3.6.0).

Be aware of the following when using binary install:

  • This method is destructive; any configuration files on the switch are not saved; copy them to a different server before upgrading via ONIE.
  • You must move configuration data to the new OS using ZTP while the OS is first booted, or soon afterwards using out-of-band management.
  • Moving a configuration file might cause issues;
    • Identifying all the locations of configuration data is not always an easy task. See Before You Upgrade Cumulus Linux.
    • Merge conflicts with configuration file changes in the new version might go undetected.
  • If configuration files are not restored correctly, you might be unable to attach to the switch from in-band management. Out-of-band connectivity (eth0 or console) is recommended.
  • You must reinstall and reconfigure third-party applications after upgrade.

To upgrade the switch using binary install:

  1. Back up the configurations off the switch.

  2. Install the binary image. Follow the instructions in Installing a New Cumulus Linux Image.

  3. Restore the configuration files to the new version — ideally with automation.
  4. Verify correct operation with the old configurations on the new version.
  5. Reinstall third party applications and associated configurations.

Package Install

Cumulus Linux completely embraces the Linux and Debian upgrade workflow, where you use an installer to install a base image, then perform any upgrades within that release train with -E apt-get update and -E apt-get upgrade commands. Any packages that have been changed since the base install get upgraded in place from the repository. All switch configuration files remain untouched, or in rare cases merged (using the Debian merge function) during the package upgrade.

When you use package install to upgrade your switch, configuration data stays in place while the packages are upgraded. If the new version changes a configuration file that you changed previously, a prompt appears during the upgrade process prompting you for the version you want to use or if you want to evaluate the differences. In addition, you do not have to reinstall third party applications and associated configurations after the upgrade.

Be aware of the following when using package install:

  • You cannot upgrade the switch to a new release train. For example, you cannot upgrade the switch from 2.5.6 to 3.6.0, but you can upgrade the switch from 3.1.0 to 3.2.1, or from 3.6.0 to 3.6.2.
  • You cannot choose the exact release version that you want to run. When you upgrade, you upgrade all packages to the latest available version.
  • Various switch functions might be intermittently available during the upgrade.
  • If you are upgrading from a release earlier than 3.6.2, certain upgrade operations terminate SSH sessions on the in-band (front panel) ports, leaving you unable to monitor the upgrade process. (As a workaround, you can use the dtach tool.)
  • After you upgrade, user names and group names created by packages might be different on different switches, depending on the configuration and package installation history.
  • The sudo -E apt-get upgrade and  sudo -E apt-get install commands cause disruptions to network services:
    • The sudo -E apt-get upgrade command might result in services being restarted or stopped as part of the upgrade process.

    • The sudo -E apt-get install command might disrupt core services by changing core service dependency packages.


To upgrade the switch using package install:

  1. Back up the configurations from the switch.

  2. Fetch the latest update metadata from the repository.

    cumulus@switch$ sudo -E apt-get update
  3. Review potential upgrade issues (in some cases, upgrading new packages might also upgrade additional existing packages due to dependencies). Run the following command to see the additional packages that will be installed or upgraded.

    cumulus@switch$ sudo -E apt-get install --dry-run
  4. Upgrade all the packages to the latest distribution.

    cumulus@switch$ sudo -E apt-get upgrade

    If no reboot is required after the upgrade completes, the upgrade stops and restarts all upgraded services, and logs messages in the /var/log/syslog file similar to the ones shown below. (In the examples below, only the frr package was upgraded.)

    Policy: Service frr.service action stop postponed
    Policy: Service frr.service action start postponed
    Policy: Restarting services: frr.service
    Policy: Finished restarting services
    Policy: Removed /usr/sbin/policy-rc.d
    Policy: Upgrade is finished


    If the upgrade process encounters changed configuration files that have new versions in the release to which you are upgrading, you see a message similar to this:

    Configuration file '/etc/frr/daemons'
    ==> Modified (by you or by a script) since installation.
    ==> Package distributor has shipped an updated version.
    What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
    D : show the differences between the versions
    Z : start a shell to examine the situation
    The default action is to keep your current version.
    *** daemons (Y/I/N/O/D/Z) [default=N] ?

    - To see the differences between the currently installed version and the new version, type D.

    - To keep the currently installed version, type N. The new package version is installed with the suffix _.dpkg-dist (for example, /etc/frr/daemons.dpkg-dist). When upgrade is complete and before you reboot, merge your changes with the changes from the newly installed file.

    -To install the new version, type I. Your currently installed version is saved with the suffix .dpkg-old.

    When the upgrade is complete, you can search for the files with the sudo find / -mount -type f -name '*.dpkg-*' command.

    If you see errors for expired GPG keys that prevent you from upgrading packages when upgrading to Cumulus Linux from version 3.5.1 or earlier, follow the steps in Upgrading Expired GPG Keys.

  5. Reboot the switch if the upgrade messages indicate that a system restart is required.

    cumulus@switch$ sudo -E apt-get upgrade
          ... upgrade messages here ...
     
    *** Caution: Service restart prior to reboot could cause unpredictable behavior
    *** System reboot required ***
    cumulus@switch$ sudo reboot
  6. Verify correct operation with the old configurations on the new version.

Upgrade Notes 

Package install always updates to the latest available release on the Cumulus Linux repository. For example, if you are currently running Cumulus Linux 3.0.1 and run the sudo -E apt-get upgrade command on that switch, the packages are upgraded to the latest versions contained in the latest 3.y.z release.

Because Cumulus Linux is a collection of different Debian Linux packages, be aware of the following:

  • The /etc/os-release and /etc/lsb-release files are updated to the currently installed Cumulus Linux version when you upgrade the switch using either package install or binary install with ONIE. For example, if you run sudo -E apt-get upgrade and the latest Cumulus Linux version on the repository is 3.7.1, these two files display the version as 3.7.1 after the upgrade.
  • The /etc/image-release file is updated only when you run a binary install with ONIE. Therefore, if you run a binary install of Cumulus Linux version 3.5.0, followed by a package upgrade using sudo -E apt-get upgrade to 3.7.1, the /etc/image-release file continues to display Cumulus Linux version is 3.5.0, which is the originally installed base image.

Configuration File Migration Script

If you are upgrading a switch from Cumulus Linux 2.5.6 or later in the 2.y.z release train to Cumulus 3.y.z, you can use the Configuration File Migration Script with the --backup option to create a backup archive of configuration files from the previous release train, copy them off the box, then install them on the new version switch.

Configuration files in Cumulus Linux 2.5.6+ typically migrate to version 3.y.z without any problems; however, there are certain known issues:

  • Never migrate certain configuration files between versions or while replacing hardware. See Files to Never Migrate between Versions or Switches, above.
  • Do not migrate /etc/passwd and /etc/shadow  to the new version directly. The examples and the Ansible script included with Configuration File Migration Script explicitly excludes these two files from the backup archive. You must change the default password for the cumulus user and add any locally created users to the new installation after the upgrade completes.
  • You must completely upgrade /etc/apt/sources.list with a new 3.x repository and repository structure. Remove repositories from Cumulus Linux 2.5.6+. If there are any custom repositories on the switch, you must migrate them into the new sources.list file or the sources.d/ directory.
  • Version 2.5.6+ configurations are not guaranteed to work in Cumulus Linux 3.y.z. Be sure to test the restoration and proper operation of the Cumulus Linux 2.5.6+ configuration in Cumulus Linux 3.y.z on a non-production switch or in a Cumulus VX image; every deployment is unique.

The following example excludes /etc/apt/etc/passwd and /etc/shadow from the backup archive.

  1. Back up the files from the previous release train (for example 2.5.6).

    # Make a temp dir
    loc=$(mktemp -d)
    # Create a backup archive to the temp dir
    sudo ./config_file_changes --backup --backup_dir $loc --exclude /etc/apt,/etc/passwd,/etc/shadow
    # Copy the archive and log file to an external server
    sudo scp -r $loc/* user@my_external_server:.

    Optional: Use the Ansible playbook included with the Configuration File Migration script to automate Cumulus Linux switch backup. The playbook creates a backup archive of the Cumulus Linux switch configuration files and retrieves them to a central server — automating backup for all deployed Cumulus Linux 2.5.6+ switches. This is a quick start to setting up automated configuration and control for your deployment.

  2. Install Cumulus Linux 3.x onto the switch using ONIE.
  3. Reinstall the files from the config file archive to the newly installed switch.

    # On the switch, copy the config file archive back from the server:
    scp user@my_external_server:PATH/SWITCHNAME-config-archive-DATE_TIME.tar.gz .
    # Untar the archive to the root of the box
    sudo tar -C / -xvf SWITCHNAME-config-archive-DATE_TIME.tar.gz

Upgrade Switches in an MLAG Pair

If you are using MLAG to dual connect two switches in your environment, follow the steps below according to the version of Cumulus Linux from which you are upgrading. 

Upgrade from Cumulus Linux 3.y.z to a Later 3.y.z Release

To upgrade the switches:

  1. Verify the switch is in the secondary role:

    cumulus@switch:~$ clagctl status
  2. Update the Cumulus Linux repositories:

    cumulus@switch:~$ sudo apt-get update
  3. Shut down the core uplink layer 3 interfaces:

    cumulus@switch:~$ sudo ip link set swpX down
  4. Shut down the peerlink:

    cumulus@switch:~$ sudo ip link set peerlink down
  5. Update the Cumulus Linux image and packages:

    cumulus@switch:~$ sudo apt-get upgrade
  6. Reboot the switch:

    cumulus@switch:~$ sudo reboot
  7. If you were originally running Cumulus Linux 3.0.0 through 3.3.2, follow the steps for upgrading from Quagga to FRRouting.
  8. Verify STP convergence across both switches:

    cumulus@switch:~$ mstpctl showall
  9. Verify core uplinks and peerlinks are UP:

    cumulus@switch:~$ net show interface
  10. Verify MLAG convergence:

    cumulus@switch:~$ clagctl status
  11. Make this secondary switch the primary:

    cumulus@switch:~$ clagctl priority 2048
  12. Verify the other switch is now in the secondary role.
  13. Repeat steps 2-9 on the new secondary switch.
  14. Remove the priority 2048 and restore the priority back to 32768 on the current primary switch:

    cumulus@switch:~$ clagctl priority 32768

Upgrade from Cumulus Linux 2.y.z to Version 3.y.z

If you are using MLAG to dual connect two switches in your environment and those switches are still running Cumulus Linux 2.5 ESR or any other release earlier than 3.0.0, the switches are not dual-connected after you upgrade the first switch. 

To upgrade the switches:

  1. Disable clagd in the /etc/network/interfaces file (set clagd-enable to no), then restart switchd, networking, and FRR services.

    cumulus@switch:~$ sudo systemctl restart switchd.service
    cumulus@switch:~$ sudo systemctl restart networking.service
    cumulus@switch:~$ sudo systemctl restart frr.service
  2. If you are using BGP, notify the BGP neighbors that the switch is going down:

    cumulus@switch:~$ sudo vtysh -c "config t" -c "router bgp" -c "neighbor X.X.X.X shutdown"
  3. Stop the Quagga service: 

    cumulus@switch:~$ sudo systemctl stop [quagga|frr].service 
  4. Bring down all the front panel ports:

    cumulus@switch:~$ sudo ip link set swp<#> down
  5. Run cl-img-select -fr to boot the switch in the secondary role into ONIE, then reboot the switch.
  6. Install Cumulus Linux onto the secondary switch using ONIE. At this time, all traffic goes to the switch in the primary role.
  7. After the install, copy the license file and all the configuration files you backed up, then restart the switchd, networking, and Quagga services. All traffic is still going to the primary switch.

    cumulus@switch:~$ sudo systemctl restart switchd.service
    cumulus@switch:~$ sudo systemctl restart networking.service
    cumulus@switch:~$ sudo systemctl restart quagga.service
  8. Run cl-img-select -fr to boot the switch in the primary role into ONIE, then reboot the switch. Now, all traffic is going to the switch in the secondary role that you just upgraded to version 3.7.
  9. Install Cumulus Linux onto the primary switch using ONIE. 
  10. After the install, copy the license file and all the configuration files you backed up.
  11. Follow the steps for upgrading from Quagga to FRRouting.
  12. Enable clagd again in the /etc/network/interfaces file (set clagd-enable to yes), then run ifreload -a.

    cumulus@switch:~$ sudo ifreload -a
  13. Bring up all the front panel ports:

    cumulus@switch:~$ sudo ip link set swp<#> up 

    The two switches are dual-connected again and traffic flows to both switches.

Roll Back a Cumulus Linux Installation

Even the most well planned and tested upgrades can result in unforeseen problems; sometimes the best solution is to roll back to the previous state.

There are three main strategies; all require detailed planning and execution:
  • Back out individual packages: If you identify the problematic package, you can downgrade the affected package directly. In rare cases, you might need to restore the configuration files from backup or edit to back out any changes made automatically by the upgrade package.
  • Flatten and rebuild: If the OS becomes unusable, you can use orchestration tools to reinstall the previous OS release from scratch and then rebuild the configuration automatically. 
  • Backup and restore: Another common strategy is to restore to a previous state using a backup captured before the upgrade.

The method you employ is specific to your deployment strategy, so providing detailed steps for each scenario is outside the scope of this document.

Third Party Packages

Third party packages in Linux host world often use the same package system as the distribution into which it is to be installed (for example, Debian uses apt-get). Or, the package might be compiled and installed by the system administrator. Configuration and executable files generally follow the same filesystem hierarchy standards as other applications.

If you install any third party applications on a Cumulus Linux switch, configuration data is typically installed into the /etc directory, but it is not guaranteed. It is your responsibility to understand the behavior and configuration file information of any third party packages installed on the switch.

After you upgrade using a full binary install, you need to reinstall any third party packages or any Cumulus Linux add-on packages, such as vxsnd or vxrd.

Related Information