In order to use Routing on the Host, you must install the Cumulus Quagga package on your host servers. Routing on the Host is supported on the following environments:

  • Ubuntu 12.04-LTS, Precise Pangolin
  • Ubuntu 14.04-LTS, Trusty Tahr
  • Ubuntu 16.04-LTS, Xenial Xerus
  • Red Hat Enterprise Linux 7
  • Docker containers

Download the tar archive containing the Cumulus Quagga installer for your server host operating system. 

Contents

Installing the Cumulus Quagga Package on a Server Host

  1. If the non-Cumulus Networks version of Quagga is already installed on the host, you must uninstall that package before you install Cumulus Quagga.
    • Ubuntu hosts:

      root@host:~# dpkg -r quagga 
    • RHEL 7 hosts: 

      root@host:~# rpm --erase quagga
  2. On the server host, untar the Cumulus Quagga installer.
    •  Ubuntu 12.04 Precise Pangolin
      root@host:~# tar xvf roh-ubuntu1204.tar
    •  Ubuntu 14.04 Trusty Tahr
      root@host:~# tar xvf roh-ubuntu1404.tar
    •  Ubuntu 16.04, Xenial Xerus
      root@host:~# tar xvf roh-ubuntu1604.tar
    •  Red Hat Enterprise Linux (RHEL) 7
      root@host:~# tar xvf roh-rhel7.tar
  3. Ubuntu 12.04 Precise Pangolin only: If it is not already installed, install the Ubuntu libjson0 package using apt:

    root@host:~# apt-get install libjson0
  4. Download the desired Cumulus Linux Quagga packages. Download the quagga*.deb (if installing on Ubuntu) or quagga*.rpm (if installing on RHEL) files.
  5. Install the Cumulus Quagga packages. Follow the steps for your host.

    For Ubuntu hosts, if Cumulus Quagga is already running, the dpkg -i command will disrupt the network, shutting down Cumulus Quagga before installing the latest version. To avoid this issue, you should use a console or out-of-band management to connect to the host.

     

    •  Ubuntu 12.04 Precise Pangolin
      root@host:~# dpkg -i quagga_0.99.23.1-1+cl3u2_precise_amd64.deb quagga-dbg_0.99.23.1-1+cl3u2_precise_amd64.deb quagga-doc_0.99.23.1-1+cl3u2_precise_all.deb
    •  Ubuntu 14.04 Trusty Tahr
      root@host:~# dpkg -i quagga_0.99.23.1-1+cl3u2_trusty_amd64.deb quagga-dbg_0.99.23.1-1+cl3u2_trusty_amd64.deb quagga-doc_0.99.23.1-1+cl3u2_trusty_all.deb
    •  Ubuntu 16.04, Xenial Xerus
      root@host:~# dpkg -i quagga_0.99.23.1-1+cl3u2_xenial_amd64.deb quagga-dbg_0.99.23.1-1+cl3u2_xenial_amd64.deb quagga-doc_0.99.23.1-1+cl3u2_xenial_all.deb 
    •  Red Hat Enterprise Linux (RHEL) 7
      root@host:~# rpm -i quagga-0.99.23.1+cl3u2-2016062001.el7.centos.x86_64.rpm quagga-devel-0.99.23.1+cl3u2-2016062001.el7.centos.x86_64.rpm quagga-debuginfo-0.99.23.1+cl3u2-2016062001.el7.centos.x86_64.rpm quagga-contrib-0.99.23.1+cl3u2-2016062001.el7.centos.x86_64.rpm
  6. Create a sysctl text file, /etc/sysctl.d/99quagga_defaults.conf, and populate it with the following content.

    # /etc/sysctl.d/99quagga_defaults.conf
    # Place this file at the location above and reload the device.
    # or run the sysctl -p /etc/sysctl.d/99quagga_defaults.conf
     
    #Required if we're to work as a router
    net.ipv4.conf.all.rp_filter = 0
    net.ipv4.conf.default.rp_filter = 0
    net.ipv4.conf.lo.rp_filter = 0
    net.ipv4.conf.all.forwarding = 1
    net.ipv4.conf.default.forwarding = 1
    net.ipv4.ip_forward = 1
    net.ipv4.conf.default.arp_announce = 2
    net.ipv4.conf.default.arp_notify = 1
    net.ipv4.conf.default.arp_ignore=1
    net.ipv4.conf.all.arp_announce = 2
    net.ipv4.conf.all.arp_notify = 1
    net.ipv4.conf.all.arp_ignore=1
    net.ipv4.icmp_errors_use_inbound_ifaddr=1
    # igmp
    net.ipv4.igmp_max_memberships=1000
    net.ipv4.neigh.default.mcast_solicit = 10
    # mld
    net.ipv6.mld_max_msf=512
    # neigh
    net.ipv4.neigh.default.gc_thresh2=7168
    net.ipv4.neigh.default.gc_thresh3=8192
    net.ipv4.neigh.default.base_reachable_time_ms=14400000
    net.ipv6.neigh.default.gc_thresh2=3584
    net.ipv6.neigh.default.gc_thresh3=4096
    net.ipv6.neigh.default.base_reachable_time_ms=14400000
    # Routes
    net.ipv6.route.max_size=131072
    net.ipv4.conf.all.ignore_routes_with_linkdown=1
    net.ipv6.conf.all.ignore_routes_with_linkdown=1
    # keep ipv6 permanent addresses on an admin down
    net.ipv6.conf.all.keep_addr_on_down=1
    # use neigh information on selection of nexthop for multipath hops
    net.ipv4.fib_multipath_use_neigh=1
    # Tweaking Read/Write Memory Values
    net.core.rmem_max = 8388608
    net.core.wmem_max = 8388608
    net.core.optmem_max = 65536
    net.core.rmem_default = 266240
    net.core.wmem_default = 266240
    # Allows Apps to Work with VRF
    net.ipv4.tcp_l3mdev_accept=1
  7. Load the new sysctl file.

    root@host:~# sysctl -p /etc/sysctl.d/99quagga_defaults.conf
    net.ipv6.conf.all.disable_ipv6=0
    net.ipv4.conf.all.rp_filter = 0
    net.ipv4.conf.default.rp_filter = 0
    net.ipv4.conf.lo.rp_filter = 0
    net.ipv4.conf.all.forwarding = 1
    net.ipv4.conf.default.forwarding = 1
    net.ipv4.ip_forward = 1
    net.ipv4.conf.default.arp_announce = 2
    net.ipv4.conf.default.arp_notify = 1
    net.ipv4.conf.default.arp_ignore = 1
    net.ipv4.conf.all.arp_announce = 2
    net.ipv4.conf.all.arp_notify = 1
    net.ipv4.conf.all.arp_ignore = 1
    net.ipv4.icmp_errors_use_inbound_ifaddr = 1
    net.ipv4.igmp_max_memberships = 1000
    net.ipv4.neigh.default.mcast_solicit = 10
    net.ipv6.mld_max_msf = 512
    net.ipv4.neigh.default.gc_thresh2 = 7168
    net.ipv4.neigh.default.gc_thresh3 = 8192
    net.ipv4.neigh.default.base_reachable_time_ms = 14400000
    net.ipv6.neigh.default.gc_thresh2 = 3584
    net.ipv6.neigh.default.gc_thresh3 = 4096
    net.ipv6.neigh.default.base_reachable_time_ms = 14400000
    net.ipv6.route.max_size = 131072
    net.ipv4.conf.all.ignore_routes_with_linkdown = 1
    net.ipv6.conf.all.ignore_routes_with_linkdown = 1
    sysctl: cannot stat /proc/sys/net/ipv6/conf/all/keep_addr_on_down: No such file or directory
    sysctl: cannot stat /proc/sys/net/ipv4/fib_multipath_use_neigh: No such file or directory
    net.core.rmem_max = 8388608
    net.core.wmem_max = 8388608
    net.core.optmem_max = 65536
    net.core.rmem_default = 266240
    net.core.wmem_default = 266240
    net.ipv4.tcp_l3mdev_accept = 1

    Depending on your kernel version and configuration, certain variables may not exist and will show "No such file or directory" errors. This is expected and can be ignored.

  8. Enable the daemons you intend to use (such as bgpd, ospfd or ospf6d). For more information, see Configuring Cumulus Quagga.

  9. Start the Cumulus Quagga service.
    •  Ubuntu 12.04 or 14.04
      root@host:~# service quagga restart
    •  Ubuntu 16.04 or RHEL 7
      root@host:~# systemctl start quagga.service

Preventing the Cumulus Quagga Package from Being Overwritten

Since the Cumulus Quagga package is not the same version of Quagga that was installed on your host initially, the next time you update the host OS, the Cumulus Quagga package will get replaced with the upstream version of Quagga included with your host OS.

  • Ubuntu hosts: You need to put the Cumulus Quagga package on apt hold. On your Ubuntu host, run the following command:

    root@host:~# apt-mark hold quagga
  • RHEL 7 hosts: You need to exclude the Cumulus Quagga package from yum updates. On your RHEL 7 host, yum.conf and add the following line: 

    exclude=quagga

Installing the Cumulus Quagga Package in a Docker Container

Before you install Cumulus Quagga in a Docker container, keep in mind that you cannot install an initd container on a systemd host and vice versa. Both the container and the host on which is resides must both use systemd or both use initd.

Docker must always be run in privileged mode; use the --privileged option.

The following steps were done on an Ubuntu 16.04 host. 

  1. Install the Docker engine.
    1. Create a file called docker.list in /etc/apt/sources.list.d with the following content:

      root@host:/etc/apt/sources.list.d# vi docker.list
      deb https://apt.dockerproject.org/repo ubuntu-xenial main
    2. Install the key.

      root@host:/etc/apt/sources.list.d# apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
       Click to see the output ...
      Executing: /tmp/tmp.HNelHyq1PD/gpg.1.sh --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
      gpg: requesting key 2C52609D from hkp server p80.pool.sks-keyservers.net
      gpg: key 2C52609D: public key "Docker Release Tool (releasedocker) <docker@docker.com>" imported
      gpg: Total number processed: 1
      gpg:               imported: 1  (RSA: 1)
    3. Update the package lists.

      root@host:/etc/apt/sources.list.d# apt-get update
       Click to see the output ...
      Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
      Hit:2 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease      
      Hit:3 http://security.ubuntu.com/ubuntu xenial-security InRelease       
      Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease    
      Get:5 https://apt.dockerproject.org/repo ubuntu-xenial InRelease [20.6 kB]
      Fetched 20.6 kB in 0s (46.3 kB/s)                         
      Reading package lists... Done
    4. Install Docker on the Ubuntu host.

      root@host:/etc/apt/sources.list.d# apt-get install docker-engine
       Click to see the output ...
      Reading package lists... Done
      Building dependency tree       
      Reading state information... Done
      The following additional packages will be installed:
      aufs-tools cgroupfs-mount git git-man liberror-perl libltdl7
      Suggested packages:
      mountall git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
      git-gui gitk gitweb git-arch git-cvs git-mediawiki git-svn
      The following NEW packages will be installed:
       aufs-tools cgroupfs-mount docker-engine git git-man liberror-perl libltdl7
      0 upgraded, 7 newly installed, 0 to remove and 86 not upgraded.
      Need to get 18.4 MB of archives.
      After this operation, 99.3 MB of additional disk space will be used.
      Do you want to continue? [Y/n] Y
      Get:1 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 aufs-tools amd64 1:3.2+20130722-1.1ubuntu1 [92.9 kB]
      Get:2 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 docker-engine amd64 1.11.2-0~xenial [14.5 MB]
      Get:3 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4,970 B]
      Get:4 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 libltdl7 amd64 2.4.6-0.1 [38.3 kB]
      Get:5 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 liberror-perl all 0.17-1.2 [19.6 kB]
      Get:6 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 git-man all 1:2.7.4-0ubuntu1 [735 kB]
      Get:7 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 git amd64 1:2.7.4-0ubuntu1 [3,006 kB]
      Fetched 18.4 MB in 8s (2,076 kB/s)                                             
      Selecting previously unselected package aufs-tools.
      (Reading database ... 59668 files and directories currently installed.)
      Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1ubuntu1_amd64.deb ...
      Unpacking aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
      Selecting previously unselected package cgroupfs-mount.
      Preparing to unpack .../cgroupfs-mount_1.2_all.deb ...
      Unpacking cgroupfs-mount (1.2) ...
      Selecting previously unselected package libltdl7:amd64.
      Preparing to unpack .../libltdl7_2.4.6-0.1_amd64.deb ...
      Unpacking libltdl7:amd64 (2.4.6-0.1) ...
      Selecting previously unselected package docker-engine.
      Preparing to unpack .../docker-engine_1.11.2-0~xenial_amd64.deb ...
      Unpacking docker-engine (1.11.2-0~xenial) ...
      Selecting previously unselected package liberror-perl.
      Preparing to unpack .../liberror-perl_0.17-1.2_all.deb ...
      Unpacking liberror-perl (0.17-1.2) ...
      Selecting previously unselected package git-man.
      Preparing to unpack .../git-man_1%3a2.7.4-0ubuntu1_all.deb ...
      Unpacking git-man (1:2.7.4-0ubuntu1) ...
      Selecting previously unselected package git.
      Preparing to unpack .../git_1%3a2.7.4-0ubuntu1_amd64.deb ...
      Unpacking git (1:2.7.4-0ubuntu1) ...
      Processing triggers for libc-bin (2.23-0ubuntu3) ...
      Processing triggers for man-db (2.7.5-1) ...
      Processing triggers for ureadahead (0.100.0-19) ...
      Processing triggers for systemd (229-4ubuntu4) ...
      Setting up aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
      Setting up cgroupfs-mount (1.2) ...
      Setting up libltdl7:amd64 (2.4.6-0.1) ...
      Setting up docker-engine (1.11.2-0~xenial) ...
      Setting up liberror-perl (0.17-1.2) ...
      Setting up git-man (1:2.7.4-0ubuntu1) ...
      Setting up git (1:2.7.4-0ubuntu1) ...
      Processing triggers for libc-bin (2.23-0ubuntu3) ...
      Processing triggers for ureadahead (0.100.0-19) ...
      Processing triggers for systemd (229-4ubuntu4) ...
  2. Check the Docker service on the Ubuntu 16.04 host.

    root@host:/etc/apt/sources.list.d# ps -ef | grep docker
    root      5378     1  0 19:35 ?        00:00:00 /usr/bin/docker daemon -H fd://
    root      5399  5378  0 19:35 ?        00:00:00 docker-containerd -l /var/run/docker/libcontainerd/docker-containerd.sock --runtime docker-runc --start-timeout 2m
    vagrant   5538  3579  0 19:38 pts/0    00:00:00 grep --color=auto docker
  3. Optional: Pull the Cumulus Quagga container image from Dockerhub. If you don't pull the image here, it will be done in the next step for you automatically.

    root@host:/etc/apt/sources.list.d# docker pull cumulusnetworks/quagga:latest
     Click to see the output ...
    latest: Pulling from cumulusnetworks/quagga
    5ba4f30e5bea: Pull complete 
    9d7d19c9dc56: Pull complete 
    ac6ad7efd0f9: Pull complete 
    e7491a747824: Pull complete 
    a3ed95caeb02: Pull complete 
    a2e15afd186f: Pull complete 
    c79a17f1dd48: Pull complete 
    b9ce745e3bfd: Pull complete 
    871c6c942d89: Pull complete 
    0295811cd443: Pull complete 
    6aaf3dc92cbd: Pull complete 
    e6e55669dd82: Pull complete 
    e3a986b5efb0: Pull complete 
    dd11dcae39d1: Pull complete 
    2feeda2bf3a6: Pull complete 
    Digest: sha256:dcabf5df4f631719a709bf84f1536d91ebd2d0c115f97b55090c9afaefac5657
    Status: Downloaded newer image for cumulusnetworks/quagga:latest
  4. Create the container.
    1. Create the container in privileged mode, naming the container Quagga. The container ID gets returned, even if you specify a name, as you can see in the example below.

      root@host:/root# docker run -t -d --net=host --privileged --restart unless-stopped --name Quagga cumulusnetworks/quagga:latest
      cf0daeb70ceaf32d15b3fd27f80a355b3e88b178f7d7d53840c395d93e568a73

      The container must run in privileged mode to interact with the kernel routing table.

    2. Optional: To create the container using a custom Quagga.conf configuration or with daemons other than BGP enabled, run the container while mounting files as volumes, as shown below.

      root@host:/etc/apt/sources.list.d# docker run -t -d --net=host --privileged --name Quagga \
          -v /root/Quagga.conf:/etc/quagga/Quagga.conf \
          -v /root/daemons:/etc/quagga/daemons \
          cumulusnetworks/quagga:latest
      
  5. Validate the container.
    1. Check all containers on the system.

      root@host:/root# docker ps -a
       Click here to see the output ...
      CONTAINER ID        IMAGE                                  COMMAND             CREATED             STATUS                      PORTS               NAMES
      cf0daeb70cea        cumulusnetworks/quagga:latest          "/bin/bash"         2 minutes ago       Up 2 minutes                                    Quagga

      Running docker ps  (without the -a option) only shows active ("up") containers on the system; the -a option shows all containers.

    2. Check the configuration on the container.

      root@host:/root# docker exec -i -t Quagga /usr/bin/vtysh
       Click here to see the output ...
      root@host:/root# docker exec -i -t Quagga /usr/bin/vtysh
      Hello, this is Quagga (version 0.99.23.1+cl3u2).
      Copyright 1996-2005 Kunihiro Ishiguro, et al.
      host# show run
      Building configuration...
      Current configuration:
      !
      username cumulus nopassword
      !
      service integrated-vtysh-config
      !
      interface docker0
       ipv6 nd suppress-ra
       link-detect
      !
      interface lo
       link-detect
      !
      interface veth0c6e8a7
       link-detect
      !
      ip forwarding
      !
      line vty
      !
      end
      host#

Starting the Container Automatically when the System Boots

  1. Create the systemd unit file /etc/systemd/system/quagga-docker.service and populate the file with the following content.

    [Unit]
    Description=Cumulus Quagga Container
    After=docker.service network-online.target
    Requires=docker.service
    
    
    [Service]
    Restart=always
    TimeoutStartSec=0
    #One ExecStart/ExecStop line to prevent hitting bugs in certain systemd versions
    ExecStart=/bin/sh -c 'docker rm -f Quagga; \
              docker pull cumulusnetworks/quagga:latest; \
              docker run -t --net=host --privileged --name Quagga cumulusnetworks/quagga:latest'
    ExecStop=-/bin/sh -c '/usr/bin/docker stop Quagga; \
              /usr/bin/docker rm -f Quagga'
    
    
    [Install]
    WantedBy=multi-user.target
  2. Set the container to start at boot time by enabling the quagga service.

    root@host:# systemctl enable quagga.service
  3. Optional: Start the container.

    root@host:# systemctl start quagga.service

Configuring Cumulus Quagga

Once you've finished installing Cumulus Quagga on your servers or in your containers, you are ready to configure it