The following sections describe how to configure network interfaces and FRRouting for a two-leaf/two-spine Cumulus VX network topology:

  • Two spine VMs that represent two spine (aggregation layer) switches on the network.

  • Two leaf VMs that represent two leaf (access layer) switches on the network.

     

These instructions assume that you have installed the relevant images and hypervisors, created four VMs with appropriate names, and that the VMs are running. Refer to the Getting Started chapter of this guide for more information on setting up the VMs.

Contents

Configure CumulusVX-leaf1

You can configure each of the VMs using the Network Command Line Utility (NCLU) or by editing the /etc/network/interfaces and /etc/frr/frr.conf files as the sudo user.

The following configuration uses unnumbered IP addressing, where you use the same /32 IPv4 address on multiple ports. OSPF unnumbered doesn't have an equivalent to RFC-5549, so you need to use an IPv4 address to bring up the adjacent OSPF neighbors, allowing you to reuse the same IP address. You can see some example unnumbered OSPF configurations in the knowledge base.

To configure CumulusVX-leaf1:

  1. Log into the CumulusVX-leaf1 VM using the default credentials:

    • username: cumulus

    • password: CumulusLinux!

  2. As the sudo user, edit the /etc/frr/daemons file in a text editor. Set zebra, bgpd, and ospfd to yes, and save the file.

    zebra=yes
    bgpd=yes
    ospfd=yes
    ...
  3. Run the following commands to configure the switch:

    cumulus@switch:~$ net add loopback lo ip address 10.2.1.1/32
    cumulus@switch:~$ net add interface swp1 ip address 10.2.1.1/32
    cumulus@switch:~$ net add interface swp2 ip address 10.2.1.1/32
    cumulus@switch:~$ net add interface swp3 ip address 10.4.1.1/24
    cumulus@switch:~$ net add interface swp1 ospf network point-to-point
    cumulus@switch:~$ net add interface swp2 ospf network point-to-point
    cumulus@switch:~$ net add ospf router-id 10.2.1.1
    cumulus@switch:~$ net add ospf network 10.2.1.1/32 area 0.0.0.0
    cumulus@switch:~$ net add ospf network 10.4.1.0/24 area 0.0.0.0
    cumulus@switch:~$ net pending
    cumulus@switch:~$ net commit

    These commands configure both /etc/network/interfaces and /etc/frr/frr.conf. The output of each file is shown below.

    To edit the configuration files directly as the sudo user, copy the configurations below.

    /etc/network/interfaces
    # The loopback network interface
    auto lo
      iface lo inet loopback
      address 10.2.1.1/32
    
    # The primary network interface
    auto eth0
      iface eth0 inet dhcp
    
    auto swp1
      iface swp1
      address 10.2.1.1/32
    
    auto swp2
      iface swp2
      address 10.2.1.1/32
    
    auto swp3
      iface swp3
      address 10.4.1.1/24
    /etc/frr/frr.conf
    service integrated-vtysh-config
    
    interface swp1
      ip ospf network point-to-point
    
    interface swp2
      ip ospf network point-to-point
    
    router-id 10.2.1.1
    
    router ospf
      ospf router-id 10.2.1.1
      network 10.2.1.1/32 area 0.0.0.0
      network 10.4.1.0/24 area 0.0.0.0
  4. Restart the networking service:

    cumulus@switch:~$ sudo systemctl restart networking
  5. Restart FRRouting:

    cumulus@switch:~$ sudo systemctl restart frr.service

Configure the Remaining VMs

The configuration steps for CumulusVX-leaf2, CumulusVX-spine1, and CumulusVX-spine2 are the same as CumulusVX-leaf1; however, the file configurations are different. Listed below are the configurations for each additional VM:

  • CumulusVX-leaf2:

    /etc/network/interfaces
    # The loopback network interface
    auto lo
      iface lo inet loopback
      address 10.2.1.2/32
    
    # The primary network interface
    auto eth0
      iface eth0 inet dhcp
    
    auto swp1
      iface swp1
      address 10.2.1.2/32
    
    auto swp2
      iface swp2
      address 10.2.1.2/32
    
    auto swp3
      iface swp3
      address 10.4.2.1/24
    /etc/frr/frr.conf
    service integrated-vtysh-config 
    
    interface swp1
      ip ospf network point-to-point
    
    interface swp2
      ip ospf network point-to-point
    
    router-id 10.2.1.2
    
    router ospf
      ospf router-id 10.2.1.2                                                           
      network 10.2.1.2/32 area 0.0.0.0  
      network 10.4.2.0/24 area 0.0.0.0
  • CumulusVX-spine1:

    /etc/network/interfaces
    # The loopback network interface
    auto lo
      iface lo inet loopback
      address 10.2.1.3/32
    
    # The primary network interface
    auto eth0
      iface eth0 inet dhcp
    
    auto swp1
      iface swp1
      address 10.2.1.3/32
    
    auto swp2
      iface swp2
      address 10.2.1.3/32
    
    auto swp3
      iface swp3
    /etc/frr/frr.conf
    service integrated-vtysh-config 
    
    interface swp1
      ip ospf network point-to-point
    
    interface swp2
      ip ospf network point-to-point
    
    router-id 10.2.1.3
    
    router ospf
      ospf router-id 10.2.1.3
      network 10.2.1.3/32 area 0.0.0.0
  • CumulusVX-spine2:

    /etc/network/interfaces
    # The loopback network interface
    auto lo
      iface lo inet loopback
      address 10.2.1.4/32
    
    # The primary network interface
    auto eth0
      iface eth0 inet dhcp
    
    auto swp1
      iface swp1
      address 10.2.1.4/32
    
    auto swp2
      iface swp2
      address 10.2.1.4/32
    
    auto swp3
      iface swp3
    /etc/frr/frr.conf
    service integrated-vtysh-config 
    
    interface swp1
      ip ospf network point-to-point
    
    interface swp2
      ip ospf network point-to-point
    
    router-id 10.2.1.4
    
    router ospf
      ospf router-id 10.2.1.4
      network 10.2.1.4/32 area 0.0.0.0

Restart the networking and FRRouting services on all VMs before continuing.

Create Point-to-Point Connections Between VMs

To use the two-leaf/two-spine Cumulus VX network topology you configured above, you need to configure the network adapter settings for each VM to create point-to-point connections. The following example shows how to create point-to-point connections between each VM in VirtualBox. If you are not using VirtualBox, refer to your hypervisor documentation to configure network adapter settings.

Follow these steps for each of the four VMs.

Make sure that the VM is powered off.

  1. In the VirtualBox Manager window, select the VM.
  2. Click Settings, then click Network.
  3. Click Adapter 2.
  4. Click the Enable Network Adapter check box.

  5. From the Attached to list, select Internal Network.

  6. In the Name field, type a name for the internal network, then click OK.
    The internal network name must match the internal network name on the corresponding network adapter on the VM to be connected to this VM. For example, in the two-leaf/two-spine Cumulus VX network topology, Adapter 2 (swp1) on CumulusVX-leaf1 is connected to Adapter 2 (swp1) on CumulusVX-spine1; the name must be the same for Adapter 2 on both VMs. Use the internal network names and the connections shown in the illustration and table below.

  7. Click Adapter 3 and repeat steps 4 thru 6. Use the internal network names and the connections shown in the illustration and table below.

SwitchswpVirtualBox InterfaceVirtualBox Network TypeName
CumulusVX-leaf1 Adapter 1NAT 
 swp1Adapter 2InternalIntnet-1
 swp2Adapter 3InternalIntnet-3
 swp3Adapter 4InternalIntnet-5
CumulusVX-leaf2 Adapter 1NAT 
 swp1Adapter 2InternalIntnet-2
 swp2Adapter 3InternalIntnet-4
 swp3Adapter 4InternalIntnet-6
CumulusVX-spine1 Adapter 1NAT 
 swp1Adapter 2InternalIntnet-1
 swp2Adapter 3InternalIntnet-2
 swp3

Adapter 4 (disabled)

  
CumulusVX-spine2 Adapter 1NAT 
 swp1Adapter 2InternalIntnet-3
 swp2Adapter 3InternalIntnet-4
 swp3

Adapter 4 (disabled)

  

Test the Network Topology Connections

After you restart the VMs, ping across VMs to test the connections:

  1. Run the following commands from CumulusVX-leaf1:

    • Ping CumulusVX-leaf2:

      cumulus@CumulusVX-leaf1:~$ ping 10.2.1.2
    • Ping CumulusVX-spine1:

      cumulus@CumulusVX-leaf1:~$ ping 10.2.1.3
    • Ping CumulusVX-spine2:

      cumulus@CumulusVX-leaf1:~$ ping 10.2.1.4
  • No labels

12 Comments

  1. Anonymous

    Generally speaking, assigning the same IP address to multiple nodes on the same network is a bad idea. But here you are doing it:

    cumulus@switch:~$ net add loopback lo ip address 10.2.1.1/32

    cumulus@switch:~$ net add interface swp1 ip address 10.2.1.1/32
    cumulus@switch:~$ net add interface swp2 ip address 10.2.1.1/32
    So what is it that you have not explained sufficiently well that makes this a good thing?
    1. Hey Anon, the configuration provided here is a technology called unnumbered IP addressing, where we use the same /32 on multiple ports. When doing OSPF unnumbered, there isn't an RFC equivalent to RFC5549 so we need to use IPv4 address to bring up the OSPF adj, however we can reuse the same IP.  Generally, more people tend to use BGP unnumbered (RFC5549) which uses IPv6 link-locals (thus negating the use for IPv4 /32s configured on every port). Here is a KB article on OSPF configuration: https://support.cumulusnetworks.com/hc/en-us/articles/202796476-OSPF-Unnumbered-Sample-Configurations

    2. Anonymous

      I just configured this step by step on my machine. It ended up thrashing itself to death due to OSPF recalculations. The adjacencies also seemed to be very wrong. Leaf 1 & 2 became peers instead of leaf{1,2} and spine{1,2}. The OSPF thrashing came from the flapping OSPF adjacencies. Based on the configuration from the Introduction all of these interfaces are on the same broadcast domain. Wouldn't you have to set up all of these interface pairs on their own separate networks to ensure they're truly point-to-point?

      1. Anonymous

        Btw I'm not the original Anon

        Regards,

        Jason Leschnik

      2. Anonymous

        I was following the getting started guide verbatim (this guide + VMware Workstation), after configuring each of the links on their own vmnet it works a treat. Is there a way to make this pitfall a little more apparent in the documentation for future travellers?

        Regards,

        Jason Leschnik

         

  2. Anonymous

    Agree with Jason L above. The instructions need editing. I used VirtualBox and all links were mapped to the same "internal network". I then followed instructions here:

    http://www.brianlinkletter.com/how-to-use-virtualbox-to-emulate-a-network/

    That helped get on the right track (i.e. Leaf1 connected to Spines, and not to Leaf2 etc.).

     

    Please fix the documentation for VirtualBox users (and other VM users).

     

    Thanks

    Bala

  3. Anonymous

    Hi, i just tried and found issue on OSPF is always back to init after a while. I don't know how to see OSPF log. Seems like I need to create dedicated VMNetwork for each spine to leaf. 

    But, is there any KB linking to the logging part?

    And also .. so, here in Cumulus we used OSPF and ECMP, combined with EVPN to enable the 'web-scale'?

    1. You should try asking this on the forums as you're more likely to get an answer there. https://forums.cumulusnetworks.com

  4. Anonymous

    when I do 'net add interface swp3 ip address 10.4.1.1/24' on leaf1, it will return 'Error: swp3 is not a physical interface on this switch'.

     I check 'Get started' and find no configuration about swp3, why?

    1. Thanks for asking this on the forums.

  5. Anonymous

  6. Anonymous

    My best guess is, since you're using p2p connectivity here between all the switches, the IP unnumbered is a good way to get things done, since in p2p scenarios, subnets aren't really a thing. I can be more detailed once I do implement this guide in a VM environment and comment on if it works or not, and why doesn't it, if it doesn't.

     

Write a comment…