Address Resolution Protocol - ARP
Address Resolution Protocol (ARP) is a communication protocol used for discovering the link layer address, such as a MAC address, associated with a given network layer address. ARP is defined by RFC 826.
The Cumulus Linux ARP implementation differs from standard Debian Linux ARP behavior in a few ways because Cumulus Linux is an operating system for routers/switches rather than servers. This chapter describes the differences in ARP behavior, why the changes were made, where the changes were implemented, and how to change port-specific values.
Standard Debian ARP Behavior and the Tunable ARP Parameters
Debian has these five tunable ARP parameters:
These parameters are described in the Linux documentation, but snippets for each parameter description are included in the table below and are highlighted in italics.
In a standard Debian installation, all of these ARP parameters are set to 0, leaving the router as wide open and unrestricted as possible. These settings are based on the assertion made long ago that Linux IP addresses are a property of the device, not a property of an individual interface. Thus an ARP request or reply could be sent on one interface containing an address residing on a different interface. While this unrestricted behavior makes sense for a server, it is not the normal behavior of a router. Routers expect the MAC/IP address mappings supplied by ARP to match the physical topology, with the IP addresses matching the interfaces on which they reside. With these tunable ARP parameters, Cumulus Linux has been able to specify the behavior to match the expectations of a router.
ARP Tunable Parameter Settings in Cumulus Linux
The ARP tunable parameters are set to the following values by default in Cumulus Linux. Each parameter is described in detail, including why Cumulus Networks chose the value used.
Define behavior for gratuitous ARP frames whose IP is not already present in the ARP table:
0 - Don't create new entries in the ARP table.
1 - Create new entries in the ARP table.
Cumulus Linux uses the default
Define different restriction levels for announcing the local source IP address from IP packets in ARP requests sent on interface:
0 - (default) Use any local address, configured on any interface.
1 - Try to avoid local addresses that are not in the target's subnet for this interface. This mode is useful when target hosts reachable via this interface require the source IP address in ARP requests to be part of their logical network configured on the receiving interface. When we generate the request we will check all our subnets that include the target IP and will preserve the source address if it is from such subnet. If there is no such subnet we select source address according to the rules for level 2.
2 - Always use the best local address for this target. In this mode we ignore the source address in the IP packet and try to select local address that we prefer for talks with the target host. Such local address is selected by looking for primary IP addresses on all our subnets on the outgoing interface that include the target IP address. If no suitable local address is found we select the first local address we have on the outgoing interface or on all other interfaces, with the hope we will receive reply for our request and even sometimes no matter the source IP address we announce.
The default Debian behavior with
Routers expect a different relationship between the IP address and the physical network. Adjoining routers look for MAC/IP addresses to reach a next-hop residing on a connecting interface for transiting traffic. By setting the
0 - (default) The kernel can respond to ARP requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces. Only for more complex setups like load- balancing, does this behavior cause problems.
1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP'd IP address out of that interface (therefore you must use source based routing for this to work). In other words, it allows control of which cards (usually 1) will respond to an ARP request.
Cumulus Linux uses the default Debian Linux arp_filter setting of 0.
Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses:
0 - (default) Reply for any local target IP address, configured on any interface.
1 - Reply only if the target IP address is local address configured on the incoming interface.
2 - Reply only if the target IP address is local address configured on the incoming interface and both with the sender's IP address are part from same subnet on this interface.
3 - Do not reply for local addresses configured with scope host, only resolutions for global and link addresses are replied.
4-7 - Reserved
8 - Do not reply for all local addresses.
The maximum value from
The default Debian
For example, if the
In Cumulus Linux, the
Define mode for notification of address and device changes.
0 - (default) Do nothing.
1 - Generate gratuitous arp requests when device is brought up or hardware address changes.
The default Debian
Change Tunable ARP Parameters
You can change the ARP parameter settings in several places, including:
/proc/sys/net/ipv4/conf/default/arp*(default for future interfaces)
The ARP parameter changes in Cumulus Linux use the default file locations.
The all and default locations sound similar, with the exception of which interfaces are impacted, but they operate in significantly different ways. The all location can potentially change the value for all interfaces running IP, both now and in the future. The reason for this uncertainty is that the all value is applied to each parameter using either MAX or OR logic between the all and any port-specific settings, as the following table shows:
For example, if the
/proc/sys/net/conf/all/arp_ignore value is set to 1 and the
/proc/sys/net/conf/swp1/arp_ignore value is set to 0, to try to disable it on a per-port basis, interface swp1 still uses the value of 1 in its operation. While it may appear that the port-specific setting should override the global all setting, it does not actually work that way. Instead, the MAX value between the all value and port-specific value defines the actual behavior. This lack of simplicity has led Cumulus Networks to implement the ARP parameter changes using the default location instead.
The default location
/proc/sys/net/ipv4/conf/default/arp* defines the values for all future IP interfaces. Changing the default setting of an ARP parameter does not impact interfaces that already contain an IP address. If changes are being made to a running system that already has IP addresses assigned to it, port-specific settings should be used instead.
The way the default setting is implemented in Linux, the value of the default parameter is copied to every port-specific location, excluding those that already have an IP address assigned, as previously mentioned. Therefore there is not any complicated logic between the default setting and the port-specific setting like there is when using the all location. This makes the application of particular port-specific policies much simpler and more deterministic.
To determine the current ARP parameter settings for each of the the locations, use the following mechanism; other methods are available but this one is quite simple:
cumulus@switch:~$ sudo grep . /proc/sys/net/ipv4/conf/all/arp* /proc/sys/net/ipv4/conf/all/arp_accept:0 /proc/sys/net/ipv4/conf/all/arp_announce:0 /proc/sys/net/ipv4/conf/all/arp_filter:0 /proc/sys/net/ipv4/conf/all/arp_ignore:0 /proc/sys/net/ipv4/conf/all/arp_notify:0 cumulus@switch:~$ sudo grep . /proc/sys/net/ipv4/conf/default/arp* /proc/sys/net/ipv4/conf/default/arp_accept:0 /proc/sys/net/ipv4/conf/default/arp_announce:2 /proc/sys/net/ipv4/conf/default/arp_filter:0 /proc/sys/net/ipv4/conf/default/arp_ignore:1 /proc/sys/net/ipv4/conf/default/arp_notify:1 cumulus@switch:~$ sudo grep . /proc/sys/net/ipv4/conf/swp1/arp* /proc/sys/net/ipv4/conf/swp1/arp_accept:0 /proc/sys/net/ipv4/conf/swp1/arp_announce:2 /proc/sys/net/ipv4/conf/swp1/arp_filter:0 /proc/sys/net/ipv4/conf/swp1/arp_ignore:1 /proc/sys/net/ipv4/conf/swp1/arp_notify:1 cumulus@switch:~$
Note that Cumulus Linux implements this change at boot time using the
arp.conf file at the following location:
cumulus@switch:~$ cat /etc/sysctl.d/arp.conf net.ipv4.conf.default.arp_announce = 2 net.ipv4.conf.default.arp_notify = 1 net.ipv4.conf.default.arp_ignore=1 cumulus@switch:~$
Change Port-specific ARP Parameters
The simplest way to configure port-specific ARP parameters in a running device is with the following command:
cumulus@switch:~$ sudo sh -c "echo 0 > /proc/sys/net/ipv4/conf/swp1/arp_ignore" cumulus@switch:~$ sudo grep . /proc/sys/net/ipv4/conf/swp1/arp* /proc/sys/net/ipv4/conf/swp1/arp_accept:0 /proc/sys/net/ipv4/conf/swp1/arp_announce:2 /proc/sys/net/ipv4/conf/swp1/arp_filter:0 /proc/sys/net/ipv4/conf/swp1/arp_ignore:0 /proc/sys/net/ipv4/conf/swp1/arp_notify:1 cumulus@switch:~$
To make the change persist through reboots, edit the
/etc/sysctl.d/arp.conf file and add your port-specific ARP setting.
Configure Proxy ARP
The proxy ARP setting is a kernel setting that you can manipulate using
sysfs. Proxy ARP works with IPv4 only, since ARP is an IPv4-only protocol.
You need to set
/proc/sys/net/ipv4/conf/<INTERFACE>/proxy_arp to 1:
cumulus@switch:~$ net add interface swp1 post-up "echo 1 > /proc/sys/net/ipv4/conf/swp1/proxy_arp" cumulus@switch:~$ net pending cumulus@switch:~$ net commit
These commands create the following snippet in the
auto swp1 iface swp1 post-up echo 1 > /proc/sys/net/ipv4/conf/swp1/proxy_arp
If you’re running two interfaces in the same broadcast domain, which is typically seen when using VRR, as it creates a “-v0” interface in the same broadcast domain, make sure to use
sysfs to let the kernel know, so that both interfaces do not respond with proxy ARP replies. To do so, set
/proc/sys/net/ipv4/conf/<INTERFACE>/medium_id to 2 on both the interface and the -v0 interface. Continuing with the previous example:
cumulus@switch:~$ net add interface swp1 post-up "echo 2 > /proc/sys/net/ipv4/conf/swp1/medium_id" cumulus@switch:~$ net add interface swp1-v0 post-up "echo 1 > /proc/sys/net/ipv4/conf/swp1-v0/proxy_arp" cumulus@switch:~$ net add interface swp1-v0 post-up "echo 2 > /proc/sys/net/ipv4/conf/swp1-v0/medium_id" cumulus@switch:~$ net pending cumulus@switch:~$ net commit
These commands create the following snippet in the
auto swp1 iface swp1 post-up echo 1 > /proc/sys/net/ipv4/conf/swp1/proxy_arp post-up echo 2 > /proc/sys/net/ipv4/conf/swp1/medium_id auto swp1-v0 iface swp1-v0 post-up echo 1 > /proc/sys/net/ipv4/conf/swp1-v0/proxy_arp post-up echo 2 > /proc/sys/net/ipv4/conf/swp1-v0/medium_id
If you are running proxy ARP on a VRR interface, add a post-up line to the VRR interface stanza similar to the following. For example, if vlan100 is the VRR interface for the configuration above:
cumulus@switch:~$ net add vlan 100 post-up "echo 1 > /proc/sys/net/ipv4/conf/swp1/proxy_arp && echo 1 > /proc/sys/net/ipv4/conf/swp1-v0/proxy_arp && echo 2 > /proc/sys/net/ipv4/conf/swp1/medium_id && echo 2 > /proc/sys/net/ipv4/conf/swp1-v0/medium_id" cumulus@switch:~$ net pending cumulus@switch:~$ net commit
Duplicate Address Detection (Windows Hosts)
In centralized VXLAN environments, where ARP/ND suppression is enabled and SVIs exist on the leaf switches but are not assigned an address within the subnet, problems with the Duplicate Address Detection process on Microsoft Windows hosts can occur. For example, in a pure layer 2 scenario or with SVIs that have the
ip-forward option set to off, the IP address is not assigned to the SVI. The
neighmgrd service selects a source IP address for an ARP probe based on the subnet match on the neighbor IP address. Because the SVI on which this neighbor is learned does not contiain an IP address, the subnet match fails. This results in
neighmgrd using UNSPEC (0.0.0.0 for IPv4) as the source IP address in the ARP probe.
To work around this issue, run the
neighmgrctl setsrcipv4 <ipaddress> command to specify a non-0.0.0.0 address for the source; for example:
cumulus@switch:~$ neighmgrctl setsrcipv4 10.1.0.2
The configuration above takes effect immediately but does not persist if you reboot the switch. To make the changes apply persistently:
- Create a new file called
/etc/cumulus/neighmgr.confand add the
setsrcipv4 <ipaddress>option; for example:
cumulus@switch:~$ sudo nano /etc/cumulus/neighmgr.conf setsrcipv4: 10.1.0.2
- Reload the configuration file using
cumulus@switch:~$ sudo systemctl daemon-reload