Throughput Testing and Troubleshooting
First you need to understand the traffic profile. You need to gather the following information:
- What is the source/destination of the connection?
- Bare metal or VM?
- IP address
- MAC address
- CPU information
- What is the full traffic path of the connection?
- L1 connections
- L2 hops
- L3 hops
- What service, protocol and ports are being used for the connection?
- Non-standard ports
- Common throughput testing - HTTP, FTP, high port TCP
- How many concurrent connections have been established between the client and server?
- What is the average packet size?
- TCP specific questions:
- What is the window?
- Is there any packet reordering?
- Is there packet loss and what is the recovery mechanism?
- Are we using SACK or normal ACK?
- How much other traffic is going through the Cumulus Linux switch?
- What is the speed of the port?
Network Testing with Open Source Tools
Testing network performance requires generating traffic at various rates and measuring throughput, loss, latency, jitter, and so forth. This article covers multiple open source network test tools, which are available for free. Running the software is only half of the story however. Tuning the host to efficiently generate and receive network traffic as well as monitoring the hardware load is equally important. All of the tools explored here have different features, statistics reporting and operation.
Generating traffic at data rates above 1Gbps requires fairly powerful server hardware. Without going into the depths of computer architecture, the CPU, memory, PCI bus and network interface controller (NIC) all play a part in generating and receiving network data. Typically the bottleneck is the CPU core executing the client or server processes. The PCI bus may also constrain the performance depending on the PCI version. Finally, the NIC must be able to support the traffic attempting to be sent.
Many of the examples and tuning information were learned from Energy Sciences Network. They go into a lot more detail about the host tuning and the various capabilities of the different tools.
This is a complete rewrite of the original iPerf tool. It has many of the same features, although it does not yet do multicast. Below are some basic commands to get started, which are the same options available in version 2. For bandwidth testing, iPerf3 is preferred over iPerf1 or 2.
Cumulus Networks tested iPerf3 and identified some functionality issues on Debian Wheezy 7.9, while we noticed no issues on Debian Jessie 8.3. iPerf versions 1 and 2 work on Debian Wheezy.
Start the server on the default port for IPv4 (default):
Start the server in daemon mode:
iperf3 -s -D
Start the server on port 5003:
iperf3 -s -p 5003
Start using 0.5 second interval times:
iperf3 -s -i 0.5
Run a 30 second test, giving results every 1 second:
iperf3 -c <dst-ip> -i 1 -t 30
Using granular interval reporting counters can help provide greater information when examining why a specific throughput test is slow.
Run a test from remotehost to localhost:
iperf3 -c <dst-ip> -i 1 -t 20 -R
Run a test with 4 parallel streams, and with a 32M TCP buffer
iperf3 -c <dst-ip> -w 32M -P 4
Window size is very important for TCP tests. The TCP window dictates how many bytes and packets are allowed in flight before receiving an ACK. A small window may result in high speeds for a directly connected server and client, but low speeds for a server and client across a network with high latency between the client and server. As RTT increases, the time between packet and ACK increases, which results in more packets in flight. If the TCP window is too small to accommodate, the sender will scale back the speed at which the packets are sent in order to not exhaust the window. This becomes even more important when using jumbo frames.
Utilizing parallel sessions can help improve throughput
numbers when using load balancing features such as LACP bonded links,
MLAG switches or ECMP routing. In these traffic load balancing features,
flows are normally hashed using the source and destination IP address
and port. This means that a single flow will always be hashed to the
same physical link and path. In order to test the full utilization of
the traffic path, the testing methodology should utilize multiple IP
addresses and ports for the client and server connections. The
throughput results for each concurrent connection should be
added together to get the total bandwidth of the infrastructure. The
-P option creates multiple sessions with varying source ports which,
depending on the traffic hashing algorithm, can result in greater
Run a 200 Mbps UDP test:
iperf3 -c <dst-ip> -u -i 1 -b 200M
When testing with UDP, the
-b modifier can be used to set a
target throughput rate. Use
-b 0 for an unlimited bandwidth target.
For UDP, without setting this value the throughput values may be very
Many client/server NIC cards perform TCP checksum/header offloading. This means that these calculations are done on the NIC itself. But for UDP, header processing is performed in software. As a result, performance numbers between UDP and TCP may not be consistent.
iPerf3 features many optimizations that work well with modern server hardware. It also has new features to improve how statistics are measured.
Run the test for 2 seconds before collecting results, to allow for TCP
slowstart to finish. The
-O flag sets the Omit mode to a 2 second
delay, and the -i flag sets a ½ second report interval.
iperf3 -c <dst-ip> -O 2 -i 0.5
-Z flag enables the Zero Copy mode, which uses the
sendfile() system call. This uses much less CPU to put the data on the
iperf3 -c <dst-ip> -Z
Output the results in JSON format using the
-J flag for easy parsing
by most coding languages. This output is printed after the entire test
iperf3 -c <dst-ip> -J
Set the CPU affinity for the sender
(-A 2) or the sender, receiver
(-A 2,3), where the cores are numbered starting at 0. This has the
same effect as running
numactl -C 4 iperf3.
iperf3 -c <dst-ip> -A 2,3
Using granular interval reporting counters can help provide greater information when examining why a specific throughput test is slow.
-A x,yif you are using the
-Pflag creates multiple connections and the server side appears to balance the threads across all available cores.
As identified in the notes above, iPerf as a tool, in general, can give a vastly varied set of results depending on the settings. However, iPerf has some issues that make it difficult to use for performance testing on modern networks. Factors that matter in testing:
- TCP window size, controlled using the -w flag
- TCP vs UDP procotol, controlled using the -u modifier
- Target throughput rate, controlled using the -b modifier
- CPU utilization, controlled using the -Z modifier and the -A modifier
- Data time sets used for average calculations, controlled using -O modifier
- Concurrent connections on port, controlled using the -P modifier
By default iPerf uses TCP/UDP port 5201/5001 (depending on version) for
ports during transfer. This does not work too well for ECMP testing.
-P flag tells iPerf to spawn multiple client threads,
where the source port uses 5201 as well as a series of ephemeral ports.
But when using this method, it is also normal to see a lot of these
bind failed: Address already in use.
iPerf (version 2)
When installing iPerf from the distribution's repository, it is most likely version 2. This is a rather old piece of software, and is not really maintained anymore.
One unique feature of iPerf2 to take a look at is the ability to send multicast traffic. Here are the client and server commands to use:
iperf -c 126.96.36.199 -u -b 512k #source iperf -B 188.8.131.52 -su #receiver
This causes the server to listen on a group address, meaning it sends an IGMP report to the connected network device. On the client side, be sure to create a static route for the multicast range, with a next-hop toward the network devices under test (let's assume this is out from eth1).
ip route add 184.108.40.206/4 dev eth1
The results are inconsistent for UDP above 2Gbps. We recommend using
nuttcp or iPerf3 for high speed UDP testing. However, it is perfectly
fine for testing networks if the traffic rate is kept below 2Gbps. Above
that, the statistics are completely unreliable. Therefore, we are not
going to cover this as a viable performance test tool.
This tool evolved over the past 30 years, and has several useful features. It supports rate limiting, multiple parallel streams and timer based execution. It also includes IPv6, IPv4 multicast and the ability to set parameters, such as TCP MSS or TOS/DSCP bits.
Looking at the application's help file, it's easy to see that this tool has a lot of unique functionality. One thing to understand is there are client (transmitter/receiver) and server entities. The server settings can be controlled from the client command, for such things as TCP window size. Server processes automatically fork into the background and run until an error condition or are killed. The examples below demonstrate some of the basics.
Start the server on the default port (5000):
Start server on the default port for IPv6:
nuttcp -s -6
Start server on port 5003:
nuttcp -S -P 5003
Run a 10 second test, giving results every second:
nuttcp -i 1 <server>
Running longer tests require the
-T flag, where you can specify
duration like this:
-T30 = 30 sec, -T1m = 1 min, and -T1h = 1 hr
Run a 10 second test, setting the transmitter/receiver window as well as the server receive/transmit window size to 32KB. You can also set the window in Megabytes (-w5m) or Gigabytes (-w5g):
nuttcp -w32 -ws32 <server>
The default traffic uses TCP, but it can also be run as UDP with 8192 byte datagrams (default):
nuttcp -u <server>
Run a 10 second UDP test at 50Mbps, and report throughput and packet loss:
nuttcp -u -R50m <server>
Run a 10 second UDP test with 1200 byte datagrams:
nuttcp -u -l1200 <server>
Run a 10 second test in reverse (server → client), giving results every 1 second:
nuttcp -r -i 1 <server>
UDP Burst Mode
Starting with version 6.2.8,
nuttcp includes a burst mode for UDP,
which is useful for finding network paths constrained a lack of
buffering. To do this, set an instantaneous data rate with optional
packet burst. Specify the rates in Kbps (-R100), Mbps (-R5m), Gbps
(-R5g) or PPS (-R1000p). The burst is just a count of packets.
Send 300 Mbps of UDP at an instantaneous rate in bursts of 20 packets for 5 seconds:
nuttcp -u -Ri300m/20 -i 1 -T5 <server>
This test may show a small amount of loss. If you increase the burst size at the same instantaneous rate, it will increase the stress to the network buffering. This will progressively show more loss as the transit device overflows its buffers.
On 10GbE network with 9000 byte MTU, we can send 8972 byte UDP datagrams at an instantaneous rate of 300 Mbps, and bursting at 100 packets. This should be lossless:
nuttcp -l8972 -T30 -u -w4m -Ri300m/100 -i1 <server>
Again, increasing the burst value and retesting eventually shows where buffer exhaustion happens in the network:
10G UDP Testing
It definitely seems like
nuttcp the best tool for high speed UDP
testing. It took very little effort to create a full 10 Gbps traffic
flow using UDP, but it requires jumbo MTU size packets (9K):
nuttcp -l8972 -T30 -u -w4m -Ru -i1 <server>
One of the newer features introduced in version 7.1 is CPU affinity. This allows the user to specify on which core to execute the process.
Sending multiple streams, you can use CPU affinity binding to make sure both processes do not end up on the same CPU core. Be sure to start the servers on the intended destination ports. Also, using the statistics output identifier helps to label the results with an s1 or s2:
nuttcp -i1 -xc 2/2 -Is1 -u -Ru -l8972 -w4m -p 5500 <server> & \ nuttcp -i1 -xc 3/3 -Is2 -u -Ru -l8972 -w4m -p 5501 <server>
Bandwidth Controller (bwctl)
This tool is a wrapper for iPerf2, iPerf3 and
nuttcp. The documentation for this wrapper is available here.
The advantage of using
bwctl is that it is a centralized bandwidth testing tool that utilizes all the tools listed above to provide bandwidth test results. You can read the installation guide here.
Ostinato is an open source tool for generating packets. It operates primarily from a GUI, but is easily installed in a Linux desktop environment.
Although there is no CLI, it does have a Python API that you can use to generate packets.
Cumulus Linux Switch Outputs
When performing bandwidth testing, it is valuable to cross reference the results from the testing tool against various outputs on the Cumulus Linux switch.
Traffic Services on the Switch
Throughput levels may change based on what services are applied to the traffic. Ideally, all traffic should be hardware switched through the switch ASIC. But certain services applied to the traffic require the traffic to be punted from the ASIC to the switch CPU. Since the switch CPU is slower at processing and forwarding packets, the throughput may suffer from these features. Some features that may cause software processing of packets are:
- Encapsulation methodologies
It is also important to look at what general features are applied to the traffic:
- Layer 2 vs. layer 3 switching
- Netfilter ACLs for traffic
You should examine these two outputs to verify the throughput:
- ethtool -S swpX
This command is a wrapper for the interface counters provided in Linux.
The output of
ip -s link show up prints counters of all TX and RX
packets. But these native Linux commands have no way to clear counters.
cl-netstat command adds this functionality, and can track
individual connections under controlled environments. The example below
shows the results of an iPerf3 test run between a client and server
through the Cumulus Linux switch:
Client iPerf3 Results
root@CLIENT:~# iperf3 -c 10.1.1.100 Connecting to host 10.1.1.100, port 5201 [ 4] local 10.1.2.100 port 53984 connected to 10.1.1.100 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 980 MBytes 8.22 Gbits/sec 0 5.17 MBytes ... [ 4] 9.00-10.00 sec 951 MBytes 7.98 Gbits/sec 0 5.17 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 9.32 GBytes 8.01 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 9.29 GBytes 7.98 Gbits/sec receiver
cumulus@switch:~$ sudo cl-netstat Kernel Interface table Iface MTU Met RX_OK RX_ERR RX_DRP RX_OVR TX_OK TX_ERR TX_DRP TX_OVR Flg ------- ----- ----- ------- -------- -------- -------- ------- -------- -------- -------- ----- swp13 1500 0 62622 0 0 0 6904270 0 0 0 BMRU swp14 1500 0 6904271 0 0 0 62623 0 0 0 BMRU
In the above example, we can see that traffic has successfully flowed between swp13 and swp14 without dropping any packets.
Similarly, the output of
ethtool -S can be used to cross reference the
output obtained from
cumulus@switch:~$ sudo ethtool -S swp13 NIC statistics: HwIfInOctets: 5583156718 HwIfInUcastPkts: 5140969 HwIfInBcastPkts: 14 HwIfInMcastPkts: 9272 HwIfOutOctets: 210229661164 HwIfOutUcastPkts: 139033950 HwIfOutMcastPkts: 17657 HwIfOutBcastPkts: 8556 HwIfInDiscards: 0 HwIfInL3Drops: 0 HwIfInBufferDrops: 0 HwIfInAclDrops: 63 HwIfInDot3LengthErrors: 0 HwIfInErrors: 0 SoftInErrors: 0 SoftInDrops: 0 SoftInFrameErrors: 0 HwIfOutDiscards: 0 HwIfOutErrors: 0 HwIfOutQDrops: 0 HwIfOutNonQDrops: 0 cumulus@switch:~$ sudo ethtool -S swp14 NIC statistics: HwIfInOctets: 210229250162 HwIfInUcastPkts: 139051524 HwIfInBcastPkts: 5 HwIfInMcastPkts: 9230 HwIfOutOctets: 5583641007 HwIfOutUcastPkts: 5124014 HwIfOutMcastPkts: 17875 HwIfOutBcastPkts: 8711 HwIfInDiscards: 0 HwIfInL3Drops: 0 HwIfInBufferDrops: 0 HwIfInAclDrops: 54 HwIfInDot3LengthErrors: 0 HwIfInErrors: 0 SoftInErrors: 0 SoftInDrops: 0 SoftInFrameErrors: 0 HwIfOutDiscards: 0 HwIfOutErrors: 0 HwIfOutQDrops: 0 HwIfOutNonQDrops: 0
The counters in
ethtool are much larger than the output of
cl-netstat because the
ethtool counters cannot be cleared. The key
output to examine is to verify whether errors in receive or transmit are
The directionality of the counters will matter as well. Drops on ingress via the HwIfIn prefix indicate that traffic rate on receive is too fast. This generally indicates a bursty traffic or choke points further along the traffic path. Drops on egress via the HwIfOut prefix indicate a many-to-one problem, where multiple RX interfaces are trying to send traffic out a single TX interface. This is not always a bad result, as it indicates full bandwidth utilization by the interface.
If the counters under RX_ERR are increasing during the bandwidth test, there may be problems with packet forwarding through the switch. Here are some previously written articles that address this scenario: