Research checkpoint #08: a careful analysis on the Raspberry Pi sniffing tests
In the last two posts in this research context, I reported the results of different tests on a Raspberry Pi verifying its performance working with machine learning models and sniffing network packets. They were done with the intention of understanding what we can expect from an IoT device in terms of processing power, which is very limited compared to our usual computers. These tests are very important to delimit what we can expect these devices to be capable of and what is beyond their capabilities, guiding our future steps in research. In this sense, considering that sniffing packets from a live network is the basis of our studies to further try to classify them in benign and malign traffic, it’s worthy taking a closer look at the data collected in the last post, when we tested different Python libraries performing this activity.
A more careful analysis of the collected data
The tests were conducted evaluating three different Python libraries: Pyshark, Scapy and Pypcap together with dpkt, collecting the metrics in a Raspberry Pi 3 Model B and also using my personal notebook (an Acer Aspire 3 A315-41-R4RB) as a remote server, both in the same local network without internet access. Below I write about each of the observed metrics, the graphics represent the collected values and their standard deviation over five repetitions of each test scenario. All of the data that will be explored here and the detailed description of the tests conducted can be found in the post “Research checkpoint #07”.
To generate the network traffic in both the localhost (in which the Raspberry is both the client and server, represented as “LH” below) and remote server (in which the Raspberry is the client and my notebook the server, represented as “RS” below) scenarios I used the iperf3 application. For reference, here are the parameters reported by the application in each test1:
| Tests, with bitrate limit | iperf3 flow data |
|---|---|
| LH TCP (without limit) | 3.53 Gbits/sec, 4.11 GBytes |
| LH UDP 1Mb/sec | 1.02 Mbits/sec, 1.22 MBytes in 39 packets |
| LH UDP 2Mb/sec | 2.02 Mbits/sec, 2.41 MBytes in 77 packets |
| LH UDP 3Mb/sec | 3.01 Mbits/sec, 3.59 MBytes in 115 packets |
| LH UDP 4Mb/sec | 4.01 Mbits/sec, 4.78 MBytes in 153 packets |
| LH UDP 5Mb/sec | 5.01 Mbits/sec, 5.97 MBytes in 191 packets |
| LH UDP 10Mb/sec | 10.00 Mbits/sec, 11.9 MBytes in 382 packets |
| LH UDP (without limit) | 7.21 Gbits/sec, 8.4 GBytes in 275246 packets |
| RS TCP (without limit) | 94.64 Mbits/sec, 113 MBytes |
| RS UDP 1Mb/sec | 1 Mbits/sec, 1.19 MBytes in 864 packets |
| RS UDP 2Mb/sec | 2.00 Mbits/sec, 2.38 MBytes in 1727 packets |
| RS UDP 3Mb/sec | 3.00 Mbits/sec, 3.58 MBytes in 2590 packets |
| RS UDP 4Mb/sec | 4.00 Mbits/sec, 4.77 MBytes in 3453 packets |
| RS UDP 5Mb/sec | 5.00 Mbits/sec, 5.96 MBytes in 4316 packets |
| RS UDP 10Mb/sec | 10.0 Mbits/sec, 11.9 MBytes in 8632 packets |
| RS UDP (without limit) | 95.7 Mbits/sec, 114 MBytes in 82620 packets |
In the unlimited scenarios, the metrics reported by iperf3 differed slightly across the three different libraries. The one above is referent to the Pyshark test, for the other libs you can find the values in the mentioned post 07. Also, the metrics reported are the averages of the five runs of each test scenario (for example, the number of total packets varied slightly).
Use of CPU
Mean of the CPU usage over the 5 repetitions. Click on the image to expand it.
The first thing to notice in the graphic above is the difference between the minimum and maximum values reached by each of the Python libraries. At some point of the tests, we can observe that the libraries begin to present constant use of the processing power: for Pyshark this happens very fast, with about 2Mb/s in UDP remote server and 4Mb/s in UDP localhost, reaching ~80% of use; for Scapy this happens at 4Mb/s in UDP remote server, reaching ~20% of use; and for Pypcap + dpkt this threshold is uncertain only with the proposed test cases, but we can see that in the unlimited bitrate cases the use of CPU is close to ~20% also.
Pyshark reached its peak of CPU use very fast, even in the localhost scenario. We can even see that its minimum use of CPU (in the cases of 1Mb/s bitrate) is bigger than the maximum use achieved by the other two libraries. Pyshark is a great library to extract processed information from the packets, but has demonstrated that it isn’t a good choice for dealing with high volumes of traffic as also concluded in the last post, especially when processing power is a limited resource.
Scapy had similar CPU usage compared with Pypcap + dpkt in the scenarios with no bitrate limits, but the key difference is that this last one joint of libraries reaches this peak more slowly, indicating that they are better able to handle less intensive traffic. In a normal network, we don’t expect each device to be constantly dealing with high packet volumes, the default is for traffic to be well behaved, so this difference of performance dealing with a low number of data is very interesting to observe, as this will probably be the main scenario that the IoT device will be dealing with. The standard deviation isn’t really expressive, giving confidence in these metrics (partially! Note that I only took 5 samples to compute the average, which is still a low number).
Increase the bitrate of the traffic (and consequentially the number of packets) for beyond these thresholds doesn’t lead to significant differences, indicating that the number of packets in the network is a very influential parameter to the performance of the libraries but just until some point, after it the libraries starts to struggle to deal with the volume of packets (as we will see analyzing the number of packets counted). However, this is not the only parameter that affects the observations: the size of the packets seems to be also an influential factor. In the localhost scenarios, the number of packets per test is lower compared with the remote server one (look at the reference table in the beginning of the section), but the total number of bytes transmitted is similar, so the size of the packets in the localhost scenario is bigger. Scapy and Pypcap + dpkt seem to present more difficulty when dealing with the traffic when the number of packets increases, but for Pyshark the size of packets is already a problem, as we can see in the localhost metrics. This parameter will be more explored in the next post, verifying how it can affect the performances.
Memory peak
Peak of memory usage over the 5 repetitions. Click on the image to expand it.
In contrast with the last metric, the memory doesn’t present high variations between the test cases, the biggest difference can be observed when comparing localhost and remote server scenarios. This is justified: we are measuring the peak of memory and, as we discussed before, the localhost scenario has bigger packets than the other one, so the parsing of a single packet will take more memory in this first case, and also it’s reasonable to observe constant values over the tests.
The most noticeable thing is the Scapy memory usage. It uses more than twice the space needed by Pyshark and even more comparing with Pypcap + dpkt. Looking only for the numbers, it’s still not much memory (around 70MB) considering the computational power that we have at our disposal today, but we need to have in mind that this library will be running on an IoT device, which can be at some cases an embedded system with really few memory, where each byte makes difference and even the amount required by Pypcap + dpkt can be much. I think this isn’t a concern for now, but it’s something that we will need to come back at some point. Having these observations can be of great value for the future.
Number of packets counted
Mean of the number of packets counted over the 5 repetitions. Click on the image to expand it.
The first inevitable thing that catches our attention when looking to the graphic above is the disparity between Pypcap + dpkt and the other libraries. It’s even hard to compare them as this performatic joint of libraries seems to be on another scale, achieving numbers much more significant than the other two. To reduce this disparity, let’s take a look only in the UDP limited scenarios in the next graphic.
Just one observation before: you may notice that the number of packets counted here can be bigger than the number of packets reported in the reference table in the beginning of the post. This happens because the number reported in the table are only referent to the UDP flow generated during the test by iperf3. Before it, the application first establishes a TCP connection between the client and the server, referred as a “control connection”, in which the parameters are exchanged, and these packets end up being counted. So, it’s natural to have a bigger number here than in the table. However, notice that in the majority of the cases the number here is lower than the one reported there, specially in the unlimited bitrate cases, indicating a loss of packets in the count (by all the libraries!).
Mean of the number of packets counted over the 5 repetitions, considering only the UDP scenarios with limited bitrate of transmission. Click on the image to expand it.
The best test cases to look by is the remote server ones (RS): we can clearly see that the bars from the three different libraries grow together, but at some point Pyshark and Scapy stop improving while Pypcap + dpkt continue. As the number of packets in the traffic grows, the libraries performance are degraded, and in the same way that happened in the CPU usage, they don’t improve anymore after a threshold.
The localhost tests presented some strange behaviors on this metric observing the graphic above, not being a good reference for conclusions. Some analysis on it:
The number of packets counted by Scapy is clearly above the other libraries and for no clear reason. Investigating this behavior, Scapy is counting all the packets twice (using the number of packets counted by Pypcap + dpkt as reference, you can see that the number reported by Scapy is always close to the double of it) in the localhost test scenarios. I’m not sure of the reason for this, my suggestion is that Scapy is counting both the sent and received packets, registering each packet twice. This interface is very different from the normal ones and, as suggested by the Scapy’s FAQ, such weird behaviors are to be expected.
The number of packets registered by Pyshark is also above the number counted by Pypcap + dpkt (and half the number of Scapy). Looking at the traffic recorded by this library, there is always heavy traffic on port 5037 after sniffing starts, an unusual situation that doesn’t happen with any other library or in the remote server tests scenario. Investigating, this port is often associated with the Android Debug Bridge (adb) and tshark has a plugin (extcap) that can transform different packets sources in interfaces to identified by *shark applications, so it tries to make a connection to adb on localhost before the sniff starts. This behavior is reported in this termshark issue. In this way, every run in localhost with Pyshark has about 40 packets more than the other libraries.
Next steps
In the next posts, I will report more tests with the Raspberry in this context of packet parsing, analyzing tests varying the packet length and performing a simulation of a DDoS attack. Our current goals according to post 05:
- Packet Parsing
- Conduct experiments to verify the performance of Python libraries sniffing the network and processing the packages in a Raspberry Pi (different from what we did in post 4, in which we tested the lib’s with the CIC-IoT-2023 dataset and in a regular computer).
- Inference
- Run the baseline models proposed in post 02 in a Raspberry Pi to verify its performance.
- Training (centralized IDS)
- TBD.
This post was made as a record of the progress of the research project “DDoS Detection in the Internet of Things using Machine Learning Methods with Retraining”, supervised by professor Daniel Batista, BCC - IME - USP. Project supported by the São Paulo Research Foundation (FAPESP), process nº 2024/10240-3. The opinions, hypothesis and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of FAPESP.
The “M” and “G” in the units of size, such as “MBytes” and “GBytes”, refers to “Mebi” (1024² bytes) and “Gibi” (1024³ bytes), respectively. However, for the unity of data rate, such as “Mbits/sec” and “Gbits/sec”, the “M” and “G” refer to “Mega” (1000² bytes) and “Giga” (1000³ bytes), respectively. ↩