How to run netmap on single queue and use another queues for Linux stack

Hello, folks!

I will do some ixgbe magic here! Please stay tuned =)

Here I could provide short reference about netmap compilation with patched ixgbe driver.

First of all you should get my patched driver. In this driver I have assigned first (0) queue to netmap and assigned another queues to Linux network stack.

Get driver sources and put they to "fake" Linux kernel source tree (netmap build system expect this):

cd /usr/src
mkdir -p /usr/src/fake_linux_kernel_sources/drivers/net/ethernet/intel
cd /usr/src/fake_linux_kernel_sources/drivers/net/ethernet/intel
git clone https://github.com/pavel-odintsov/ixgbe-linux-netmap-single-queue.git ixgbe_temp
mv ixgbe_temp/ixgbe-3.23.2.1/src/ ixgbe

Let's get netmap:

cd /usr/src
git clone https://github.com/luigirizzo/netmap.git -b next
cd netmap/LINUX/

Do some netmap patching:

sed -i 's/#define ixgbe_driver_name netmap_ixgbe_driver_name/\/\/\0/'  ixgbe_netmap_linux.h
sed -i 's/^char ixgbe_driver_name\[\]/\/\/\0/'  ixgbe_netmap_linux.h
sed -i '/$t\s\{1,\}if \[ \-f patches/d' configure

Let's compile it:

./configure --kernel-sources=/usr/src/fake_linux_kernel_sources --drivers=ixgbe
make

Unload old ixgbe not patched driver:

rmmod ixgbe

Load netmap:

insmod /usr/src/netmap/LINUX/netmap.ko
insmod /usr/src/netmap/LINUX/ixgbe/ixgbe.ko

Well, we have netmap which could process only first NIC hardware queue.

We should check it. For tests I will use flow director and flow all udp packets to 53 port to first queue:

ethtool -K eth5 ntuple on
ethtool --config-ntuple eth5 flow-type udp4 dst-port 53 action 0

Then we should built test environment.

Please compule test netmap receiver:

cd /usr/src/netmap/examples/
make

Yes, we are ready for tests!

Please run linux network stack receiver app in one console session:

tcpdump -n -i eth5

And netmap reciver app in another console session:

/usr/src/netmap/examples/pkt-gen -f rx -X -i netmap:eth5

Actually, we need external machine and please start pinging of target host from it and let's send udp packet to it from another session.

For udp packets generation you could use nc:

echo asdasda| nc -u 10.10.10.221  53

And you will saw:

 ./pkt-gen -f rx -i netmap:eth5 -X
689.290532 main [1651] interface is netmap:eth5
689.290977 extract_ip_range [288] range is 10.0.0.1:0 to 10.0.0.1:0
689.291015 extract_ip_range [288] range is 10.1.0.1:0 to 10.1.0.1:0
689.517212 main [1848] mapped 334980KB at 0x7fcf508ea000
Receiving from netmap:eth5: 1 queues, 1 threads and 1 cpus.
689.517331 main [1910] --- SPECIAL OPTIONS:

689.517345 main [1934] Wait 2 secs for phy reset
691.517508 main [1936] Ready...
691.517870 nm_open [456] overriding ifname eth5 ringid 0x0 flags 0x1
691.522007 receiver_body [1184] reading from netmap:eth5 fd 4 main_fd 3
692.523020 main_thread [1448] 0 pps (0 pkts in 1001104 usec)
692.525560 receiver_body [1191] waiting for initial packets, poll returns 0 0
693.524487 main_thread [1448] 0 pps (0 pkts in 1001468 usec)
693.528806 receiver_body [1191] waiting for initial packets, poll returns 0 0
694.525850 main_thread [1448] 0 pps (0 pkts in 1001363 usec)
694.532073 receiver_body [1191] waiting for initial packets, poll returns 0 0
695.526988 main_thread [1448] 0 pps (0 pkts in 1001137 usec)
695.535358 receiver_body [1191] waiting for initial packets, poll returns 0 0
696.528438 main_thread [1448] 0 pps (0 pkts in 1001450 usec)
696.538669 receiver_body [1191] waiting for initial packets, poll returns 0 0
697.529608 main_thread [1448] 0 pps (0 pkts in 1001170 usec)
697.542189 receiver_body [1191] waiting for initial packets, poll returns 0 0
698.530749 main_thread [1448] 0 pps (0 pkts in 1001141 usec)
698.545628 receiver_body [1191] waiting for initial packets, poll returns 0 0
699.531875 main_thread [1448] 0 pps (0 pkts in 1001126 usec)
699.549208 receiver_body [1191] waiting for initial packets, poll returns 0 0
700.532999 main_thread [1448] 0 pps (0 pkts in 1001124 usec)
700.552431 receiver_body [1191] waiting for initial packets, poll returns 0 0
ring 0x7fcf50954000 cur     0 [buf   4611 flags 0x0000 len    60]
    0: 90 e2 ba 83 3f 25 90 e2 ba 78 26 8d 08 00 45 00 ....?%...x&...E.
   16: 00 24 ce 85 40 00 40 11 43 49 0a 0a 0a 0a 0a 0a .$..@.@.CI......
   32: 0a dd ed 13 00 35 00 10 4f 47 61 73 64 61 73 64 .....5..OGasdasd
   48: 61 0a 00 00 00 00 00 00 00 00 00 00
701.534128 main_thread [1448] 1 pps (1 pkts in 1001129 usec)
702.535260 main_thread [1448] 0 pps (0 pkts in 1001132 usec)
703.536380 main_thread [1448] 0 pps (0 pkts in 1001120 usec)

And in tcpdump window:

tcpdump -n -i eth5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth5, link-type EN10MB (Ethernet), capture size 262144 bytes
08:01:36.520074 IP 10.10.10.10 > 10.10.10.221: ICMP echo request, id 6581, seq 21, length 64
08:01:36.520114 IP 10.10.10.221 > 10.10.10.10: ICMP echo reply, id 6581, seq 21, length 64
08:01:37.519971 IP 10.10.10.10 > 10.10.10.221: ICMP echo request, id 6581, seq 22, length 64
08:01:37.520009 IP 10.10.10.221 > 10.10.10.10: ICMP echo reply, id 6581, seq 22, length 64
08:01:38.520028 IP 10.10.10.10 > 10.10.10.221: ICMP echo request, id 6581, seq 23, length 64
08:01:38.520060 IP 10.10.10.221 > 10.10.10.10: ICMP echo reply, id 6581, seq 23, length 64
08:01:39.520091 IP 10.10.10.10 > 10.10.10.221: ICMP echo request, id 6581, seq 24, length 64
08:01:39.520130 IP 10.10.10.221 > 10.10.10.10: ICMP echo reply, id 6581, seq 24, length 64
08:01:40.520096 IP 10.10.10.10 > 10.10.10.221: ICMP echo request, id 6581, seq 25, length 64
08:01:40.520134 IP 10.10.10.221 > 10.10.10.10: ICMP echo reply, id 6581, seq 25, length 64
08:01:41.520030 IP 10.10.10.10 > 10.10.10.221: ICMP echo request, id 6581, seq 26, length 64
08:01:41.520064 IP 10.10.10.221 > 10.10.10.10: ICMP echo reply, id 6581, seq 26, length 64
08:01:42.520016 IP 10.10.10.10 > 10.10.10.221: ICMP echo request, id 6581, seq 27, length 64
08:01:42.520053 IP 10.10.10.221 > 10.10.10.10: ICMP echo reply, id 6581, seq 27, length 64
08:01:43.520086 IP 10.10.10.10 > 10.10.10.221: ICMP echo request, id 6581, seq 28, length 64
08:01:43.520125 IP 10.10.10.221 > 10.10.10.10: ICMP echo reply, id 6581, seq 28, length 64
^C
16 packets captured
16 packets received by filter
0 packets dropped by kernel

As you can see. Linux haven't saw UDP packets to 53 port but still process icmp packets. Everything works well! Hurra!

 Folks, be aware. This patch is very rude and not tested well. And we need some way for detaching this queue from Linux side because for ICMP packets case Linux try to send reply packets over detached queue. Haha, actually, we could do it very simple. We could disable tx queue detaching and use it for Linux.

And we need do some custom RING number tuning for ixgbe driver.

Finalyly, this approach working but need some enhancements :)