DPDK
DPDK, the Data Plane Development Toolkit, is an open source project of the Linux Foundation that provides offloading of TCP packet processing from system kernel interrupt-driven execution to the the user space, thereby significantly increasing performance.
Prerequisites
Required tools & libraries to build the kernel & DPDK:
libxml2-dev gcc g++ make meson bison flex ninja-build libssl-dev git bc cmake pkg-config libnuma-dev
Kernel
Download & build the the 4.19 kernel.
- Branch: whle-ls1-usdpaa
- Defconfig: whle_ls1_usdpaa_defconfig
There are two FDT files configured for USDPAA:
- freescale/fsl-ls1046a-whle-usdpaa.dtb
- freescale/fsl-ls1046a-whle-usdpaa-2.dtb
First one sets all of the ethernet/SFP ports up for userspace, second one leaves one ethernet port for kernel use.
Booting
Set bootargs in U-Boot to setup hugepages, DPAA manager portals, isolate Linux so it runs on a single core & leave the other cores for DPDK:
env set bootargs console=ttyS0,115200 root=/dev/mmcblk0p1 earlycon=uart8250,mmio,0x21c0500 default_hugepagesz=1024m hugepagesz=1024m hugepages=4 isolcpus=1-3 bportals=s0 qportals=s0 iommu.passthrough=1
run mmc_boot
Building DPDK
In order for DPDK to work correctly with WHLE-LS1 we will have to build it with the DPAA driver & the rte_kni kernel module, that means we have to supply the path to 4.19 kernel headers & source files for DPDK. Newer versions of DPDK use meson/ninja for the build process, so make sure these are available on your build machine.
Keep in mind DPDK expects a specific directory structure for the kernel headers/source in order for the build process to finish correctly:
- Export 4.19 kernel headers to a certain directory, which we're going to call KERNEL_PATH.
- At KERNEL_PATH create a symlink called build which links to the kernel source directory KERNEL_SRC.
- You'll need to do a full kernel build once before running a DPDK build, else rte_kni part will fail.
Expected KERNEL_PATH directory tree looks like this:
KERNEL_PATH:
- build -> KERNEL_SRC
- {Kernel headers}
Links:
Build on target device
meson arm64-build --cross-file config/arm/arm64_dpaa_linux_gcc -Denable_kmods=true -Dkernel_dir={KERNEL_PATH} -Dprefix=/usr
ninja -C arm64-build install
- KERNEL_PATH is the kernel headers directory, as explained above.
- By default DPDK builds with a debug-optimized build, which contains only partial debug information. If you want a full debug build, add -Dbuildtype=debug . Keep in mind this will drastically increase binary sizes up to 70MB+ for a single example app, so you might encounter space problems on the target device!
- Add -Dexamples=all to build all the example dpdk apps, or -Dexamples=l2fwd,ipsec-secgw,etc to build only specific ones.
Prerequisites for running DPDK applications
- Download dpdk-extras and put rte_kni.ko from the DPDK build folder in dpdk-extras/dpaa
- Install fmc either by:
- Using make install in dpdk-extras/fmc
- or build & install from source - Run dpdk-extras/dpaa/dpdk_env.sh
Example applications
We're assuming your device has at least one ethernet port configured in the FDT for userspace. If you're not using the fm1-mac9 and fm1-mac10 (SFP) ports, you'll most probably have to block them due to DPDK applications quitting abnormally if they're not connected. For testing purposes you might also want to create a virtual tap device.
Links:
l2fwd
This will run the L2 forwarding app setup to forward all packets between adjacent ports, e.g. 0 and 1.
dpdk-l2fwd -l 1,2,3 -n 1 -b "dpaa_bus:fm1-mac9" -b "dpaa_bus:fm1-mac10" --vdev="net_tap0,iface=dtap0" -- -p 0xff --portmap="(0,1)"
l2fwd-crypto
This is analogous to l2fwd with a crypto operation specified from the command line ran on each packet forwarded.
dpdk-l2fwd-crypto -l 1,2,3 -n 1 -b "dpaa_bus:fm1-mac9" -b "dpaa_bus:fm1-mac10" --vdev="net_tap0,iface=dtap0" -- -p 0xff --cdev_type HW --chain CIPHER_HASH --cipher_op ENCRYPT --cipher_algo aes-cbc --cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f --auth_op GENERATE --auth_algo sha1-hmac --auth_key 10:11:12:13:14:15:16:17:18:19:1a:1b:1c:1d:1e:1f
ipsec-secgw
dpdk-ipsec-secgw -l 1,2,3 -n 1 -b "dpaa_bus:fm1-mac9" -b "dpaa_bus:fm1-mac10" --vdev=net_tap0,iface=dtap0,mac="de:de:de:de:de:de" -- -P -f "ep0.cfg/ep1.cfg" -p 0xff --config="(0,0,1),(1,0,2)"
dpdk-ethtool
Allows you to query and set parameters of userspace controlled NICs.
dpdk-ethtool -l 1,2,3 -n 1 -b "dpaa_bus:fm1-mac9" -b "dpaa_bus:fm1-mac10"