1、DPDK

        DPDK (Data Plane Development Kit)是由6WIND，Intel等多家公司开发，主要基于Linux系统运行，用于快速数据包处理的函数库与驱动集合，可以极大提高数据处理性能和吞吐量，提高数据平面应用程序的工作效率。

2、环境配套信息

VMware Workstation 15 Pro
Ubuntu 18.04.1
DPDK 19.08.2

3、VMware配置

添加2张网卡：

 配置网卡支持多队列：

处理器数量需要大于1

 修改虚拟机配置

修改配置文件：

ethernet0.virtualDev = "e1000"

改为：
ethernet0.virtualDev = "vmxnet3"

确认配置生效：

 4、DPDK源码下载

 DPDK官网：dpdk-19.08.2

 CSDN:         dpdk-19.08.2

 5、系统环境准备

• 依赖工具安装

sudo apt install build-essential

sudo apt-get install libnuma-dev

• 配置大页内存，开启immon功能

sudo vim /etc/default/grub 

在GRUB_CMDLINE_LINUX末尾追加：hugepages=512 intel_iommu=on iommu=pt
GRUB_CMDLINE_LINUX="find_preseed=/preseed.cfg auto noprompt priority=critical locale=en_US hugepages=512 intel_iommu=on iommu=pt"

• 重新生成引导配置

sudo grub-mkconfig -o /boot/grub/grub.cfg

• 重启并验证

cat /proc/meminfo |grep -i HugePages
dmesg | grep DMAR

 6、编译DPDK 

• 设置环境变量

cd dpdk-stable-19.08.2
export RTE_SDK=`pwd`  
export RTE_TARGET=x86_64-default-linuxapp-gcc 

• 运行dpdk-setup.sh 编译

sunny@ubuntu:~/dpdk-stable-19.08.2$ cd usertools/
sunny@ubuntu:~/dpdk-stable-19.08.2/usertools$ ./dpdk-setup.sh 
------------------------------------------------------------------------------
 RTE_SDK exported as /home/sunny/dpdk-stable-19.08.2
------------------------------------------------------------------------------
----------------------------------------------------------
 Step 1: Select the DPDK environment to build
----------------------------------------------------------
[1] arm64-armada-linuxapp-gcc
[2] arm64-armada-linux-gcc
[3] arm64-armv8a-linuxapp-clang
[4] arm64-armv8a-linuxapp-gcc
[5] arm64-armv8a-linux-clang
[6] arm64-armv8a-linux-gcc
[7] arm64-bluefield-linuxapp-gcc
[8] arm64-bluefield-linux-gcc
[9] arm64-dpaa2-linuxapp-gcc
[10] arm64-dpaa2-linux-gcc
[11] arm64-dpaa-linuxapp-gcc
[12] arm64-dpaa-linux-gcc
[13] arm64-octeontx2-linuxapp-gcc
[14] arm64-octeontx2-linux-gcc
[15] arm64-stingray-linuxapp-gcc
[16] arm64-stingray-linux-gcc
[17] arm64-thunderx2-linuxapp-gcc
[18] arm64-thunderx2-linux-gcc
[19] arm64-thunderx-linuxapp-gcc
[20] arm64-thunderx-linux-gcc
[21] arm64-xgene1-linuxapp-gcc
[22] arm64-xgene1-linux-gcc
[23] arm-armv7a-linuxapp-gcc
[24] arm-armv7a-linux-gcc
[25] i686-native-linuxapp-gcc
[26] i686-native-linuxapp-icc
[27] i686-native-linux-gcc
[28] i686-native-linux-icc
[29] ppc_64-power8-linuxapp-gcc
[30] ppc_64-power8-linux-gcc
[31] x86_64-native-bsdapp-clang
[32] x86_64-native-bsdapp-gcc
[33] x86_64-native-freebsd-clang
[34] x86_64-native-freebsd-gcc
[35] x86_64-native-linuxapp-clang
[36] x86_64-native-linuxapp-gcc
[37] x86_64-native-linuxapp-icc
[38] x86_64-native-linux-clang
[39] x86_64-native-linux-gcc
[40] x86_64-native-linux-icc
[41] x86_x32-native-linuxapp-gcc
[42] x86_x32-native-linux-gcc

----------------------------------------------------------
 Step 2: Setup linux environment
----------------------------------------------------------
[43] Insert IGB UIO module
[44] Insert VFIO module
[45] Insert KNI module
[46] Setup hugepage mappings for non-NUMA systems
[47] Setup hugepage mappings for NUMA systems
[48] Display current Ethernet/Baseband/Crypto device settings
[49] Bind Ethernet/Baseband/Crypto device to IGB UIO module
[50] Bind Ethernet/Baseband/Crypto device to VFIO module
[51] Setup VFIO permissions

----------------------------------------------------------
 Step 3: Run test application for linux environment
----------------------------------------------------------
[52] Run test application ($RTE_TARGET/app/test)
[53] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)

----------------------------------------------------------
 Step 4: Other tools
----------------------------------------------------------
[54] List hugepage info from /proc/meminfo

----------------------------------------------------------
 Step 5: Uninstall and system cleanup
----------------------------------------------------------
[55] Unbind devices from IGB UIO or VFIO driver
[56] Remove IGB UIO module
[57] Remove VFIO module
[58] Remove KNI module
[59] Remove hugepage mappings

[60] Exit Script

Option: 

• 版本选择，编译36

[36] x86_64-native-linuxapp-gcc
Option: 36

Configuration done using x86_64-native-linuxapp-gcc
== Build lib
== Build lib/librte_kvargs
== Build lib/librte_eal
/home/sunny/dpdk-stable-19.08.2/mk/toolchain/gcc/rte.toolchain-compat.mk:30: AVX512 support disabled because of binutils 2.30. See Bug 97
== Build lib/librte_eal/common
== Build lib/librte_eal/linux
== Build lib/librte_eal/linux/eal
  CC eal_memory.o
  CC eal_thread.o
  CC eal_log.o
  CC eal_vfio.o
  CC eal_vfio_mp_sync.o
  CC eal_memalloc.o
  CC eal_debug.o
  CC eal_lcore.o
  CC eal_timer.o

• 加载驱动43

[43] Insert IGB UIO module

Option: 43

Unloading any existing DPDK UIO module
Loading uio module
Loading DPDK UIO module

• 配置大页内存46

[46] Setup hugepage mappings for non-NUMA systems

Option: 46

Removing currently reserved hugepages
Unmounting /mnt/huge and removing directory

  Input the number of 2048kB hugepages
  Example: to have 128MB of hugepages available in a 2MB huge page system,
  enter '64' to reserve 64 * 2MB pages
Number of pages: 512
Reserving hugepages
Creating /mnt/huge and mounting as hugetlbfs

• 查看网卡状态48，0000:03:00.0即网卡PCI地址

[48] Display current Ethernet/Baseband/Crypto device settings

Option: 48

Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3,vfio-pci

Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio,vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

• 绑定网卡49

需要先将网卡down:
sunny@ubuntu:~/dpdk-stable-19.08.2/usertools$ ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.160  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::7074:2a70:8ce9:db20  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:71:e5:61  txqueuelen 1000  (Ethernet)
        RX packets 1122  bytes 148489 (148.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 202  bytes 16795 (16.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.144  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::d784:b54e:4161:626c  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:71:e5:57  txqueuelen 1000  (Ethernet)
        RX packets 68354  bytes 100105014 (100.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25950  bytes 1935742 (1.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 403  bytes 35535 (35.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 403  bytes 35535 (35.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

sunny@ubuntu:~/dpdk-stable-19.08.2/usertools$ ifconfig ens160 down

[49] Bind Ethernet/Baseband/Crypto device to IGB UIO module

Option: 49


Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=igb_uio,vfio-pci *Active*
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens160 drv=vmxnet3 unused=igb_uio,vfio-pci 

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

Enter PCI address of device to bind to IGB UIO driver: 03:00.0
OK

• 运行测试程序53

[53] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)

Option: 53

  Enter hex bitmask of cores to execute testpmd app on
  Example: to execute app on cores 0 to 7, enter 0xff
bitmask: 7
Launching app
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:01.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15ad:7b0 net_vmxnet3
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
Port 0: 00:0C:29:71:E5:57
Checking link statuses...
Done
testpmd> start
io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=1
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
      RX desc=0 - RX free threshold=0
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=0 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=0
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 335            RX-dropped: 0             RX-total: 335
  TX-packets: 335            TX-dropped: 0             TX-total: 335
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 335            RX-dropped: 0             RX-total: 335
  TX-packets: 335            TX-dropped: 0             TX-total: 335
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

• 查看大页内存使用情况

sunny@ubuntu:~/dpdk-stable-19.08.2/usertools$ cat /proc/meminfo |grep -i HugePages
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:     512
HugePages_Free:      254
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

• 环境搭建完成，60退出

[60] Exit Script

Option: 60

•  以上过程中可能会出现各种错误，需要逐步分析解决

 7、使用DPDK Kit

• 编译完成Kit位置

sunny@ubuntu:~/dpdk-stable-19.08.2$ pwd
/home/sunny/dpdk-stable-19.08.2
sunny@ubuntu:~/dpdk-stable-19.08.2$ ls
app                       config    drivers      kernel   MAINTAINERS  meson_options.txt  usertools
arm64-armv8a-linux-clang  devtools  examples     lib      Makefile     mk                 VERSION
buildtools                doc       GNUmakefile  license  meson.build  README             x86_64-native-linuxapp-gcc
sunny@ubuntu:~/dpdk-stable-19.08.2$ cd x86_64-native-linuxapp-gcc/
sunny@ubuntu:~/dpdk-stable-19.08.2/x86_64-native-linuxapp-gcc$ ls
app  build  include  kmod  lib  Makefile

• Kit组件

sunny@ubuntu:~/dpdk-stable-19.08.2/x86_64-native-linuxapp-gcc$ tree -d
.
├── app
├── build
│   ├── app
│   │   ├── pdump
│   │   ├── proc-info
│   │   ├── test
│   │   ├── test-acl
│   │   ├── test-bbdev
│   │   ├── test-cmdline
│   │   ├── test-compress-perf
│   │   ├── test-crypto-perf
│   │   ├── test-eventdev
│   │   ├── test-pipeline
│   │   └── test-pmd
│   ├── buildtools
│   │   └── pmdinfogen
│   ├── drivers
│   │   ├── baseband
│   │   │   ├── fpga_lte_fec
│   │   │   ├── null
│   │   │   └── turbo_sw
│   │   ├── bus
│   │   │   ├── dpaa
│   │   │   │   └── base
│   │   │   │       ├── fman
│   │   │   │       └── qbman
│   │   │   ├── fslmc
│   │   │   │   ├── mc
│   │   │   │   ├── portal
│   │   │   │   └── qbman
│   │   │   ├── ifpga
│   │   │   ├── pci
│   │   │   │   └── linux
│   │   │   ├── vdev
│   │   │   └── vmbus
│   │   │       └── linux
│   │   ├── common
│   │   │   ├── cpt
│   │   │   ├── dpaax
│   │   │   ├── octeontx
│   │   │   ├── octeontx2
│   │   │   └── qat
│   │   ├── compress
│   │   │   └── octeontx
│   │   ├── crypto
│   │   │   ├── caam_jr
│   │   │   ├── dpaa2_sec
│   │   │   │   └── mc
│   │   │   ├── dpaa_sec
│   │   │   ├── null
│   │   │   ├── octeontx
│   │   │   ├── scheduler
│   │   │   └── virtio
│   │   ├── event
│   │   │   ├── dpaa
│   │   │   ├── dpaa2
│   │   │   ├── dsw
│   │   │   ├── octeontx
│   │   │   ├── octeontx2
│   │   │   ├── opdl
│   │   │   ├── skeleton
│   │   │   └── sw
│   │   ├── mempool
│   │   │   ├── bucket
│   │   │   ├── dpaa
│   │   │   ├── dpaa2
│   │   │   ├── octeontx
│   │   │   ├── octeontx2
│   │   │   ├── ring
│   │   │   └── stack
│   │   ├── net
│   │   │   ├── af_packet
│   │   │   ├── ark
│   │   │   ├── atlantic
│   │   │   ├── avp
│   │   │   ├── axgbe
│   │   │   ├── bnxt
│   │   │   ├── bonding
│   │   │   ├── cxgbe
│   │   │   ├── dpaa
│   │   │   ├── dpaa2
│   │   │   │   ├── base
│   │   │   │   └── mc
│   │   │   ├── e1000
│   │   │   ├── ena
│   │   │   ├── enetc
│   │   │   ├── enic
│   │   │   │   └── base
│   │   │   ├── failsafe
│   │   │   ├── fm10k
│   │   │   ├── hinic
│   │   │   ├── i40e
│   │   │   ├── iavf
│   │   │   ├── ice
│   │   │   ├── ifc
│   │   │   ├── ipn3ke
│   │   │   ├── ixgbe
│   │   │   ├── kni
│   │   │   ├── liquidio
│   │   │   ├── memif
│   │   │   ├── netvsc
│   │   │   ├── nfp
│   │   │   ├── null
│   │   │   ├── octeontx
│   │   │   ├── octeontx2
│   │   │   ├── qede
│   │   │   ├── ring
│   │   │   ├── sfc
│   │   │   ├── softnic
│   │   │   ├── tap
│   │   │   ├── thunderx
│   │   │   ├── vdev_netvsc
│   │   │   ├── vhost
│   │   │   ├── virtio
│   │   │   │   └── virtio_user
│   │   │   └── vmxnet3
│   │   └── raw
│   │       ├── dpaa2_cmdif
│   │       ├── dpaa2_qdma
│   │       ├── ifpga
│   │       ├── ioat
│   │       ├── ntb
│   │       ├── octeontx2_dma
│   │       └── skeleton
│   ├── kernel
│   │   └── linux
│   │       ├── igb_uio
│   │       └── kni
│   └── lib
│       ├── librte_acl
│       ├── librte_bbdev
│       ├── librte_bitratestats
│       ├── librte_bpf
│       ├── librte_cfgfile
│       ├── librte_cmdline
│       ├── librte_compressdev
│       ├── librte_cryptodev
│       ├── librte_distributor
│       ├── librte_eal
│       │   ├── common
│       │   └── linux
│       │       └── eal
│       ├── librte_efd
│       ├── librte_ethdev
│       ├── librte_eventdev
│       ├── librte_flow_classify
│       ├── librte_gro
│       ├── librte_gso
│       ├── librte_hash
│       ├── librte_ip_frag
│       ├── librte_ipsec
│       ├── librte_jobstats
│       ├── librte_kni
│       ├── librte_kvargs
│       ├── librte_latencystats
│       ├── librte_lpm
│       ├── librte_mbuf
│       ├── librte_member
│       ├── librte_mempool
│       ├── librte_meter
│       ├── librte_metrics
│       ├── librte_net
│       ├── librte_pci
│       ├── librte_pdump
│       ├── librte_pipeline
│       ├── librte_port
│       ├── librte_power
│       ├── librte_rawdev
│       ├── librte_rcu
│       ├── librte_reorder
│       ├── librte_ring
│       ├── librte_sched
│       ├── librte_security
│       ├── librte_stack
│       ├── librte_table
│       ├── librte_timer
│       └── librte_vhost
├── include
│   └── generic
├── kmod
└── lib

181 directories