kevin laatz & ciara loftus fosdem 2020 · 2020-01-30 · af_xdp umem* vs dpdk mbuf pool chunk 0...
TRANSCRIPT
![Page 1: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/1.jpg)
Kevin Laatz & Ciara Loftus
FOSDEM 2020
![Page 2: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/2.jpg)
Introduction
https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf
https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
![Page 3: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/3.jpg)
Introduction
• Userspace Libraries and Drivers.• Accelerate packet processing workloads.• Runs on wide variety of CPU architectures.• Has its own memory management subsystem.• Device specific PMDs (Poll Mode Drivers).
https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf
https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
![Page 4: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/4.jpg)
Introduction
• Userspace Libraries and Drivers.• Accelerate packet processing workloads.• Runs on wide variety of CPU architectures.• Has its own memory management subsystem.• Device specific PMDs (Poll Mode Drivers).
AF_XDP• Kernel based address family.• Optimized for high performance packet processing.• AF_XDP sockets redirect packets to userspace.• By-passes kernel network stack
• “In-kernel fast path”.
https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf
https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
![Page 5: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/5.jpg)
Introduction
• Userspace Libraries and Drivers.• Accelerate packet processing workloads.• Runs on wide variety of CPU architectures.• Has its own memory management subsystem.• Device specific PMDs (Poll Mode Drivers).
AF_XDP• Kernel based address family.• Optimized for high performance packet processing.• AF_XDP sockets redirect packets to userspace.• By-passes kernel network stack
• “In-kernel fast path”.
Traditional DPDK Model
Userspace
DPDK apps
i40ePMD
ixgbe PMD
OtherPMDs
Kernel space
igb_uio vfio
https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf
https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
![Page 6: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/6.jpg)
Introduction
• Userspace Libraries and Drivers.• Accelerate packet processing workloads.• Runs on wide variety of CPU architectures.• Has its own memory management subsystem.• Device specific PMDs (Poll Mode Drivers).
AF_XDP• Kernel based address family.• Optimized for high performance packet processing.• AF_XDP sockets redirect packets to userspace.• By-passes kernel network stack
• “In-kernel fast path”.
Traditional DPDK Model
Userspace
DPDK apps
i40ePMD
ixgbe PMD
OtherPMDs
Kernel space
igb_uio vfio
DPDK with AF_XDP
Userspace
DPDK apps
DPDK AF_XDP PMD
Kernel space
NIC driver
https://www.dpdk.org/wp-content/uploads/sites/35/2019/07/14-AF_XDP-dpdk-summit-china-2019.pdf
https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwikxb2Z7pnnAhVNY8AKHchXDfAQjhx6BAgBEAI&url=https%3A%2F%2Fwww.cdrecycler.com%2Finterstate-waste-apex-merger.aspx&psig=AOvVaw0QFTtUxvONAgVjKXlMC_-v&ust=1579873577016036
![Page 7: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/7.jpg)
Problem Statement
![Page 8: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/8.jpg)
Problem Statement
• The Goal:
• All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD.
• DPDK app + AF_XDP PMD should run with good performance.
![Page 9: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/9.jpg)
Problem Statement
• The Goal:
• All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD.
• DPDK app + AF_XDP PMD should run with good performance.
• The Challenge:
• Frameworks like DPDK have their own memory management which come with their own assumptions and constraints.
![Page 10: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/10.jpg)
Problem Statement
• The Goal:
• All DPDK applications should be able to run out-of-the-box with the AF_XDP PMD.
• DPDK app + AF_XDP PMD should run with good performance.
• The Challenge:
• Frameworks like DPDK have their own memory management which come with their own assumptions and constraints.
• Discrepancy between the DPDK and AF_XDP buffer alignment.
• Prevents direct mapping of DPDK mempool to AF_XDP UMEM.
• Extra complexity/work negatively impacting performance.
![Page 11: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/11.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
![Page 12: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/12.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
• Area of memoryallocated by the userfor packet data
![Page 13: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/13.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
• Area of memoryallocated by the userfor packet data
• Split up into equalsized-chunks
![Page 14: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/14.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
• Area of memoryallocated by the userfor packet data
• Split up into equalsized-chunks
• RX Path: Kernel places packet data in a chunk for userspace(DPDK) to retrieve
![Page 15: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/15.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
• Area of memoryallocated by the userfor packet data
• Split up into equalsized-chunks
• RX Path: Kernel places packet data in a chunk for userspace(DPDK) to retrieve
• TX Path: Userspace places packet data in chunk for kernel NIC driver to transmit
![Page 16: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/16.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
*Prior to Kernel 5.4
![Page 17: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/17.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
*Prior to Kernel 5.4
Start address must be PAGE_SIZE aligned
PA
GE
_SIZ
E =
4K
![Page 18: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/18.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
*Prior to Kernel 5.4
Start address must be PAGE_SIZE aligned
Chunks must be ^2 sized
PA
GE
_SIZ
E =
4K
![Page 19: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/19.jpg)
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
Page 0
Page 1
Page 2
*Prior to Kernel 5.4
Start address must be PAGE_SIZE aligned
Chunks must be ^2 sized
Chunks may not cross page boundaries
PA
GE
_SIZ
E =
4K
![Page 20: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/20.jpg)
Start address must be PAGE_SIZE aligned
Chunks must be ^2 sized
Chunks may not cross page boundaries
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
DPDK MBUF 0
DPDK MBUF 1
DPDK MBUF 2
DPDK MBUF 3
Page 0
Page 1
Page 2
Page 3
*Prior to Kernel 5.4
![Page 21: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/21.jpg)
Start address must be PAGE_SIZE aligned
Chunks must be ^2 sized
Chunks may not cross page boundaries
Mbufs can be any size
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
DPDK MBUF 0
DPDK MBUF 1
DPDK MBUF 2
DPDK MBUF 3
Page 0
Page 1
Page 2
Page 3
*Prior to Kernel 5.4
![Page 22: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/22.jpg)
Start address must be PAGE_SIZE aligned
Chunks must be ^2 sized
Chunks may not cross page boundaries
Mbufs can be any size
Mbufs can have arbitrary alignment
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
DPDK MBUF 0
DPDK MBUF 1
DPDK MBUF 2
DPDK MBUF 3
Page 0
Page 1
Page 2
Page 3
*Prior to Kernel 5.4
![Page 23: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/23.jpg)
Start address must be PAGE_SIZE aligned
Chunks must be ^2 sized
Chunks may not cross page boundaries
Mbufs can be any size
Mbufs can have arbitrary alignment
AF_XDP UMEM* vs DPDK Mbuf Pool
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
DPDK MBUF 0
DPDK MBUF 1
DPDK MBUF 2
DPDK MBUF 3
Page 0
Page 1
Page 2
Page 3
To get the highest performing integration of AF_XDP and DPDK, the DPDK mbuf pool must be mapped into the UMEM for a zero copy datapath.
*Prior to Kernel 5.4
![Page 24: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/24.jpg)
DPDK Solutions
![Page 25: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/25.jpg)
DPDK Solutions
• Memcpy packets between UMEM & mbufpool
• Cycle-heavy memcpy
• DPDK 19.05
memcpy
UMEM CHUNK DPDK
MBUFpkt
pkt
Copy Mode
![Page 26: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/26.jpg)
DPDK Solutions
• Memcpy packets between UMEM & mbufpool
• Cycle-heavy memcpy
• DPDK 19.05
UMEM CHUNK
DPDK MBUF
pkt pkt
memcpy
UMEM CHUNK DPDK
MBUFpkt
pkt
Copy Mode Alignment API
• API to change mbuf pool alignment, then 1:1 map
• Invasive change
• No DPDK release
![Page 27: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/27.jpg)
DPDK Solutions
• Memcpy packets between UMEM & mbufpool
• Cycle-heavy memcpy
• DPDK 19.05
UMEM CHUNK DPDK
MBUFpkt
*pkt
UMEM CHUNK
DPDK MBUF
pkt pkt
memcpy
UMEM CHUNK DPDK
MBUFpkt
pkt
Copy Mode Alignment API External Mbuf
• API to change mbuf pool alignment, then 1:1 map
• Invasive change
• No DPDK release
• Mbuf points to packet data in UMEM chunk
• Additional complexity
• DPDK 19.08
*
![Page 28: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/28.jpg)
New Solution: Kernel Arbitrary Alignment
![Page 29: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/29.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
![Page 30: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/30.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
Chunk 0
Chunk 1
Chunk 2
Chunk 3
![Page 31: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/31.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
• Enables use of arbitrary chunk size (up to 4k).
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
Chunk 0
Chunk 1
Chunk 2
Chunk 3
![Page 32: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/32.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
• Enables use of arbitrary chunk size (up to 4k).
• Chunks can cross page boundaries.
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
Chunk 0
Chunk 1
Chunk 2
Chunk 3
![Page 33: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/33.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
• Enables use of arbitrary chunk size (up to 4k).
• Chunks can cross page boundaries.
• If physically contiguous!
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
Chunk 0
Chunk 1
Chunk 2
Crosses physically non-contiguouspage boundary
Chunk 3
![Page 34: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/34.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
• Enables use of arbitrary chunk size (up to 4k).
• Chunks can cross page boundaries.
• If physically contiguous!
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
Chunk 0
Chunk 1
Chunk 2
Chunk 3
![Page 35: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/35.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
• Enables use of arbitrary chunk size (up to 4k).
• Chunks can cross page boundaries.
• If physically contiguous!
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
Chunk 0
Chunk 1
Chunk 2
Chunk 3
64-bit “address” field
AF_XDP Rx/Tx Descriptor:
![Page 36: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/36.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
• Enables use of arbitrary chunk size (up to 4k).
• Chunks can cross page boundaries.
• If physically contiguous!
• New descriptor format keeps original address which is useful for buffer recycling.
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
Chunk 0
Chunk 1
Chunk 2
Chunk 3
64-bit “address” field
AF_XDP Rx/Tx Descriptor:
48 bits16 bitsOffset “Address”
![Page 37: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/37.jpg)
New Solution: Kernel Arbitrary Alignment
Main benefits of the new solution:
• Enables arbitrary chunk alignment.
• Enables use of arbitrary chunk size (up to 4k).
• Chunks can cross page boundaries.
• If physically contiguous!
• New descriptor format keeps original address which is useful for buffer recycling.
• Makes integrating AF_XDP with existing frameworks more seamless.
Chunk 0
Chunk 1
Chunk 2
Chunk 3
PA
GE
_SIZ
E =
4K
Page 0
Page 1
Page 2
Page 3
Chunk 0
Chunk 1
Chunk 2
48 bits16 bitsOffset “Address”
64-bit “address” field
AF_XDP Rx/Tx Descriptor:
New arbitrary alignment enables the UMEM to adapt to the requirements of the application
Chunk 3
![Page 38: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/38.jpg)
DPDK Integration
UMEM CHUNK
DPDK MBUF
pkt pkt
Page 0
Page 1
![Page 39: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/39.jpg)
• Now that UMEM alignment constraints are relaxed, DPDK mbuf pools can be directly mapped into the UMEM
DPDK Integration
UMEM CHUNK
DPDK MBUF
pkt pkt
Page 0
Page 1
![Page 40: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/40.jpg)
• Now that UMEM alignment constraints are relaxed, DPDK mbuf pools can be directly mapped into the UMEM
• Seamless zero-copy is now achievable
DPDK Integration
UMEM CHUNK
DPDK MBUF
pkt pkt
Page 0
Page 1
![Page 41: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/41.jpg)
• Now that UMEM alignment constraints are relaxed, DPDK mbuf pools can be directly mapped into the UMEM
• Seamless zero-copy is now achievable
• No need to modify existing DPDK applications - they work OOTB
DPDK Integration
UMEM CHUNK
DPDK MBUF
pkt pkt
Page 0
Page 1
![Page 42: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/42.jpg)
• Now that UMEM alignment constraints are relaxed, DPDK mbuf pools can be directly mapped into the UMEM
• Seamless zero-copy is now achievable
• No need to modify existing DPDK applications - they work OOTB
• Performant and portable solution
DPDK Integration
UMEM CHUNK
DPDK MBUF
pkt pkt
Page 0
Page 1
![Page 43: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/43.jpg)
• Now that UMEM alignment constraints are relaxed, DPDK mbuf pools can be directly mapped into the UMEM
• Seamless zero-copy is now achievable
• No need to modify existing DPDK applications - they work OOTB
• Performant and portable solution
• DPDK 19.11 feature
DPDK Integration
UMEM CHUNK
DPDK MBUF
pkt pkt
Page 0
Page 1
![Page 44: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/44.jpg)
Summary
Userspace
DPDK apps
DPDK AF_XDP PMD
Kernel space
NIC driver
Classification
Power Qos
Crypto Memory Mgmt
vHost
Ma
ny
m
ore
!
ethtool Ifconfig
DPDK with AF_XDP
Other kernel tools!
![Page 45: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/45.jpg)
• DPDK provides a wide range of functionality to an application, eg.:
Memory & power management, crypto libraries, virtual networking & many more!
Summary
Userspace
DPDK apps
DPDK AF_XDP PMD
Kernel space
NIC driver
Classification
Power Qos
Crypto Memory Mgmt
vHost
Ma
ny
m
ore
!
ethtool Ifconfig
DPDK with AF_XDP
Other kernel tools!
![Page 46: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/46.jpg)
• DPDK provides a wide range of functionality to an application, eg.:
Memory & power management, crypto libraries, virtual networking & many more!
• AF_XDP provides flexibility and usability through kernel control paths.
Familiar tools eg. ifconfig, ethtool, etc.
Summary
Userspace
DPDK apps
DPDK AF_XDP PMD
Kernel space
NIC driver
Classification
Power Qos
Crypto Memory Mgmt
vHost
Ma
ny
m
ore
!
ethtool Ifconfig
DPDK with AF_XDP
Other kernel tools!
![Page 47: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/47.jpg)
• DPDK provides a wide range of functionality to an application, eg.:
Memory & power management, crypto libraries, virtual networking & many more!
• AF_XDP provides flexibility and usability through kernel control paths.
Familiar tools eg. ifconfig, ethtool, etc.
• Together, the best of both worlds can be enjoyed.
Summary
Userspace
DPDK apps
DPDK AF_XDP PMD
Kernel space
NIC driver
Classification
Power Qos
Crypto Memory Mgmt
vHost
Ma
ny
m
ore
!
ethtool Ifconfig
DPDK with AF_XDP
Other kernel tools!
![Page 48: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/48.jpg)
• DPDK provides a wide range of functionality to an application, eg.:
Memory & power management, crypto libraries, virtual networking & many more!
• AF_XDP provides flexibility and usability through kernel control paths.
Familiar tools eg. ifconfig, ethtool, etc.
• Together, the best of both worlds can be enjoyed.
Summary
Userspace
DPDK apps
DPDK AF_XDP PMD
Kernel space
NIC driver
Classification
Power Qos
Crypto Memory Mgmt
vHost
Ma
ny
m
ore
!
ethtool Ifconfig
DPDK with AF_XDP
Other kernel tools!
High performing, portable, fully-featured, accelerated, usable and flexible applications are possible with DPDK + AF_XDP
![Page 49: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/49.jpg)
Magnus Karlsson, Björn Töpel,Bruce Richardson, Qi Zhang, Xiaolong Ye,DPDK & Kernel Communities
Thanks to..
![Page 50: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/50.jpg)
![Page 51: Kevin Laatz & Ciara Loftus FOSDEM 2020 · 2020-01-30 · AF_XDP UMEM* vs DPDK Mbuf Pool Chunk 0 Chunk 1 Chunk 2 Chunk 3 K DPDK MBUF 0 DPDK MBUF 1 DPDK MBUF 2 DPDK MBUF 3 Page 0 Page](https://reader033.vdocuments.pub/reader033/viewer/2022060222/5f0791517e708231d41da0ea/html5/thumbnails/51.jpg)