Programmable Host Networking

The first tutorial of the workshop was by Satnam Singh who works at Groq [1], a startup that developed a custom ML inference cluster architecture, and implements this architecture using FPGAs. Using FPGA improves flexibility: Satnam explained that Groq can choose how to implement operations, how to connect them, etc. However, it should come at a performance cost because GPU clock speeds are significantly higher than FPGAs.

Satnam explained that this is a case where the flexibility of programmable logic is a bigger help than any loss in performance: like other ML cluster, Groq distributed computation across the cluster, but unlike with GPUs they know at a cycle granularity when each computation will complete. This allows them to design clusters where explicit coordination (through barriers) is not required, and where no cycles are wasted waiting for data to arrive. Both in combination allow them to achieve better performance than GPU clusters.

The group’s takeaway from this talk was that programmable logic is useful, even if it comes at the cost of clock speeds or other low-level performance metrics.

In this tutorial, Boris provided his perspective on how to rethink the interface between NICs and CPU for high-performance processing [1, 2, 3].

The starting point was that network software interface are evolving with a number of technology trends and NIC upgrades are more cost-effective relative to CPU upgrades. One insight was that NIC offloads are mostly free. These together motivate to look at offloading CPU functionality to the NIC. Boris showed two trends about CPUs available from Intel, AMD, and ARM vendors: (1) CPU I/O speeds are growing faster than CPU core number; and (2) CPU I/O speeds are growing faster than CPU memory bandwidth. These indicate that cores are increasingly unable to process data at line-rate and that memory bandwidth isn’t sufficient to store all I/O data, making NIC offloads and other ways the NIC can improve CPU efficiency even more important.

This lead to a discussion on how several software techniques can optimize performance: batching at different layers of the I/O stack; zerocopy on receive, transmit, within applications, and by using device memory. The tutorial provided insights on current research that tackles memory bandwidth bottlenecks by backpressure on cores and the network, and research that fits the workload to the last-level cache to benefit from DDIO.

Finally, Boris discussed several NIC hardware techniques to deal with multi-core CPUs that introduce challenges on fair and efficient packet scheduling on transmit and receive and also advances in NIC SRIOV support that enable matching paravirtualized NIC functionality while gaining the performance benefits of SRIOV.

Timothy Roscoe presented lessons learned while building Enzian [1], an open platform designed to make it easier to experiment with hybrid computing platforms that include CPU cores, FPGAs and various accelerators. Enzian has been used by multiple projects, and it allows a great degree of flexibility. However, this flexibility has required the Enzian team to often eschew the use of off-the-shelf commercial parts, and instead design new parts: for example, Enzian required the development of a new server board, new baseboard management component, a new interconnect between CPU and FPGA, etc. These increased costs and time to development, but have been crucial to Enzian’s success. Mothy’s talk discussed the history of this development, and ongoing challenges with maintaining and producing new instances of this hardware.

In this tutorial, Rolf provided an overview of PCIe, the de-facto I/O interconnect in contemporary servers. PCIe uses a high-speed serial interconnect based on a point-to-point topology consisting of several serial links (or lanes) between endpoints. It is a protocol with three layers: physical, data link and transaction. While the data link layer (DLL) implements error correction, flow control and acknowledgments, the transaction layer turns user application data, or completion data, into PCIe transactions using Transaction Layer Packets (TLPs).

After a brief introduction, he delved in the overheads associated to PCIe trasactions taking as a reference point a 40 Gb/s NIC with a PCIe Gen 3 interface with 8 lanes. Here, each lane offers 8 GT/s (Giga Transactions per second) using a 128b/130b encoding, resulting in $8\times 7.87$ Gb/s = 62.96 Gb/s at the physical layer. The DLL adds around 8–10% of overheads due to flow control and acknowledgment messages, leaving around 57.88 Gb/s available at the TLP layer. For each transaction, the physical layer adds 2B of framing and the DLL adds a 6B header. Apart from transferring the packet data itself, a NIC also has to read TX and freelist descriptors, write back RX (and sometimes TX) descriptors and generate interrupts. Device drivers also have to read and update queue pointers on the device. All these interactions are PCIe transactions consuming additional PCIe bandwidth.

The tutorial then moved to specific features of the PCIe such as the IOMMU, which is in modern systems interposed in the data path between a PCIe device and the host. The IOMMU performs address-translation for addresses present in PCIe transactions and utilizes an internal Transaction Lookaside Buffer (TLB) as a cache for translated addresses. On a TLB miss, the IOMMU must perform a full page table walk, which may delay the transaction and thus may increase latency and impact throughput. The problem highlighted is that this can introduce introduce variance as transactions are now depending on the temporal state of caches, IOMMU TLB and the characteristics of the CPU interconnects.

The tutorial concluded with a discussion of scale-up interconnect technologies for machine learning clusters with a specific focus on NVLink. NVLink is a high-speed, wire-based, point-to-point connection technology that allows GPUs to communicate directly with each other within a server. It’s a faster alternative to traditional PCIe-based solutions and is used to scale memory and performance for computing demanding workloads.

In this tutorial, Emmanuel provided an overview of Compute Express Link (CXL), an open standard interconnect for low latency, high throughput I/O designed to interconnect hosts, accelerators, and memory devices. CXL uses the ubiquitous PCIe electrical interface and defines three protocols: CXL.io, CXL.cache, and CXL.mem. CXL.io provides I/O semantics and is implemented by all CXL devices because it is used by the control path. CXL.cache implements a cache synchronization protocol for device and CPU caches, and CXL.mem provides a transactional interface between CPUs and CXL memory devices.

The tutorial touched upon the three device-types defined by CXL (i.e., Type-1 and Type-2 which are accelerators and Type-3 which are memory devices) and discussed the main motivation behind this new standard interconnect, taking as example memory disaggregation. Many prior research efforts, indeed, has shown the potential of memory disaggregation with high-performance network fabrics. These efforts rely on software memory disaggregation: software inititates requests to access disaggregated memory and acknowledges completions. For instance, using RDMA, application libraries or the OS must post memory access requests to network queues; the NIC then adds completions to completion queues, which software drains. This process is slow and poorly aligned with CPU architectural features. Further, this problem is common to most network transports, including TCP, where CPU overheads are even larger, exacerbating the performance impact of software memory disaggregation. By integrating CXL ports directly into CPUs, processors can directly address disaggregated memory, enabling CPU architectural features like caching, prefetching, pipelining, and speculative execution. As a result, hardware memory disaggregation reduces CPU overheads, lowers latency, and increases throughput compared to previous software approaches.

The tutorial then moved on early performance results presented by previous research papers alongside new research directions opened by this standard. In particular, Emmanuel first focussed on the question if it is possible to build data structures that provide consistency atop incoherent CXL.mem and took as example in-memory storage for data flow computation. Then, he discussed about how to enable resource composability by providing better efficiency and flexibility on top of a CXL interconnect.

There are currently two ways to write applications: the “traditional” way, which uses familiar APIs, and the “hard” way, in which these APIs are discarded in favor of lower-level ones with more control over factors that impact performance. I argue that neither of these ways is easy, and this leaves a lot of applications behind, causing programmers to reach for traditional APIs at the expense of benefiting from advances in networking technology.

Offloading compute-intensive workloads is a routine for the hyperscalers, and cryptographic primitives (e.g., encryption and decryption) are one of the important offload candidates. Current approaches use various solutions such as on-CPU accelerators, off-network-path fixed-function ASICs, and reconfigurable FPGA-based accelerators. However, these solutions mainly focus on a single cryptographic algorithm, while real-world workloads often involve multiple algorithms. For instance, a TLS data path connection uses algorithms such as AES-GCM, AES-CTR, AES-CCM, and Chacha for (de)encryption. FPGA-based accelerators primarily focus on performance or power efficiency, but non-parallelizable workloads see negligible gains via performance accelerators, leading to a waste of hardware resources and power. We propose to design workload-aware cryptographic accelerator primitives for FPGA NIC, and a workload-aware scheduler. We are working on addressing challenges such as work-preservation, HOL blocking and performance isolation for tenants and would highly appreciate your inputs and feedback.

Recently there have been a few designs that propose providing Trusted Execution in SmartNICs. Are there enough use cases for this? I posit that the problem to be solved is either way deeper than a TEE, or it is so niche that it is not that important in practice.

We had started the day hearing about two different interconnect technologies: PCIe and CXL. Furthermore, Mothy’s talk on Enzian (§6.1) had indicated that current interconnects might be holding us back. To resolve this, we organized a debate about what is necessary and desirable for interconnects within the server. We had Rolf, Emmanuel, Mothy, and Satnam up on stage, and had each of them talk about interconnect differences and efficiencies. The core message was that PCIe and CXL have high-overheads, but these might be necessary to support the range of use-cases that they are currently employed for.

In the past few decades, networking systems and frameworks have evolved to address modular design, reusability, and performance. Network I/O is asynchronous by nature, yet most frameworks rely on event-driven programming with callbacks. While event-driven programming ensures high performance, it compromises reusability due to application-specific dependencies among callback functions. The async/await paradigm is gaining momentum as a general abstraction for asynchronous programming by encapsulating asynchronous behaviors into application-independent combinators. As a result, a program’s asynchronous behavior is composed from few primitive building blocks. However, the async/await paradigm has had limited exposure in networking systems and frameworks due to their high-throughput, low-latency requirements.

We present AsyncNet, a novel networking system framework that adopts theasync/await paradigm to meet the high- throughput, low-latency requirements of network systems. AsyncNet addresses three major challenges while adopting the combinator pattern of the async/await paradigm. First, network systems eagerly use polling to eliminate context-switching overhead; however, existing schedulers cause imbalanced scheduling between polling and non-polling tasks. We distinguish between the two types of tasks on separate run queues and provide a tailored scheduling algorithm. Second, techniques for high-throughput networking, such as direct memory access, zero-copy API, and timer management, were not encapsulated within a framework, forcing programmers to embed such techniques along with the protocol logic. We embed these techniques as part of our framework. With the fine declaration of access patterns, our framework automatically determines whether to make a copy or not. Finally, some unique data movement patterns of network systems have not been encapsulated as combinators in existing frameworks. We provide dedicated combinator implementations, notably for batching, demuxing, muxing, and broadcasting, to encapsulate these unique data movement patterns. On top of AsyncNet, we build a prototype networking stack which is split into layers of independent protocol layers. Thanks to the specialized scheduler, embedded high-performance techniques, and dedicated combinators, the prototype networking stack, which is split into layers of independent protocol logic, outperforms the existing full- featured networking stack from Linux. Under the same la- tency constraints, AsyncNet’s networking stack achieves 8.85x throughput on a UDP echo application and 1.9x throughput on a web server application.

We have seen a surge of kernel-bypass network stacks with different design decisions for higher throughput and lower latency than the kernel stack, but how do they perform in comparison to each others in a variety of workload, given that modern stacks have to handle both bulk data transfers over multi-hundred gigabit ethernet and small RPCs that require low latency well? We found that even representative kernel-bypass stacks have never been compared for a set of basic workloads, likely because of difficulty to reproduce their implementation. This paper takes the first step towards answering that question by comparing six in-kernel or kernel bypass stacks. We show that existing stacks cannot handle those workloads at the same time or lack generality.

While software updates typically require system reboots, leading to service downtimes, this talk addresses dynamically reprogrammable data planes. Our approach aims for reprogrammability while avoiding service degradation by integrating eBPF into the P4 pipeline, to combine the flexibility and dynamic adaptability of eBPF with the efficiency of P4. We investigate the approach for the P4 target T4P4S. The talk also explains the framework used to perform the experiments: our testbed in combination with our traffic generator MoonGen and our experiment framework pos (plain orchestrating service), which is part of the ESFRI SLICES research infrastructure.