No Signal to Rule Them All: A Systematic Analysis of In-Network Congestion Signals

Because it plays such an important role in the resulting performance of individual flows, congestion control is a mainstay of the networking literature, with new congestion control algorithms appearing in almost every networking conference. But even if we look only at widely deployed congestion control algorithms, rather than including research proposals, the field has made impressive progress over the years. There have been several generations of widely deployed congestion control algorithms, starting with loss-based congestion control algorithms that focused on the WAN [19, 42, 38] which were followed by congestion control algorithms – most notably BBR [10] – that utilized delay and throughput rather than loss as the primary congestion signal. When datacenters became prevalent, with their short RTTs and emphasis on tail latencies, a new generation of congestion control algorithms was deployed, starting with DCTCP [5] and leading to more recent algorithms [32, 24, 28, 50].

One constant throughout this ongoing evolution was that widely deployed congestion control algorithms were mostly based on end-to-end (E2E) signals, such as RTT, packet drops, and observed throughput. The one obvious exception to the end-to-end nature of the signals used by these congestion control algorithms was ECN, whose origins date back to DECBit [36]. In the WAN context, ECN provided limited value since it was not universally adopted. Thus, congestion control algorithms had to expect a mixture of ECN-enabled and ECN-incapable routers along the path. But in the datacenter context, because uniform ECN adoption could be achieved, DCTCP leveraged ECN’s flexibility to provide a richer form of feedback from individual routers.

We have recently entered a new era in datacenter congestion control, since a much richer set of router-based signals is becoming available from individual routers. These newly emerging signals, which are referred to as in-network telemetry (INT), are collected at routers and made available to hosts via fields in packet headers, as documented in the CSIG Internet Draft [37]. There are many possible signals that could be generated at individual routers, but we must be careful about which ones the field eventually adopts, for three reasons. First, not all INT signals can be reported to a congestion control algorithm, because header real estate is scarce. Second, the mechanisms for collecting these signals operate at packet speeds and therefore must be embedded in router hardware/firmware. Thus, vendors must make a choice about which to support rather than enabling each customer to choose whatever signals they want. Lastly, we expect that some INT signals will be better than others, in the sense of leading to better datacenter congestion control algorithms. Thus, we must make a choice about INT signals, and that choice matters.

To provide some guidance on this choice, in this paper we address the question: what INT signals would be most valuable for datacenter congestion control algorithms (CCAs)? But to answer this question fully and fundamentally, we must take a general approach that does not start with limiting assumptions about the particular context in which these INT signals will be used. More specifically, we do not want to fix on a given CCA, or on a particular performance goal, or on a given workload. Instead, we want to understand how the choice of INT signals might be impacted by these various factors and whether there are any general lessons to be learned. Such lessons will allow us to ask what INT signals are best for CCAs that are designed from the ground up to leverage those signals, which is the question we care about in the long-term.

Of course, others have looked at how to leverage INT signals. In particular, as we discuss in §8, prior work has explored co-designing INT and CCAs for specific performance metrics or workloads (e.g., HPCC [28]), or optimizing a specific CCA to use a specific INT signal [43], but we are unaware of work that attempts to answer the question as generally as we tried to do.

This generality poses a methodological problem, because the value of a particular INT signal is fundamentally dependent on how well a CCA utilizes it. Ideally, we would evaluate a set of INT signals $S$ based on the performance of a CCA that was optimally designed to leverage that set of INT signals. And, given our long experience with congestion control algorithms, we expect that the “optimal” CCA will depend on both the set of INT signals and the desired performance objectives (and in what follows we assume that these objectives are some function of the resulting throughput and delay). But we have no known methods for generating the optimal CCAs for a specific set of signals and performance objectives.

As a pragmatic way out of this methodological bind, we propose to leverage prior work on learned congestion control algorithms. These learning-based techniques search the space of congestion control algorithms that have access to a set of INT signals, $S$, during training and return the optimally performing algorithm discovered during the search process. Our proposal is to use this best learned algorithm – denoted $l$CCA($S$) – as a stand-in for the optimal datacenter congestion control algorithm that uses $S$. This turns the unattainable ideal of optimality into the tangible result of learning, which we think will suffice for our efforts.

While one might leverage any learning-based approach, for reasons we discuss in §3, we build on one of the earliest approaches to learning-based congestion control – Remy [44] – and add several key modifications to improve its search capabilities in our context. We make no claim that our learning technique, which we call R+, is the best, but we find that the algorithms it produces – the $l$CCAs – generally match or outperform both the state-of-the-art hand-crafted CCAs and other learned CCAs that we obtaining from other learning algorithms. Thus, our research approach is to take, for any given set of INT signals $S$, the resulting $l$CCA($S$) as a “tractable approximation of the optimal” [46] and use it to evaluate the benefit of the set of signals $S$.

Note that the resulting $l$CCAs are not candidates for deployment. We recognize that learned CCAs often do not generalize beyond their training environment, and the world of datacenters environments is far broader than what our test cases encompass. But we are comparing $l$CCAs within the same environments that we use for training, so we believe that the basic comparisons will still provide insights, even if the resulting $l$CCAs may not sufficiently generalize to all settings they might encounter in real deployments.

The bulk of this paper is devoted to a series of experiments comparing the performance of the resulting $l$CCA($S$) as we vary the set of signals and the performance objectives, over a range of datacenter network scenarios. For the range of network configurations and performance objectives that we investigate, our core findings are twofold:

1. (1)

   CCAs that exploit INT signals see little average improvement over those that use just the E2E signals of delay, loss, sending rate, and receiving rate. More specifically, the best performance with INT is on average within 3% of that achieved using only E2E signals.

2. (2)

   There is no clear “winner” amongst the set of INT signals. On average, the performance they achieve is within 1% of each other.

These core findings were a surprise to us, and hence the majority of this paper is about understanding them more deeply. Exploring the first finding leads to the following underlying results:

1a Delay and send/receive rates suffice.

We find that finding (1) would not hold if loss was the primary E2E congestion signal (as was the case in the past); for example, the best INT dominates loss alone by 30% on average, but other E2E signals such as RTT and send/receive rates close the gap.

1b While we typically average over a range of network scenarios,

the specifics of a network scenario occasionally matters to a modest degree. Finding (1) holds in part because we are averaging across network scenarios, but there are specific scenarios where INT offers modest benefits. For instance, for one of our performance objectives (that values delay and throughput roughly equally), INT offers up to 7% higher performance in up to 11% of scenarios. These gains can be relevant for performance-sensitive datacenter contexts with specific operating regimes.

1c INT typically helps the most in extreme network scenarios.

Investigating when INT wins, we find that these specific scenarios are typically characterized by having extreme (high or low) levels of load or numbers of senders. Our hypothesis is that, in such cases, the E2E signals for an individual flow capture only a partial view of router conditions, while INT signals always capture a more “global” view of router performance, and this gives INT an edge. Validating this hypothesis remains an open question, which we hope to return to in future work.

1d In these extreme scenarios, INT is a better “multitasker”.

Our investigation of why INT wins in the above scenarios reveals that, when considering performance objectives that optimize for both throughput and delay but prioritize one over the other (e.g., weighting delay as twice as important than throughput), INT is comparable to E2E signals in optimizing the primary metric but can do a better job on optimizing the secondary metric. This may also explain why the benefits due to INT are relatively modest, since it only wins on the secondary metric.

Exploring finding (2) leads to the following underlying results:

2a Different INT signals result in different tradeoffs between performance metrics.

Wefind that within the extreme scenarios from above, queue-based INT metrics result in CCAs that maintain high throughput for a tradeoff of extra delay, while link-based metrics (e.g., link utilization) result in very low delay in many cases.

2b While different INT signals perform similarly on average,

there are specific scenarios where an INT signal dominates. Specifically, in these extreme scenarios, we find that queue-based INT signals shine in scenarios with very few senders. Meanwhile, link-based INT signals help in scenarios with long flows and many senders, but do not do well with very high numbers of senders.

We show that these results are robust across several different network topologies, traffic matrices, and specific forms of signals. Of course, the scope of our evaluation is not exhaustive or representative of all valid use cases. Thus, we encourage the application of our evaluation framework to other scenarios and hope to explore the same in future work.

This paper presents a large collection of simulation results, but they are all tied together by a single intellectual approach, which we now describe.

To quantify the performance of a CCA we must specify three important factors under which it operates. The first is the range of network characteristics (e.g., link rates, delays, buffer sizes) and workloads (e.g., number and nature of flows) under which the CCA is expected to operate. We refer to this as the range of network scenarios, which we denote by a set $N$. Second, we must specify what INT and end-to-end signals are available, which we describe by a set $S$. Finally, we must specify what performance objective is used to evaluate the CCA. Following the NUM paradigm [22], we define our objective as the average of a per-flow utility function $u$, denoted $P_u$.

For a given network scenario $N$ and utility function $P_u$, we are interested in what in-network telemetry signals $S$ a network should offer. Hence, we do not focus on what a specific CCA would do with such signals but rather what is the best any CCA could do. If we could determine the optimal CCA for any given $N$, $S$, and $P_u$, then we could address the above question by simply evaluating the optimal CCA for different candidate $S$. For example, we could simulate the optimal CCA for each $S$ over different network scenarios in $N$ and pick the $S$ that yields the maximum $P_u$. However, there is currently no known solution that searches the CCA design space to find the optimal CCA and hence we must resort to practical approximations of the optimal.

As discussed in the previous section, our strategy will be to leverage learning-based approaches to generate a CCA that we use as our approximation of the optimal. To that end, we define $l$CCA$(N,S,P_u)$ to be the learned CCA that, given the set of signals $S$, optimizes the performance objective $P_u$ when averaged over a set of network scenarios $N$. Thus, we are interested in, for a given range of networks $N$ and performance objective $P_u$, how well $l$CCA$(N,S,P_u)$ does as $S$ varies.

With $l$CCA as our approximation of an optimal CCA, we now provide an overview of our evaluation process, leaving a detailed description to §4. Our evaluation uses a learning-based CCA generator and a network simulator. In our work, we used an improved version of Remy [44], which we call R+, to generate our learned CCAs, and ns-3 [2] as our network simulator to evaluate $l$CCAs. We believe our process could easily be extended to other CCA generators and simulators.

The process of evaluating a signal set $S$ consists of three main steps as shown in Figure 1:

To approximate the optimal CCA, we turn to prior work in automating the development of congestion control. While there are many works in the space of automated CCA development that may be applicable ([47, 3, 20] for a sampling), we build on Remy [44]. In this section, we start with a brief overview of Remy and how we extended it to create R+, and then discuss our rationale for building on Remy.

We present the experimental setup for the three main factors in our evaluation: the range of network scenarios ($N$), utilities ($P_u$), and signals ($S$) that we consider. In each case, we discuss the parameter space in general and then present the specific parameter values we experimented with. Finally, we elaborate on relevant aspects of how we configure R+ and ns-3 for generating and evaluating $l$CCAs. Notably, for $N$, $P_u$, $S$, and other parts of the experimental setup, we attempt to select a reasonable initial value. However, we hope that our framework will be used to explore more configurations, topologies, etc. in future work.

We now present the results from using our framework to analyze the value of INT signals. The results show that for average case utility (score), the value is minimal. However, analyzing individual network scenarios and other performance metrics reveals the merits of INT and the specific strengths of each signal type.

Figure 3 shows the utility score averaged across all evaluated network scenarios, for different signal set options. The error bars indicate the standard deviation in scores which we found to be small ($\le 5\%$). We observed a similar trend for our 100t and 100d utilities and hence omit them from the figure due to space considerations.

We see two trends from Figure 3: (1) for a given utility, there is no significant difference between CCAs that use INT signals and those that use only E2E signals, and (2) for a given utility, the average score varies very little across different INT signal sets (Link, Queue, Link $+$ Queue). We found these results surprising and hence investigate each in §5.1 and §5.2 respectively.

In order to confirm that our finding – namely that INT signals do not provide much value in the average case – was not simply an artifact of our methodology, we modified an existing CCA to use INT signals. The selection of CCA was important since we should not change the control loop logic of the CCA (which may improve the performance regardless of the quality of the signal, and could make the CCA arguably just a different algorithm altogether). Thus, we chose BBR [10]⁴⁴4We note that we specifically used BBRv1 as a starting point since it is a simpler protocol and is readily available for ns-3. since it has a clear goal state that is independent of the signals used to achieve this state. However, we also performed similar experiments with DCTCP in C. As mentioned in [10], BBR is designed to (1) keep a BDP of packets in flight and (2) send at the bottleneck rate. At a high level, it does this by probing for bandwidth and RTT estimates, which requires purposely sending above/below the current estimate of the bottleneck rate.

BBR implements this with four states: PROBE_BW, PROBE_RTT, STARTUP, and DRAIN. It is designed to spend $\sim 98\%$ of time in the PROBE_BW phase [10]. Accordingly, we modified the conditions for changing the pacing rate in the PROBE_BW phase. In PROBE_BW, the pacing rate is set above the estimated bottleneck rate, then below, then maintains the estimated rate. The phase of the pacing rate is determined partially by time spent in the phase, as well as some exit conditions (such as incurring loss). For example, the high rate phase is exited once the minimum time has passed and either bytes have been lost or more than a BDP is inflight.

These checks presented a natural place to add INT signals. We added to the conditions to change gain phases (i.e., not modifying existing conditions). Specifically, required that utilization be above 90% to exit the high-rate phase and that queueing be less than 5 packets to exit the low-rate phase as an initial implementation; we arrived at these values after some experimentation. We also tested changes to the transitions between states, but found almost all changes to have no noticeable impact. The only transition change that did result in noticeable performance changes was exiting the PROBE_BW phase, where we add conditions to exit if the queue length is greater than 0 or the link utilization is greater than 95%.

As shown in Figure 15, these changes, denoted generally as BBR’, resulted in essentially no change in throughput performance, but a drastic improvement in the delay achieved.⁵⁵5From our evaluations, this improvement in delay comes at the cost of some fairness. Further evaluation/design may determine the correct tradeoff between the two and if fairness is a necessary concession. Although this may initially seem to indicate that the INT signals were necessary to produce this result, we made similar changes to the exit conditions based on the observed RTT instead (i.e., exit high rate if the RTT crosses some threshold indicating queueing, exit low rate when it gets below a threshold, exit PROBE_BW if above some threshold). As shown in Figure 15, we are able to achieve nearly the exact same results with RTT, a traditional end-to-end signal. This indicates that the improvement was based on an algorithmic change, rather than the virtue of the signals themselves, aligning with our results that nearly “any signal will do” in the average case.

These results are also robust to more complex topologies and scenarios. The same modifications to BBR on a line topology with extra flows to create multiple bottlenecks produced largely the same behavior. All BBR’ versions had significantly better delay in $\sim 50$% of scenarios (vs. $\sim 75$% on a dumbbell topology).

We also compare these results to our findings about the relative strengths of each individual INT signal. Using these signals in our BBR modification requires fine-tuning thresholds. As we manually experimented with trying to tune these thresholds for throughput or delay, our experience mirrored the tradeoffs found in §5. The best-performing version for each signal and each metric are shown in Table 4. These results, while necessarily anecdotal from a manual optimization process and notably small in performance, reiterate the strengths of the signals: Queue can achieve high throughput with a bounded but necessary sacrifice in delay, while Link can achieve low delay, but at the cost of some throughput. Overall, the signals must sacrifice the metric they observe more directly in order to optimize the other.

Our experience with applying INT signals to BBR supports our finding that nearly any signal could be sufficient in the average case. Of course, we cannot guarantee that a better use of the INT signals does not exist, but our best efforts did not result in any performance gain that was unique to INT signals. Further, this demonstrates that these findings may be true even when the utility function is different than our default form and the CCA is not optimized for the given setting. However, we note that other benefits of a particular signal such as understandability may be important for a CCA designer when not using a $l$CCA.

To the best of our knowledge, we are the only work which specifically tries to answer which in-network signals are the most useful to a generic CCA. There is, however, a significant body of work both in congestion control algorithms which use some form of in-network support [17, 21, 14, 7, 5, 50, 6, 45] and those that specifically use the type of INT signals we target [28, 43]. These approaches use a variety of different signals (link bandwidth, per-hop delay, etc.) and reasonably justify these choices. Further, while their goals may be similar, this can result in very different designs (e.g., TIMELY [32] and pFabric [6] both cite minimizing FCT as their goals). However, none of these works systematically address the question of what signals would be best for their (or other) goals.

While these proposed CCAs often build their own infrastructure for collecting in-network signals, [37, 1, 25] propose formats for carrying this data in packet headers. In particular, [37] proposes a general and extensible method for collecting any in-network signals for CCAs.

Meanwhile, the growing interest in these signals builds on a large body of work on in-network telemetry ([51, 18, 29, 27, 26, 49, 34] as a sample). Often this work focuses on collecting metrics for out-of-band analysis and therefore operates on timescales that are not appropriate for congestion control or are not available on a per-path basis. However, the underlying methods for efficiently collecting data at the switch are critical to the per-packet metrics used for CCAs.

There is plenty of existing work in the space of answering general questions about what is possible in congestion control design [48, 9, 8, 11, 40, 23, 4]. These works use very different methodologies (analysis and theorem-proving) than we proposed and accordingly provide quite general insights for their chosen settings/problems. However, most of these works do not aim to answer questions about the quality of the signals available to a CCA. Notably, [4] aims, in part, to determine the signals and other components of a CCA sufficient to achieve a desired property, but is limited to single-flow scenarios and end-to-end signals due to its network model. Further, it primarily focuses on worst-case scenarios rather than average case analysis. We adopt an optimization-based approach to achieve our desired generality properties (independence from the specific control loop) rather than using analysis for both a more descriptive network model and visibility into the full distribution of performance.

We chose Remy [44] as the basis for our CCA generator due to a few desirable properties: (1) there is a significant amount of work available on its capabilities and limitations [44, 46, 41, 3], (2) it produces interpretable CCAs generated in an offline setting, and (3) it can be easily extended/refined. In contrast, some other CCAs which seek to optimize a utility function often perform their exploration online [12, 13], providing no insights across uses for us to analyze. Similarly, many ML-based CCAs may achieve better performance in some cases [20, 46] but employ learning schemes (e.g., DRL) that do not produce an interpretable artifact. Despite this limitation, we experimented with [20] in §B and found it to perform both worse and more inconsistently than R+ in our setting. In addition, some other works combine various methods of learning with existing CCAs to achieve high adaptability to many scenarios [3, 47, 35], but this is not a priority for our methodology (determining the best set of signals for a given set of scenarios).

The design space our work addressed is infinite, so this paper is not intended to make definitive declarations about in-network signals, but instead is offered as a modest first step in identifying the question of which INT signals are best and proposing a methodology to answer it. Here we outline ways in which our preliminary investigation was limited, and how those limitations could be addressed in future work.

We only trained our CCAs on a few network configurations. Broadening the range of these configurations might lead to new findings, such as perhaps one set of signals generalizes better across a wide range of network conditions than others. In addition, there are many variations of signals that we did not address (e.g., number of flows, different aggregation functions), and perhaps one of these will prove superior to the signals we investigated.

Similarly, we generated CCAs for five different utility functions of the same general form. There exist many other valid forms of utility for which one may want to learn which signals perform best (e.g., message completion time). Likewise, one may be particularly interested in the tail of per-flow performance, rather than the average. While not explored in this work, our framework can accommodate these alternative objectives. Further, we do not evaluate our CCAs in scenarios where they compete with CCAs generated with different utility functions - i.e., we did not have competing utility functions. A model with multiple utilities would more accurately capture the differing goals of flows on real networks, so work in this area could be useful to demonstrate the robustness of the results.

We evaluated our generated CCAs on a few simple topologies and traffic matrices, attempting to extract some important and difficult scenarios for these CCAs. However, the design space here is vast as well. Extensions to evaluate larger topologies (such as a full datacenter network), other difficult scenarios (e.g., incast), and transports with other capabilities (e.g., receiver-driven CCAs [33, 17]) all merit additional investigation if of particular interest to a given network operator.

While we use R+ to produce our $l$CCAs and achieve independence from any particular control loop, this approach relies on R+’s training algorithm being a good search methodology. We believe it is a reasonable approach and determined that it at least produces significantly better optimization than a naive form of parameter tuning or other learning-based approaches (Appendices B). We recognize that there are likely other reasonable approaches to optimizing congestion control algorithms and we encourage exploration of the design space.