Contracts: A Unified Lens on Congestion Control Robustness, Fairness, Congestion, and Generality

This variety in contracts raises the question: “What are the tradeoffs between different contracts?” In exploring this question, we discovered a surprising result. The contract – a design aspect that may not have been chosen explicitly – fully determines four key congestion control metrics: (1) robustness to noise in congestion signals, (2) fairness in a multi-bottleneck network, (3) amount of congestion (e.g., delay, loss), and (4) generality (e.g., the range of link rates the CCA supports). Robustness and fairness are better with gradual contracts functions (e.g., Vegas [12, 45] “$1/\text{delay}$”), while congestion and generality are better with steeper contracts (e.g., Swift [36] “$1/{\text{delay}}^2$”) (§ 4). Thus, we cannot have the best in all four metrics.

Additionally, for a fixed contract, tolerating more congestion allows supporting a larger bandwidth range. In that sense congestion and generality are also at odds. Contracts also determine link utilization and total throughput. Our analysis subsumes these under fairness definitions (§ 4), where throughput and fairness are known to be at odds [50].

Similar tradeoffs have been explored before. Arun et al. [8] discuss a special case where full generality precludes simultaneous robustness and bounded variation in congestion. We find that their proposed workarounds cause unfairness due to other reasons (§ 4, § 5). Zhu et al. [72] show that delay-based CCAs cannot simultaneously ensure fixed delays and fairness. That is, if a delay-based CCA maps the same delay value to multiple rates, it does not have a contract, and cannot be fair. The NUM (Network Utility Maximization) literature [35, 61, 43, 45] studies a subset of these tradeoffs for individual CCAs. To our knowledge, we analyze a wider range of tradeoffs and also generalize them to many CCAs.

Contracts also go beyond prior work by enabling us to identify previously unknown scenarios and (avoidable) design mistakes that cause severe performance degradation or suboptimal performance in a wide range of CCAs (§ 2, § 5). We find that CCAs whose contracts have extreme shapes (e.g., logarithmic, exponential), shifts (e.g., ${\text{rate}}_{}=1/(\text{delay}-\text{c})$ vs. ${\text{rate}}_{}=1/\text{delay}$), clamps (e.g., $\text{delay}=\max (\text{c},1/{\text{rate}}_{})$), or intercepts can starve flows. We observe this in BBR [14], ICC [33], and Astraea [42]. Further, CCAs with an explicit contract (e.g., Swift [36]) do not need AIMD (Additive Increase, Multiplicative Decrease) updates to reach fairness. Instead, MIMD (Multiplicative Increase, Multiplicative Decrease) updates allow exponentially fast convergence to both fairness and efficiency (§ 6). Finally, we show that having fixed end-to-end thresholds, e.g., AIMD on delay [59, 11, 38], causes starvation on topologies with multiple bottlenecks.

Our methodology applies to not only analytically designed CCAs, but also black-box implementations, such as Tao (Remy) [60, 68], and learned CCAs, such as PCC [18] (online-learning) and Astraea [42] (reinforcement-learning or RL).

Using contracts, we build blueprints for CCA design and analysis. These blueprints take a “contracts-first” approach, making the tradeoffs and mistakes intuitive and actionable (§ 2). First, the tradeoffs are obvious from the contract choice (after a few to no algebraic steps), forcing the designer to deliberately choose between uncomfortable tradeoffs and picking Pareto-optimal points. Second, contracts align with how our community designs CCAs: a variety of CCAs start with a target rate equation and build dynamics around it, e.g., Swift [36], TFRC [21].

In contrast, other methodologies often start from a utility function: local utilities (e.g., “$\log (\text{tput})-\log (\text{delay})$” in Copa [9]), global utilities (e.g., NUM/Remy [68]), or reward functions in RL [42]. These hide tradeoffs giving an illusion of unilaterally optimizing latency or throughput when these are intertwined with other metrics like fairness, robustness, and generality (§ 4). Consequently, many designs pick sub-optimal tradeoffs as inadvertent artifacts of other design decisions (§ 3, § 4, § 5).

We envision that future work will use our blueprints for the design and analysis of CCAs enabling deliberate selection of Pareto-optimal tradeoffs and avoiding performance mistakes. We demonstrate these uses by (1) analyzing and exposing performance issues in a variety of CCAs (§ 3.1, § 5, § 6), and (2) designing a few canonical contract-based CCAs (§ 4, § 6, § 7). In another paper [3], we also use contracts to design the first CCA that that provably avoids starvation on networks with delay jitter showing that the starvation result by Arun et al. [8] is not as pessimistic as it seemed.

We build contracts to formalize the mechanisms CCAs use to coordinate fairness (§ 2, § 3). We discover and analytically quantify the tradeoffs (§ 4) and avoidable design mistakes (§ 2, § 5) exposed by contracts. We build blueprints of CCA design and analysis (§ 2) to help avoid the mistakes and navigate the tradeoffs. We validate our findings using simulation and emulation, finding a near-perfect match between analysis and measurements (§ 7). We end by discussing potential ways to work around the tradeoffs, limitations of contracts, and how we may expand their uses (§ 8, § 9). For instance, much of our current focus is on steady-state performance, and we hope to extend reasoning to short-flows and inter-CCA fairness. Further, if the network supports mechanisms like fair queueing, or if fairness is not desired, then contracts are not needed, and the tradeoffs do not apply.

We list the metrics that CCAs typically optimize, and study how contracts affect them.

1. 1.

   Link utilization, flow throughput

2. 2.

   Amount of congestion (e.g., delay, loss)

3. 3.

   Stability, and convergence time/reactivity

4. 4.

   Fairness (on general topologies)

5. 5.

   Robustness to noise in congestion signals

6. 6.

   Generality (range of link rates, flow counts)

Of these metrics, fairness notions (see below) subsume Pareto-optimality; that is, the bottleneck links are fully utilized. They also subsume the tradeoff between total throughput and fairness (see below). Stability and convergence time are concerned with transient behavior as opposed to steady-state equilibrium. While contracts may affect them, we can often independently optimize them by choosing how fast a CCA’s sending rate moves toward the target rate (§ 6). We are left with the following four metrics: robustness, fairness, congestion, and generality.

The choice of contract determines these four performance metrics. We show this by expressing each metric explicitly as a function of the contract func. Conversely, specifying a desired value for one metric constrains the contract, which in turn constrains the other metrics (i.e., a tradeoff). We derive these tradeoffs quantitatively by characterizing the feasible values of one metric under a desired constraint on another metric. We keep our metric definitions unit-less to minimize dependence on network parameters (e.g., link capacity, RTprop).

We use running examples of Vegas [12, 45] ($r=1/s$) and Swift [36] ($r=1/s^2$), where $s$ is delay. We omit constant factors that ensure that the units are consistent. E.g., the unit of $1$ in Vegas’s and Swift’s contract is bytes, and $\text{byte}\cdot \text{seconds}$ respectively, so that the function value has the unit $\text{bytes}/\text{second}$. Table 3 shows where existing CCAs fall in the tradeoff space.

Practical constraints limit congestion signals to a finite range, $s\in [S_{\min },S_{\max }]$. Small delays and losses are difficult to measure due to noise, while large delays and losses increase application perceived latency. Since the contract maps congestion signals to rates, this range confines the CCA’s operating range of link rates to $[\text{func}(S_{\max }),\text{func}(S_{\min })]$, where $\text{func}(S_{\min })>\text{func}(S_{\max })$ because func is decreasing. CCAs must select the constants in func to suit the networks they operate on. For instance, if delay spans a $200\times$ range, e.g., $\text{delay}\in [0.5\text{ms},100\text{ms}]$, Vegas can support only a $200\times$ range of sending rates (e.g. 1 Mbps to 200 Mbps), whereas Swift can support a ${200}^2\times$ range (e.g. 1 Mbps to 40 Gbps). In general, steeper contracts enable a broader range of bandwidths for the same domain of the congestion signal.

Note, our community does not generally say that a given CCA does not support a particular link rate. Here, by saying that a CCA cannot support a link rate, we mean that when the CCA is running on that on that link rate, some performance metric (e.g., high utilization, fairness, low delay, etc.) breaks down. For example, at low link rates, multiple competing Astraea flows can exhibit starvation (Figure 11). Conversely, at high link rates, all “delay-convergent” CCAs exhibit flow starvation [8].

We study the impact of “$\delta s$” error in the statistic “$s$” on the CCA’s rate. We quantify the impact by looking at the (worst-case) ratio of rates without and with the noise: $\text{Error factor}={\max }_s\frac{\text{func}(s)}{\text{func}(s+\delta s)}(2)$. A higher error factor means that the CCA is more sensitive to noise and is less robust. From § 4, the error factors for Vegas ($r=1/s$) and Swift ($r=1/s^2$) are “$1+\delta s/S_{\min }$”, and “${\left(1+\delta s/S_{\min }\right)}^2$” respectively. If $\delta s$ error perturbs Vegas by $2\times$, then the same error perturbs Swift by $4\times$. Steeper contracts yield higher (worse) error factors (Figure 3).

The magnitude of noise “$\delta s$” that a CCA must tolerate depends on the chosen statistic. For example, delay estimates may exhibit tens of milliseconds of noise [24], whereas loss-rate estimates may fluctuate by a few percent [37]. CCAs often attempt to mitigate such noise by filtering or smoothing raw congestion signals (e.g., using min, max, or averages over raw delay samples). In this setting, $\delta s$ represents the residual noise in the statistic after filtering, rather than noise in the raw signal itself.

Even small residual noise can significantly distort rate allocations when combined with certain contract shapes. For instance, Figure 15 shows that flows can starve under BBR and Copa despite the use of filtering mechanisms (BBR takes the max over ACK rates; Copa takes the min over delays). While the contract’s shape and input/output choice play a central role in determining robustness, they may not be sufficient in isolation. CCA designers should holistically reason about how noise in raw signals propagates through the aggregate statistic and the contract function to ultimately affect rate allocation.

For general network topologies, there exist multiple notions of fairness (Table 2). These are typically parameterized using $\alpha$-fair utility functions [50] given by: $U_{\alpha }(r)=\frac{r^{1-\alpha }}{1-\alpha }$, where $U$ represents the utility a flow derives from a rate of $r$, and $\alpha >0$. The rate allocation maximizes the sum of utilities across all flows $\left({\sum }_i{U}_{\alpha }(r_i)\right)$ (global utility). Larger values of $\alpha$ indicate greater fairness at the cost of total throughput [62]. Typically, $\alpha \ge 1$ is desired.

Not all contracts correspond to an $\alpha$-fair utility. As a proxy, to compare the fairness of contracts, we study their behavior in the parking lot topology (Figure 8). This topology exposes differences between fairness notions (Table 2), and is common in data centers (e.g., when inter- and intra-rack flows compete). In Figure 8, each link has a capacity of $C$ bytes/second and flows experience congestion signals from multiple congested links. The long flow ($f_0$) observes signals from two hops, while the short flows ($f_1$ and $f_2$) observe signals from only one hop. More generally, for $k$ hops, we consider flows $⟨f_0,f_1,\mathrm{\dots },f_k⟩$, where $f_0$ observes signals from all $k$ hops, and other flows see signals from a single hop.

We quantify unfairness as the (worst-case) throughput ratio of $f_k$ to $f_0$. We derive unfairness using three steady-state equations. First, at each hop, the sum of the incoming rates is equal to the link capacity: $\forall i\ge 1,r_0+r_i=C(3)$. Second, each flow’s rate ($r_i$) and statistic ($s_i$) follow the contract $\forall i\ge 0,r_i=\text{func}(s_i)(4)$, Third, $f_0$ sees an accumulation of signals from all the hops: $s_0=\text{fold}(s_1,s_2,\mathrm{\dots },s_k)(5)$. Here, fold describes the statistic $f_0$ sees when other flows see $s_1,s_2,\mathrm{\dots },s_k$. For delay, $s_0={\sum }_{i=1}^k{s}_i$, as delays add up over hops. For loss rate and ECN marking rate, $s_0=1-{\prod }_{i=1}^k(1-s_i)$, as survival probability gets multiplied over the hops. Note, when the absolute value of loss rates (or ECN marking rates) is small (e.g., ${10}^{-2}$), then fold can be approximated as a sum. For max per-hop delay (e.g., Poseidon [64]), $s_0=\max (s_1,s_2,\mathrm{\dots },s_k)$.

From § 4, we get $r_1=r_2=\mathrm{\dots }{r}_k$. Given, func is invertible, we get $s_1=s_2=\mathrm{\cdots }=s_k$. Then, the worst-case throughput ratio is given by: ${\max }_C\frac{r_k}{r_0}=\frac{\text{func}(s_k)}{\text{func}(s_0)}={\max }_C\frac{\text{func}(s_k)}{\text{func}(\text{fold}(s_k,s_k,\mathrm{\dots },k\text{ times}))}(6)$. Here, $s_k$ solves $\text{func}(s_k)+\text{func}(\text{fold}(s_k,\mathrm{\dots },k\text{ times}))=C$ (from § 4).

Substituting the contracts for Vegas and Swift, and fold as $\sum$, we get the throughput ratios of $k$ and $k^2$ respectively. Steeper contracts imply worse fairness.

We study how congestion grows with (decreasing) fair share or (increasing) flow count. Consider a dumbbell topology with capacity $C$ and $n$ flows. For each flow to get its fair share of $r_i=C/n$, the statistic needs to satisfy: $\forall i$, $r_i=C/n=\text{func}(s_i)$. This implies that ${\text{func}}^{-1}(C/n)=s_n=s_i$. We define worst-case growth (i.e., ($s$ for n flows)/($s$ for 1 flow)) as: $\text{growth}(n)={\max }_C\frac{{\text{func}}^{-1}(C/n)}{{\text{func}}^{-1}(C)}(7)$. For Vegas and Swift, $\text{growth}(n)$ is $n$ and $\sqrt{n}$ respectively. Steeper contracts imply slower growth.

Gradual contracts give better robustness and fairness, while steeper contracts give better congestion and generality. In Appendix A, we mathematically show these tradeoffs by considering each pair of contending metrics. E.g., if we want error factor $\le {ϵ}_r$ for $\delta s$ noise, then $\text{growth}(n)$ is $\mathrm{\Omega }\left(\delta s\frac{\log (n)}{\log ({ϵ}_r)}\right)$. These bounds are tight, i.e., we show CCAs that meet these bounds (see below).

The tradeoffs involving fairness depend on fold. We consider $\text{fold}\in \{\sum ,\max ,\min \}$. The tradeoff exists for $\text{fold}=\sum$, e.g., delay, loss rate, and ECN marking rate (assuming loss and ECN marking rates are small enough). There is no tradeoff for $\max$ and $\min$, e.g., max per-hop delay, as this allows independently ensuring max-min fairness. So, for CCAs like Poseidon [64], the only tradeoff is between robustness versus congestion and generality.

There are exactly two corner points in this tradeoff space. If we fix desired robustness, e.g., we want to tolerate $\delta s$ noise, then the Exponential CCA [8] gives the best generality and congestion growth, but it has poor fairness (Exp. in Table 3). If we fix desired fairness, e.g., proportional fairness ($\alpha =1$), then Vegas gives the best generality and congestion growth (for $\text{fold}=\sum$). In general, if we want $\alpha$-fairness, then the contract “$\text{func}(s)=\frac{1}{\sqrt[\alpha ]{s}}$” gives the best generality and congestion growth (for $\text{fold}=\sum$). We mathematically show that these are corner points in Appendix A.

Astraea (linear contract) and ICC (logarithmic contract) exhibit poor robustness and fairness due to extreme shapes and X-intercepts. The issue is that all the low rates (e.g., 0.1 to 1 Mbps) map to delays near the X-intercept (e.g., 40 ms) (Figure 9 middle). Consequently, small delay jitter creates large discrepancies in inferring fair shares. Similarly, on parking lot (Figure 8), if the hop delays are close to the intercept delay, the long flow ($f_0$) observes their sum, which exceeds the intercept delay. Attempting to reduce this “excessive delay”, $f_0$ reduces its rate to zero and starves.

Figure 11 demonstrates these empirically. Ambient emulation noise causes Astraea to starve flows when the fair share is low (4 flows on a dumbbell topology with 10 Mbps capacity, 30 ms RTprop, and infinite buffer). ICC starves the long flow ($f_0$) on a parking lot topology with 3 flows (2 hops) and the same network parameters. Importantly, contract analysis reveals these performance issues without running the CCAs or understanding their internals.

Note, unlike [8], the starvation here is not fundamental and is easily avoided by removing the X-intercept (e.g., ${\text{rate}}_{}$ = 1/delay). This increases the delay at low fair shares, however, this is unavoidable as transmission delays anyway grow as $1/C$ with decreasing capacity. In contrast, the issue in [8] is fundamental and is caused by the asymptote on the Y-axis, where all the high fair shares (e.g., > 100 Mbps) map to near-zero delays (Figure 9 left). This asymptote is hard to remove while supporting arbitrarily large rates with a monotonically decreasing contract.

CCAs like Swift [36] define target delay as: $\text{target\_delay}=b+1/{\text{rate}}_{}$. In such formulations, it may seem convenient to shift the contract (set a positive $b$) to maintain a minimum delay for high utilization. However, shifting changes the steady-state fixed-point causing undesired unfairness (Figure 9 right). For instance, in the parking lot topology with 2 hops, the short and long flows see delays of $d$ and $2d$ respectively. These delays are such that the corresponding contract rates add up to the link capacity. Shifting the contract rightward changes these delays in a way that increases discrepancies in the rates. We show this empirically for BBR in § 7. To avoid this issue, one should tune the scaling (§ 4) to maintain a minimum delay instead.

While we described shifting in the context of delay-based contracts, its impact more generally depends on the aggregate statistic and how it accumulates over hops (i.e., the fold function). The effect of such shifts can be analyzed using the parking lot steady-state equations (§ 4, § 4, § 4). For example, with Poseidon’s max per-hop delay statistic, shifting the contract does not introduce unfairness: in the parking lot topology, both short and long flows observe the same max per-hop delay even after shifting.

A related but distinct mechanism arises with active queue management (AQM), such as RED-based ECN marking [20, 57]. Here, it is possible to effectively shift queue buildup (and hence delay) without shifting either the contract or the aggregate statistic (i.e., ECN marking probabilities). Specifically, under fixed network parameters and topology, increasing the RED thresholds $K_{\min }$ and $K_{\max }$ raises the steady-state queuing while leaving ECN marking probabilities – and resulting rate allocations – unchanged. This behavior follows from jointly solving the parking lot steady-state equations and the ECN marking probability equation.

We illustrate how AIMD on delay (i.e., MD when delay crosses a fixed threshold and AI otherwise) causes starvation. AIMD on delay appears in 1RMA [59] and the CCAs (SMaRTT [11] and STrack [38]) proposed for standardization in the Ultra Ethernet Consortium [63]. It was also proposed by [8] to work around their impossibility result.

On a dumbbell topology, AIMD on delay works fine. It induces a Reno-like contract: “$r=1/\sqrt{s}$”, where $s$ is the high-delay probability (inverse of bytes between high-delay events) instead of loss probability. However, it creates starvation with multiple bottlenecks. All hosts attempt to maintain delay around the same (fixed) target value, but when flows observe delays from different combinations of hops, they cannot simultaneously maintain the same delay. Figure 10 shows this on a 2-hop parking lot. The short flows ($f_1$ and $f_2$) oscillate delay between the delay threshold and half the threshold (shown in gray). The long flow ($f_0$) observes their sum, which always exceeds the threshold. Consequently, $f_0$ never increments cwnd and starves.

We can resolve this by either (1) not using a fixed threshold, e.g., Swift scales the (target delay) threshold with rate, which transforms the contract to use delay instead of high-delay probability, allowing different flows to maintain different delays, or (2) moving the threshold to links instead of end-hosts, e.g., packet drops (Reno) and ECN marks (DCTCP) occur when delay crosses a threshold at the links.

This issue may arise for any CCA that uses fixed end-to-end thresholds to decide when to increase or decrease their rate, e.g., BBRv3 uses $\text{BBRLossThresh}=2\%$ [15].

MPRDMA [46], DCTCP [4], and Reno [31] all perform AIMD on binary feedback. In fact, [28] uses MPRDMA as an approximation of DCTCP. However, the minor differences in their design result in different contracts: Reno: “$r=1/\sqrt{s}$”, DCTCP: “$r=1/s^2$”, MPRDMA: “$r=1/s$”. Similarly, we found an algebraic mistake in the Linux implementation of TCP Vegas that changes its contract from “$r=1/\text{delay}$” to “$r=1/{\text{delay}}^2$” when RTprop is small. We have confirmed this with the maintainers. This bug has existed for 17 years due to a refactoring commit [40]. Such bugs may be caught immediately if CCA implementations explicitly delineate contracts.

We can interpret a contract (e.g., ${\text{rate}}_{}=1/{\text{delay}}^2$), in two ways: (1) target rate given current delay ($\text{target\_rate}=1/{\text{current\_delay}}^2$), or (2) “target delay as a function of current rate” ($\text{target\_delay}=1/\sqrt{\text{current\_rate}}$). Where, $\text{current\_rate}={\text{cwnd}}_{}/{\text{RTT}}_{}$, and $\text{current\_delay}={\text{RTT}}_{}-\min {\text{RTT}}_{}$. These yield two natural MIMD updates to implement a contract:

Where cwnd moves towards the target “$\text{next}\mathrm{\_}{\text{cwnd}}_{}\leftarrow (1-\alpha )\cdot {\text{cwnd}}_{}+\alpha \cdot {\text{target\_cwnd}}_{}$”, with optional clamps bounding the change “${\text{cwnd}}_{}\leftarrow \text{clamp}(\text{next}\mathrm{\_}{\text{cwnd}}_{},{\beta }_{\min }{\text{cwnd}}_{},{\beta }_{\max }{\text{cwnd}}_{})$”. Outside of the contract “${\text{rate}}_{}$ = 1/delay”, rate and delay ratios are different yielding different dynamics. For each $\alpha$, clamp choice, both updates are only stable for specific network parameters (e.g., capacity, flow count).

To ensure stability, we want to consider how rate and delay affect or are affected by changes in cwnd. The following update achieves stability without requiring any averaging ($\alpha$), clamps, or assumptions on network parameters:

Intuitively, cwnds map to “packets in queue + packets in pipe”. We only want to move the “packets in queue” part from the current delay to the target delay. This ${\text{cwnd}}_{}$ update isolates the term responsible for queueing delay and only scales that term using the delay ratio. We can also interpret the update as: ${\text{cwnd}}_{}\leftarrow {\text{rate}}_{}\ast \text{target}\mathrm{\_}\text{RTT}={\text{cwnd}}_{}/{\text{RTT}}_{}\ast (\min {\text{RTT}}_{}+\text{target\_delay})$.

We discussed optimal dynamics for delay-based contracts. For other signals, e.g., ECN or loss, the best cwnd update may differ. The current and target RTT in the update depends on the relation between RTT and aggregate statistic. For instance, for RED-based ECN, given target_ECN_rate, the target_RTT is “$\min {\text{RTT}}_{}+(1/C)\cdot (K_{min}+(K_{max}-K_{min})\ast \text{target\_ECN\_rate})$”. We derive this by inverting the mapping from queue size (and hence queueing delay) to ECN marking probability. This works directly when ground truth link capacity “$C$” is known (e.g., a data center), otherwise either the capacity needs to be estimated or one needs to use delay or rate ratio with a value of $\alpha$ tuned for the range of network parameters one wants to support.

TFRC [21] explores cwnd update for loss-based contracts. The challenge is that while a constant cwnd creates a constant delay, it may not create a constant loss rate (e.g., when the loss rate is less than one loss per window). Thus, we may need cwnd variations to maintain a persistent loss rate even when the target and current loss rates are equal.

A complete the CCA needs other decisions including: (O1) how to aggregate multiple statistic samples, (O2) how long to measure the samples, (O3) time between cwnd updates, and (O4) how to compute any other estimates (e.g., bandwidth or RTprop estimate). For instance, Copa computes standing RTT by taking the “minimum” (O1) over RTT samples in the last “half srtt” (standard smoothed RTT) (O2). It updates cwnd “every ACK” (O3) and uses minimum RTT over last 10 seconds to estimate RTprop (O4). Alternatively, one can estimate RTprop using BBR’s RTT probes. Implementations may also require other features like using rate instead of cwnd when the BDP is less than 1 packet [36]. Such details are orthogonal to contracts.

We use packet-level simulation and emulation. Simulations allow controlling noise and isolating one source of tradeoff at a time where emulation always has ambient noise, e.g., due to OS scheduling jitter. We use htsim [28] for simulation (also used in MPTCP [55], NDP [27], EQDS [52]), and mahimahi [51], Pantheon [70], and mininet [17] for emulation.

In simulation only, for two reasons, we give oracular knowledge of RTprop to all CCAs. First, CCAs like Swift target data center deployments where RTprop may be known. For consistency, we provide RTprop to all CCAs. Second, CCAs like Vegas do not explicitly drain queues resulting in misestimating RTprop and poor performance. We remove this source of poor performance as this is not fundamental unlike the tradeoffs imposed by contracts. For instance, Copa and BBR explicitly drain queues to estimate RTprop accurately (at least in the absence of noise).

Note that we are validating negative results (i.e., tradeoffs). Simplifications only make the validation stronger. If tradeoffs exist with oracular knowledge of RTprop, then performance is only worse without it, e.g., under-estimation causes under-utilization, and over-estimation increases congestion.

We implement and test Swift [36] and Vegas [12]. For reference, we also show 3 canonical (§ 6) contract implementations: $1/\sqrt{s}$, $1/s$, and $1/s^2$, where $s=\text{delay}={\text{RTT}}_{}-\text{RTprop}$. These avoid RTT-bias unlike vanilla Vegas/Swift, and use MIMD instead of AIAD/AIMD. In the canonical CCAs, we update cwnd every 2 RTTs and aggregate delay as the minimum over delay samples since the last cwnd update. To isolate the impact of contract shape, we also tune the scale parameters symmetrically. All CCAs have the same $S_{\min }=1.2$ $\mu s$ $=0.1$ RTprop and $C_{\max }=100$ Gbps = link capacity. $S_{\max }$ is then decided by $C_{\min }$ and the contract shape.

We set link capacity $=100$ Gbps, RTprop $=12$ $\mu s$, packet size $=4$ KB. These are default parameters in htsim for data-center deployments. In § 4 we defined the metrics in to be unitless and the tradeoffs we showed exist for all choices of network parameters (link capacity, RTprop, etc.). Consequently, the specific parameter values are of little importance and we could have used any other values as well. We set the buffer size to be infinite to remove any effects of packet losses, since we use delay-based CCAs. To measure robustness error, we use a dumbbell topology with 2 flows where one of them witnesses noise. We inject controlled error by adding a hop that persistently delays packets by “$\delta s$” $\mu s$, and vary $\delta s$. We do not include this in the RTprop provided to the CCAs. We inject noise this way to show trends. In emulation, we show the impact of realistic noise. To measure unfairness and congestion growth, we instantiate parking lot (with varying hops) and dumbbell (with varying flow count) topologies respectively. For the CCAs we test, generality is just the inverse of congestion growth (Table 3), so we do not show generality.

Empirical performance matches that estimated by contracts (Figure 13). Swift’s RTT-bias causes slightly worse fairness and robustness than the equivalent “$1/s^2$” canonical CCA which removes the bias. Vanilla Vegas has RTprop-bias (instead of RTT-bias). Since the RTprop is the same for all flows, the steady-state performance of Vegas is the same as the canonical “$1/s$” CCA.

We run Cubic [25], BBR [14] (Linux kernel v5.15.0) and Copa [9, 7]. The empirical contract derivations (§ 3) already showed generality and congestion growth, e.g., increase in delay or loss rate with decreasing fair share (increasing flow count). In Figure 15 and Figure 16, we show robustness and fairness. We run flows for 5 mins on dumbbell and parking lot topologies with capacity $=100$ Mbps, buffer $=1$ BDP, and describe RTprop and flow count below. Emulation does not scale to data-center link speeds (unlike htsim). Here, our parameter choices align with Internet deployments. Again, the tradeoffs are independent of the absolute parameter values and we get qualitatively similar results with other values.

Note that noise in the raw signal may not create an equivalent amount of error in the statistic used by the CCA. Hence, we do not see a persistent trend in robustness error with varying noise. To validate that robustness is an issue, we show that the CCAs incur large throughput ratios (starvation) with small delay jitter. We inject jitter in two ways: (J1) slightly different RTprops (3 flows with RTprop of 10, 20, and 30 ms), and (J2) ACK-aggregation (3 flows with RTprop of 32 ms but 1 flow additionally witnesses 32 ms of ACK aggregation). We emulate ACK-aggregation in the same way as Pantheon [70]. In Figure 15, Cubic and Copa show starvation with J2 and BBR shows starvation with J1. Note, BBR’s unfairness in J1 is different from RTT-unfairness in traditional CCAs [31, 25]. For BBR, a small difference in RTprops leads to large unfairness that increases with the link rate [8].

Figure 16 shows unfairness on parking lot with 5 ms RTprop. Copa matches the trend estimated by the contract. With BBR, the shape (derivative) of the contract is same as Copa. However the shift and RTT-bias in BBR’s contract causes worse unfairness. For Cubic, the throughput ratio should be at least ${\text{hops}}^{4/3}$ (contract is ${\text{rate}}_{}={\text{loss rate}}^{-0.75}$ [39]). Reality is worse due to RTT-bias.

In defining contracts, we faced a tradeoff between expressivity (breadth of statements we can make) and tractability (mathematically backing the statements). We erred on the side of tractability to mathematically derive tradeoffs in Appendix A. For instance, our current definition makes assumptions about the network and other flows. As a result, we are unable to reason about inter-CCA fairness. Due to similar reasons, we also found it hard to prove/disprove that contracts are necessary or sufficient for fairness. Below, we describe the challenges, benefits, and possible approaches to improve expressivity.

Challenges. Two CCAs may achieve fairness even when they have different contracts. Consider Copa, it employs a delay-based contract when competing with itself, but it switches to emulating Reno when it detects Reno flows. Even CCAs like Vegas, that employ a delay-based contract all the time, may compete fairly with Reno depending on the network conditions [43]. For instance, RED-based packet drops [20] create a mapping between queuing delay and loss rates. If the loss-rate-based and delay-based fair shares match, then Vegas and Reno may compete fairly.

Benefits. Improving expressivity can further guide CCA design. For instance, to be TCP-friendly, BBRv3 [15] leaves “headroom” for loss-based CCAs. If contracts are necessary for fairness, then it is better to explicitly follow Reno’s contract on detecting competing Reno flows than leaving headroom which may or may not cause BBRv3 to follow Reno’s contract.

Approach. We hope to extend reasoning using the formal methods literature that also uses contract-like abstractions to reason about distributed algorithms. For instance, [6] defines contracts as a set of traces described using $\omega -$regular grammars, while we defined contracts as a function.

Our current blueprints do not capture all scenarios that CCAs must handle, e.g., short flows, incast, micro-bursts [58, 32], frequent flow arrivals and departures, or bursty traffic. In such settings, flows may never reach steady state; nevertheless, we still desire fairness guarantees for long-lived flows. Similarly, we do not model all metrics that may be relevant to applications, such as flow or co-flow completion times. Supporting these scenarios and metrics often requires complementary mechanisms beyond contracts, including slow start, receiver- or credit-based control, and flow scheduling.

Moreover, while we showed that contracts determine key aspects of CCA performance, they are not all-encompassing. Other design nuances can also significantly impact performance. For instance, CCAs periodically explore latent network parameters and smooth or filter noise from raw congestion signals. Although these choices influence the observed aggregate statistics and thus appear in the resulting contract, the contract itself does not directly tell us how such mechanisms should be designed. Nevertheless, we believe that the lens of contracts can help designers reason about these choices indirectly, by clarifying their impact on steady-state behavior and tradeoffs.

In the future, we envision extending our blueprints into a living repository that incorporates these evolving scenarios, metrics, and design nuances, providing a comprehensive view of CCA behavior. Outside CCA design and analysis, we believe a contracts-first approach would also improve reverse-engineering and classification of CCAs in the wild [22, 19, 66, 48].

Measurement of workloads and network environments can guide which metrics to prioritize in the tradeoff space. For instance, assessing fairness requires understanding how often flows experience multi-hop congestion and the typical number of flows or hops involved. Recent work [13] suggests that contention may be rare on the Internet, implying fairness may be less critical in some contexts. For robustness, it would be useful to quantify the size and frequency of non-congestive delays and losses. On the workload front, we want to understand which network-level metrics correlate with application-level metrics. For instance, [56] shows that unfairness is better for AI collectives.