Monitoring Latency on Submarine Cables: Limitations and Opportunities

RIPE Atlas is a widely-utilized measurement infrastructure hosting a global network of vantage points (anchors) that are used to launch active probe-based measurements [26]. The large number (approximately 800) of anchors distributed across different continents guarantees that some portion of the SCN will be traversed by ICMP-based traceroute mesh measurements between these anchors. We selected RIPE Atlas for SL monitoring not only due to this coverage, but also because of the publicly available repository of relatively frequent measurements that are launched in a full mesh between all anchors at a rate of one measurement every 15-minutes.

Given the availability of these consistent and geographically diverse measurements, we did not launch any additional measurements using RIPE Atlas. We focused instead on analyzing publicly available data.¹¹1For this we used the Python wrapper for the RIPE Atlas API [1]. While this design choice is somewhat restrictive, it facilitates reproducibility of our results, and is sufficient for our goals. Note that there is nothing about our methodology that is specific to RIPE Atlas. Our techniques can be directly applied to other similar datasets that are available from other widely-deployed measurement infrastructures, some of which are discussed below.

Despite the accessibility and geographic scope of RIPE Atlas infrastructure and measurement data, using it to monitor the SCN poses a significant challenge. RIPE Atlas anchors tend to be located in universities and laboratories geographically and topologically distant from SL ingress points and major Points of Presence (PoPs). Greater distance results in the possibility of greater measurement “noise” as discussed below. The geographic distribution also limits the conclusions one could offer on the SL network as a whole, however this is not an issue for this paper, which does not aim to provide a complete map of SLs, but rather to develop a methodology and use it to explore fundamental limitations for the SL monitoring problem.

Submarine cable systems form the backbone of global Internet connectivity, linking continents and enabling the high-capacity transmission of data across vast distances. A submarine cable system is often composed of multiple individual submarine segments, each connecting two specific endpoints along the cable’s route. These segments function as discrete physical links within the broader network, collectively forming the complete intercontinental pathway. Monitoring these individual segments provides insight into the performance and resilience of the entire submarine cable system.

To analyze the network-layer structure of these systems, we define a submarine link (SL) as the set of IP pairs extracted from consecutive traceroute hops traversing submarine segments in a single direction (see Figure 1), with a submarine segment monitored bidirectionally corresponding to two SLs. Submarine cable ingress and egress points are typically housed within facilities equipped with co-located routers that possess multiple ingress and egress interfaces. Consequently, multiple IP address pairs often correspond to the same underlying infrastructure. To geolocate these IP addresses, we utilize a combination of RTT measurements, geographic hints embedded in DNS records, and the Hoiho geolocation tool. The SCN infrastructure we seek to monitor lies between these identified near-side (ingress) and far-side (egress) IP addresses. We monitor conditions on 101 SLs.

Ideally, we would like to instrument the SL with dedicated, time synchronized monitoring infrastructure at each end, to obtain a direct RTT timeseries measurement. Our task is to make the best possible estimate of the properties of this ideal timeseries by exploiting available traceroute measurements. It is not clear a priori which of its properties can be effectively extracted.

We list below a set of concerns inherent in traceroute-based measurement that create difficulties and limitations on what is achievable for our purposes [23]. Here we provide a brief general overview of these sources of “noise.” A more technical and detailed description is given in Section 5, where we develop methods to address them.

Positing that vantage points topologically and geographically near to submarine ingress points provide a more accurate view of cable behavior, we manually construct a knowledge base coalescing cable system landing point locations and information about known owners and users of specific submarine infrastructure. This repository includes details of geographic and topological information for over 350 submarine cable systems, which we use in vantage point selection, traceroute collection, and SL identification. We reference comprehensive submarine cable system maps hosted by Infrapedia and Telegeography, network maps provided by ISPs with major global footprints (i.e., Arelion, CenturyLink, and PCCW), industry reports, and cable system information provided by online data sets such as Submarine Network’s [12, 31, 32].

To measure latency on a submarine segment, a vantage point(s) providing an accurate perspective, or as accurate a perspective as possible, is required. We first identify vantage points that meet the baseline criteria of launching measurements which cross the target submarine infrastructure and then filter for the highest quality candidates.

SL ingress points are typically situated at further inland network PoPs and data centers which connect to submarine cables via hidden layer 2 infrastructure (i.e., AEC-1 ²²2This segment comprises AEC-1, CeltixConnect, and a terrestrial component over Ireland, as reported in [31]. Our ground truth measurements provided via ESnet report measurements over this combined infrastructure [33]. connects facilities Equinix LD6 in London and Equinix NY5 in New York City [31]). Because of this hidden layer-2 infrastructure, submarine links in traceroute measurements must be mapped to the major PoPs connected to the segment, and vantage points near these PoPs provide a better perspective of network behavior than those situated near the actual landing points (LPs).

We developed an approach to select RIPE Atlas anchors geographically near these PoPs, first extracting the latitude and longitude of LP cities using the python library GeoPy, then using iGDB to identify up to three PoPs in nearby major city centers [2, 14]. When identifying PoPs near an LP, we give priority to those used by known operators and users of the target submarine cable system. Next, we collect a set of candidate RIPE Atlas $(O,D)$ pairs wherein each node is within 100km of a PoP near the ingress or egress point of the target submarine segment. The purpose of using three nearby PoPs is to increase the number of candidate VPs meeting our baseline criteria. The remainder of our methodology seeks to reduce this set of VPs crossing a target SL to those that provide the highest quality data (i.e., lowest noise), so our key results are not sensitive to this threshold. We collect measurements between each $(O,D)$ pair from the pre-existing RIPE Atlas anchor mesh measurement repository rather than launching our own, user-defined measurements.

Following a similar methodology as proposed by Ramanathan et al. [24], we use a combination of IP geolocation, speed of light constraint RTT prediction, and AS identification to verify that a traceroute crosses the target submarine infrastructure. However, our methodology differs in that we identify vantage points specifically near SL ingress points, towards the goal of minimizing noise in the measurements.

In this subsection we assume stable routing conditions. The RTT over any path (for fixed sized probe packets) can then be written as a constant minimum value corresponding to zero congestion, and a random congestion term. Thus,

where $\{r1$, $r2$, $r\}$ are the minima, and $\{q1$, $q2$, $q\}$ the congestion terms.

In terms of available data, a single traceroute provides a trio of closely spaced measurements $\{r{1}^i,r{1}^{ii},r{1}^{iii}\}$ of $R1$, closely followed by a trio of $R2$ measurements: $\{r{2}^i,r{2}^{ii},r{2}^{iii}\}$. Thus a trace of $M$ traceroutes consists of $N=3M$ measurements of each of $R1$ and $R2$, organised temporally into $M$ trio-pairs. For $R1$ (similarly for $R2$) we label these in order as $\{r{1}_j\}$, $j=1$, $2,\mathrm{\cdots },N$. Where measurements are missing (traceroute returns $\ast$), the missing elements are ignored.

To extract precise information on the SL from the $(R1,R2)$ measurements requires a critical assumption, that of maximally shared paths (MSP), meaning that the paths taken by packets sent to ingress and egress are identical in each direction, except for the SL itself. We illustrate the concept of MSP in Figure 4.

Assume MSP holds. Then the minima obey $\underset{¯}{r}=\underset{¯}{r2}-\underset{¯}{r1}$, and a natural unpaired estimator is

Under MSP the mean latency over the SL is also accessible. From the linearity of expectation a natural estimator is ${\mu }_R=({\sum }_{j}r{2}_j)/N_1-({\sum }_{j}r{1}_j)/N_2$, where $N_1,N_2\le N$ account for missing measurements. This is unbiased regardless of any correlations between congestion terms, including differing patterns or degree of missing measurements over the two routes.

Path changes are endemic in the Internet, and result in sudden level shifts (LSes) in latency statistics. The upper left plot in Figure 5 provides an example where 2 LSes are seen in $R1$, and 3 in $R2$. The earliest and latest LSes match exactly, revealing that those changes occur on shared portions of the $O⮌I$ and $O⮌E$ paths, and so they cancel in the ${\mathrm{\Delta }}_R=R2-R1$ timeseries. The remaining LS is in $R2$ only. Such an unshared LS induces a LS in ${\mathrm{\Delta }}_R$ - nominally impossible for an SL - proving in particular that MSP does not hold for this trace overall.

One of the limitations of the unpaired estimate (1) above is that it is not LS-aware - it sees only the lowest level in a multi-level data set, which results in unused samples at best, or completely erroneous estimates at worst. We now describe an improved approach to replace ${\mathrm{\Delta }}_R$, leading to a robust $\underset{¯}{\overset{^}{r}}$.

We pre-process each trio-pair into a 2D timeseries $(P1,P2)$, where $P{1}_i$ is the min of the values in the $R1$ trio, $i=1$, $2,\mathrm{\cdots },M$, and similarly for $P2$. A sample $(p1,p2)$ is dropped when a trio is ($\ast ,\ast ,\ast$). On the short intra-trio-pair timescale, path changes are very unlikely. Thus $P1$ and $P2$ will act as min-filtered versions of $R1$, $R2$ over non-overlapping windows of width 3. This compresses values toward the $\underset{¯}{r1}$, $\underset{¯}{r2}$ minima without distorting LS structure, so that the timeseries $D=\{D_i=P{2}_i-P{1}_i\}$ will still cancel shared LSes.

Consider again the noise about the underlying minima. Each measurement arises from a different packet, hence the $N$ samples $\{q{1}_j\}$ of the congestion term for $R1$ are all distinct from those of $R2$ and so cannot cancel them, resulting in marked two-sided noise in ${\mathrm{\Delta }}_R$. Being bidirectional, it cannot be suppressed using minimum-based filtering (to do so would lead to very significant errors, see Section 6.1). In the case of $D$, the minimum filtering underlying $(P1,P2)$ significantly reduces the amplitude of the bidirectional noise. Just as importantly however, it also brings the minimum and median values closer together in each direction, so that the resulting noise in the difference $D$ is much more symmetrically centered. To augment this advantage we apply a sliding median filtering of window width $w$ to obtain ${\mathrm{\Delta }}_P={\text{Med}}_w(D)$ (we use $w=31$ corresponding to $\approx 7.8$hrs, see below). As this filter has the property that it preserves the position and amplitude of discontinuities, the result is effectively a form of $\widehat{}\underset{¯}{r}$ timeseries that preserves unshared LSes (without the need for explicit detection), and enjoys low amplitude and locally symmetric noise over each level. The lower plot in Figure 5 shows $P_1$, $P_2$, and ${\mathrm{\Delta }}_P$, the cleaner view compared to the upper plot on the same data, in particular with respect to ${\mathrm{\Delta }}_P$, is striking. Further examples are provided in the center and right columns of Figure 5 where ${\mathrm{\Delta }}_P$ is free of LSes, so that ${\mathrm{\Delta }}_P$ is just the constant $\underset{¯}{r2}-\underset{¯}{r1}$ plus residual bidirectional noise that is approximately symmetric. When MSP holds this constant is simply $r$, and the approximate symmetry motivates

This robust estimator avoids the inherent dangers of the unpaired estimator, ${\underset{¯}{\overset{^}{r}}}_u$ which, being based on direct minima, is highly sensitive to anomalies of any kind (including many forms of clock error) even if very rare. The sliding median filtering also cleanly cancels very short lived LSes, which would not have provided adequate samples for estimation.

The clean view afforded by ${\mathrm{\Delta }}_P$ has another key advantage: it makes automatic segmentation into inter-LS zones with acceptable reliability feasible. We use the Python ruptures package for this purpose. To reduce the impact of errors in the inferred locations that nonetheless sometimes occur, for each returned zone we trim $\approx 20$ values from each end. Estimation is performed separately over each trimmed zone.

Using the above methodology, for a given SL we obtain a set of $K$ timeseries $\{{\mathrm{\Delta }}_P^k\}$, one per ${(O,D)}_k$ probe pair satisfying the careful VP selection described in Section 3. A timeseries with $l\ge 0$ LSes gives rise to $l+1$ zones to be assessed, for a total of $L={\sum }_k{l}_k\ge K$ estimates capturing the different levels we see in the timeseries.

Careful examination against cases where we have ground truth (see below) failed to find any predictor enabling preferred level(s) to be identified. In fact even in the case of no LSes where $L=K$, we have no way of confirming which ${\mathrm{\Delta }}_P^k$ obey MSP. As a result, we adopt a neutral approach and average estimates made using (2) to form

In the next section we will compare against an unpaired version of (3), that we define here as

which does not not require any LS segmentation to calculate.

When it comes to ground truth data, we use ${\mathrm{\Delta }}_P$ for visualization, but the unpaired estimator (1), as the robustness imperative is much reduced. In such a case the fact that the unpaired estimator directly targets minima (hence robust to congestion), does not suffer from any errors in LS segmentation, and makes use of all of the data, makes it preferable to (2).

To neutrally characterise the spread of available level estimates, we define the min-latency variability as

Figure 6 provides examples of all $\{{\mathrm{\Delta }}_P^k\}$ timeseries available for four SLs where ground truth data was available. Each case provides counter-examples to otherwise tempting principles we had hoped to propose as a basis for selecting the best level(s), and in particular those obeying MSP.

In the top plot many levels are close to 67ms, however the ground truth is elsewhere: a clustering heuristic would fail here. In the second plot the highest (purple) curve enjoys the smallest AS hop count of 2 compared to the other levels. Here routing might be expected to be more efficient and hence more likely to obey MSP, however it is the furthest from the ground truth. The third plot tells a similar tale for Origin probes (thick green and yellow curves at the top) that are closest to ingress in networking terms (smallest $\min (R1)$), limiting the scope for routing diversity. Again they are actually the furthest from the ground truth. The last plot is paired with Table 1 supplying values of potential predictive criteria for each case, together with a measure of closeness to ground truth via Earth Mover Distance (EMD). No simple rule emerges. For example, pairs $(O_3,D_1)$ and $(O_3,D_2)$ are equal best with respect to network closeness and #AS hops, yet one is close to the ground truth and the other is not. Finally, collectively the examples show that the lowest level is not necessarily the best, contradicting the minimum based intuition focused on congestion.

In ground truth data, where MSP holds, estimates of the (unshared) mean estimator and ${\underset{¯}{\overset{^}{r}}}_P$ were typically found to be so close as to be practically indistinguishable (invisible in plots if we tried to show them). This is why, in the above, we have focussed on the minimum $\underset{¯}{r}$ only. The immediate implication is that the standard deviation of RTT over SLs would be exceedingly hard to measure as it would lie under measurement uncertainty. However the implications of this finding are even more serious. It means that the difference is negligible compared to the spread of the $L$ levels observed when MSP does not hold, as we saw in detail in the previous subsection. Thus, as long as the MSP problem is not fully mastered, it is pointless to imagine we can “split the levels” to assess variance, or anything finer grained, with data of this type.

There are six SLs where ground truth (GT) measurements are available, thanks to perfSONAR nodes deployed directly on ingress routers. A summary of these SLs and the number of $(O,D)$ pairs that monitor them is provided in Table 2 and in the Appendix. The number of $(O,D)$ pairs with measurements that traverse these SLs ranges from 4 to 12.

For these GT cases, in addition to our usual estimates based on RIPE Atlas traceroutes, we also calculate GT estimates according to (1), as described in the previous section. The performance of our estimates compared to the GT value is presented in Figure 8 in both relative (left plot as an accuracy) and absolute (center and right plots as an error in [ms]) terms.

To provide some perspective on the importance of careful estimator design, in the left plot in Figure 8 we add results for a third naive estimator, corresponding to (4) with ${\underset{¯}{\overset{^}{r}}}_u$ replaced by ${\underset{¯}{\overset{^}{r}}}_n:={\min }_j({\underset{¯}{r2}}_j-{\underset{¯}{r1}}_j)$, where subtracting before taking the minimum is a fatal mismatch to the bidirectional nature of the noise. A visualization of this phenomenon is shown in Figure 7.

Both the unpaired and robust estimators generate values that are within 9.5% of the ground truth, which improves on the naive estimate by between 40% and 88%. As the result is so poor for the naive estimator, we drop any further comparisons with it.

The central plot gives the same results reconfigured as average absolute error. As can be seen, the average error in each case is less than 5ms. The right hand plot isolates the worse of the individual $L$ (in the robust case) or $K$ (in the unpaired case) estimates available to form a worse case comparison. We see the worst case is only about twice the size of the average case. It is up to 10ms on SLs that have minRTTs on the order of 65ms.

Our view on these results is that given that the ground truth itself is not perfect (due to ICMP packet handling issues and others noises that could be responsible for much of the difference in level between different estimates), that our two estimators have comparable performance.

The relative improvement in the robust estimator in the worse-case compared to average results is consistent with its robust character. To expand on this, Figure 9 gives a specific example where the unpaired estimate ${\underset{¯}{\overset{^}{r}}}_u=69.01$, which focusses on the lower level after the LS, is worse than ${\underset{¯}{\overset{^}{r}}}_{SL}=69.27$ which averages the two levels, because the ground truth estimate is aligned with the upper level at $69.50$.

There are a total of 95 SLs that we can measure via RIPE Atlas for which we have no ground truth. We expect this to be by far the most common case in the future so it’s important to carefully consider the values that are reported and assess them relative to what we see in other measurements (including those with ground truth).

We provide details on all of the SLs that we can monitor in Table 3 in the Appendix. As illustrated in Figure 2, we find that our method applied to RIPE Atlas data identifies SLs with landing points on all continents (except Antartica) and with diverse deployment characteristics all over the world. Among all of these, there is relatively high coverage of transatlantic cable infrastructure (21 SLs monitored), which is a reflection of the dense deployment of RIPE Atlas nodes in Europe and the US. Transpacific coverage is also well represented in our data (13 SLs monitored). The remainder of the SLs are primarily intra-european and intra-asian. Perhaps not surprisingly due to the lack of RIPE Atlas nodes in these areas, coverage along the South American and African coasts is quite sparse, as is coverage of Oceania.

Examining the details of measurement coverage, we find two SLs (both transpacific) with 19 vantage points and up to 50 $(O,D)$ pair paths that traverse these. This is explained by the dense deployment of RIPE Atlas nodes on the California coast of the US. On average, there are about 8 $(O,D)$ pair paths that traverse SLs in our data. This means that in most cases, there is good measurement coverage on links that can be monitored. However, in the most limited cases, there is a single $(O,D)$ pair that can be used for monitoring – we find this to be the case on 21 of the SLs. While not inherently problematic, this does limit our ability to generate MLV values for those SLs. In closer examination of the RIPE Atlas data, this is due to our constraint of only selecting as vantage points anchors that are within 5.5ms of the ingress of the SL.

The values that we identify for minRTT range between about 5ms (e.g., the Circe North cable [18] between Zandvoort, NL and Lowestoft, GB shown in Figure 10), and 170ms (e.g., the Jupiter cable [19] between Daet, PH and Hermosa Beach, CA, US shown in Figure 10). We generally find a close correspondence between our unpaired and robust estimators for minRTT (less than 5ms difference) and manual checks with cable lengths indicate that the measurements are consistent with speed of light-based estimates [29]. This can be interpreted as indicating that most of these SLs have minimal noise profiles. However, in some instances, we find a somewhat significant difference. For example, on the Atlantic Crossing 1 cable [17] between Brookhaven, NY, US and Whitesands Bay, GB our unpaired estimator reports 66ms while the robust estimator reports 81ms. This SL like others with some disparity tend to appear on SLs where there is a single $(O,D)$ pair. In these cases, it’s impossible to generate an MLV to provide wider context.

Finally, turning our attention to MLV, these values vary widely over the monitored SLs. In general, they tend to be less than 10ms indicating a close correspondence in the timeseries of minRTT estimates from our robust estimator. This is a strong indicator of low noise on these SLs. Relatively higher MLV values (e.g., on the Pacific Crossing infrastructure [20], which includes MLVs of between 18ms and 73ms) indicate high levels of noise in timeseries of minRTT estimates and point to our robust estimator as a better predictor of minRTT.