Passive Data-Plane Telemetry to Mitigate Long-Distance BGP Hijacks

A BGP hijack occurs when a malicious or compromised AS falsely advertises routes to IP prefixes it does not own or cannot reach, misleading other ASes into rerouting traffic through its infrastructure. Suppose $AS100$ legitimately owns the IP prefix $\mathrm{1.1.1.0}/24$. A malicious AS, $AS200$, falsely announces ownership of $\mathrm{1.1.1.0}/24$ to its BGP peers. These peers may accept the announcement as valid and propagate it to their own peers, spreading the false route across the network. As a result, traffic destined for $\mathrm{1.1.1.0}/24$ – for instance, originating from another prefix like $\mathrm{2.2.2.0}/24$ owned by $AS300$ – may get misrouted to $AS200$ instead of reaching $AS100$. This enables the attacker to eavesdrop on, fingerprint, manipulate, or drop this illegitimately-obtained traffic. In some cases, the attacker may even serve malicious content by impersonating the legitimate destination (e.g., by acquiring a valid certificate first by exploiting weakness in the certificate issuance verification process [8]).

A BGP interception attack (Figure 1) is a specific type of BGP hijack where the attacker intercepts traffic but forwards it to the original destination, enabling analysis or manipulation while remaining undetected by the end hosts. Of particular concern is a stealthy subset of these attacks where a sophisticated attacker employs techniques like AS-path poisoning and the manipulation of BGP communities to limit the propagation of malicious announcements in a bid to evade detection by BGP monitors near the victim. For instance, in the example above, $AS200$ could manipulate its announcements to propagate only to routers near $AS300$ while suppressing those to routers near $AS100$. This could cause traffic from $\mathrm{2.2.2.0}/24$ to $\mathrm{1.1.1.0}/24$ to get misrouted via $AS200$, while $AS100$ remains unaware as BGP monitors near it never observe the malicious route. Such an attack has been demonstrated by Birge-Lee et al. [9], which we ethically reproduce in §7.2.

We use the Natural Earth Admin-0 Countries dataset – one of the most popular boundary datasets in the Geographic Information System community. It provides high-resolution boundary coordinates (avg. $2.5$ km) for 258 countries (Figure 2(a)) [16]. Since some countries consist of multiple disconnected regions (e.g., contiguous US plus Alaska and islands), we distinguish each country’s mainland (largest contiguous landmass) from entire country (all regions combined).

For each country (entire country and mainland), we calculate: (1) Max. intra-country distance: the largest pairwise great-circle distance among all boundary points of the country, and (2) Min. inter-country distance: the smallest great-circle distance from any point in this country to any point in another. Figure 2(b) illustrates these distances within the US (top) and between the US and China (bottom).

Figure 2(c) shows: (1) min. inter-country distances far exceed max. intra-country distances, and (2) these distances for entire countries vs. mainlands are similar: we proceed with mainlands in the rest of this paper for better interpretability. At the speed of light, the ${25}^{th}$/${50}^{th}$/${75}^{th}$ percentile max. intra-country one-way delay (OWD) are 0.2 ms, 2.1 ms, and 5.6 ms, respectively. The corresponding values for min. inter-country OWD are 20 ms, 38 ms, and 57 ms – at least an order of magnitude larger.

Ideal conditions include: (1) Ideal network, i.e., data travels at the speed of light, and (2) Ideal measurements, i.e., our measurements capture the actual distance-based propagation delay. Under these assumptions, propagation delay = minimum RTT (minRTT) = RTT. Therefore, in this subsection, RTT is synonymous with propagation delay. We relax these assumptions in the next subsection where we analyze real-world measurements (§3.5).

Optimal attacks determine the lower bound of our defense capabilities. To assess defendability under optimal attacks, we compute propagation delay as the round-trip distance (${\delta }_{pre}^{\ast }$ or ${\delta }_{mid}^{\ast }$) divided by the speed of light. To illustrate the difference in defendability across countries, we select two examples: Russia, which has among the smallest median ${\tau }_{deviation}^{\ast }$, and New Zealand (NZ), which has one of the largest median ${\tau }_{deviation}^{\ast }$. Figure 4(a) shows their pre-attack and mid-attack RTT distributions. NZ’s RTTs increase significantly mid-attack, making it more defendable; Russia’s RTT changes are smaller making it less defendable. In general, small countries with distant neighbors (low $\delta (S,D)$, high $\delta (S,A)+\delta (D,A)$) are the most defendable, while large countries with many close neighbors are the least defendable.

To quantify defendability, we define attack coverage as the percentage of optimal attacks that can be detected under some given condition. To compute overall coverage, we set the condition ${\tau }_{deviation}^{\ast }\ge 5$ ms because: (1) 5 ms far exceeds typical noise in measurements (e.g., due to coarse-grained timestamps, rounding off errors, approximations in distance calculations, etc.), and (2) At speed of light, 5 ms corresponds to approx. 1,000 km of extra path length, so it captures significant geographic detours. The attack coverage for Russia (over 257 optimal attacks) is 85% (the minimum for any country), while the same for NZ is 100%. When expanded to all countries (i.e., 258 x 257 optimal attacks), the coverage is 96.6%. Figure 4(b) shows that 100% (i.e., all) countries can be defended against 84% attacks, 75% against 95%, 23% against 99%, and 11% against 100%. These results illustrate the promise and generality of propagation delay-based detection (in ideal conditions).

To analyze the extent to which optimal attacks increase RTT in ideal conditions, we plot the attack coverage (x-axis) given the ratio between mid- and pre-attack RTT (Figure 4(c)). For NZ (purple), this ratio ranges from 1-100x (5-190 ms absolute difference), while for Russia (green), due to its large pre-attack RTTs, it is 1-4x (5-110 ms). For attacks on all countries (red), it is 1-198x (5-200 ms). The ratio is 2x (8 ms) for 95% attack coverage on all countries, and 4x (23 ms) for 85% coverage.

The analysis of cross-country attacks does not inform us directly about the relationship between distance and RTT deviation. To bridge this gap, Figure 4(d) shows RTT deviation (z-axis) – as a function of pre-attack distance ($\delta (S,D)$) (x-axis), and average distance between S-A and D-A (y-axis) – in a heatmap. The x- and y-axis are capped at the maximum intra- and minimum inter-country distances. The ${85}^{th}/{95}^{th}$ percentile pre-attack distances are 1,090 km and 1,872 km; to induce an RTT deviation of 25 ms, the attacker needs to be at an average distance of 3,045 km and 3,436 km, respectively, from S and D.

Our datasets are outlined below:

1. 1.

   Campus dataset: We collect traffic from 7.5 M TCP flows on our campus over 12h on a weekday in May 2022, and compute RTTs by matching data packets with ACKs [14].

2. 2.

   MLab dataset: We collect minRTTs of 4.3 M TCP flows from NDT7-based measurements over 5 days in Dec. 2024 [25].

For each flow in each dataset, we collect geolocations of the source and destination. We discard flows whose minRTT indicates shorter distances than those permitted by their reported geolocations, which is physically impossible. Finally, we group remaining measurements by source-destination /24 prefixes, because (1) a /24 prefix is the smallest unit on which a BGP hijack can be launched, and (2) aggregating by prefix improves the chance of measuring true minRTT (see §5.2 for a more detailed discussion on the advantage of aggregation by prefix).

To quantify defendability in realistic settings, we estimate real-world propagation delay between any two hosts from their great-circle distance $d$. This real-world propagation delay may vary across host pairs separated by the same distance $d$ depending on: (1) Geolocation: Some regions in the world have denser network connectivity than others, (2) Routing policies: Some providers make shorter paths available than others, (3) Consistent queues: Some paths experience consistent queuing delay due to deep buffers and consistent traffic, etc. Furthermore, even if the true propagation delay is same, we may measure different minRTTs due to transient congestion. It is infeasible to collect reliable minRTTs from all possible attack locations in all 258 countries. Instead, we apply the following method to both our campus dataset (215 countries) and Google MLab (234 countries) to capture variability:

1. 1.

   Bin distances: Divide all distances up to 20,075 km (Earth’s diameter) into 200 km bins ($\approx$1 ms at $c_f$).

2. 2.

   Assign prefixes to bins: For each source–destination prefix pair, compute its great-circle distance, assign it to the appropriate bin, and record its minOWD (minRTT/2).

3. 3.

   Percentile computation: Within each bin, compute the $p$th percentile of these minOWDs for $p=1,\mathrm{\dots },100$.

4. 4.

   Regression fitting: Fit one linear regression per percentile across all bins (e.g., a $p$=1 line through every bin’s $1$-percentile).

Figures 5(a) and 5(b) plot the $p$=25 and $p$=75 regression lines for the campus and MLab datasets, respectively. The campus data shows a narrower inter-quartile range – likely because the location of one end is fixed (on campus) and the entire variability is due to the remote host – whereas for MLab, the servers and clients are in different locations. With these delay estimates, we proceed to evaluate defendability against optimal attacks under realistic conditions.

The variability of minOWDs for the same distance poses a challenge: if our detection is unlucky, it could measure a higher percentile minRTT before the attack and a lower percentile during the attack, causing the deviation in minRTT to be much lower than speed-of-light deviation. To model such a scenario, we use the upper quartile (${75}^{th}$ percentile) minOWDs to estimate pre-attack minRTTs, and the lower quartile (${25}^{th}$ percentile) minOWDs to estimate mid-attack minRTTs. Then, we evaluate defendability using the same metrics as before.

Using the condition ${\tau }_{deviation}^{\ast }\ge 5$ms, the overall attack coverage is 91% on the campus dataset and 86% on MLab. Figure 5(c) shows that in the campus data, 100% countries can be defended against 63% attacks, 75% against 89%, and 2% against 100%. In MLab, 100% countries can be defended against 53% attacks, 75% against 80%, and <1% against 100%. Real-world defendability is therefore less than in the ideal case – especially in MLab, where minRTT variability is higher. Figure 5(d) plots coverage versus the mid-/pre-attack minRTT ratio. The ratio ranges from 1–12x (5–290 ms) for campus; 1–7.5x (5–250 ms) for MLab.

The user provides HiDe with the IP prefix of the home network and the threat regions they want to protect their data from, based on policy decisions or anticipation of threats. The threat regions are either names of countries or enclosed polygons of geographic coordinates. HiDe also obtains from its data plane the destination prefixes observed by it. The user can, optionally, set threat regions per destination prefix. Eventually, the control plane converts all the information into triplets of geolocation information: {source_coordinates, destination_coordinates, threat_coordinates_list} using public geolocation services (IPinfo, MaxMind) and public geographic datasets (Natural Earth Admin-0) [16, 24, 41].

For each location triplet, we first identify the optimal attack, i.e., the attacker’s location in the threat region that minimizes mid-attack round-trip distance. Note that this distance can be much higher than in the optimal attacks in §3 because there, source and destination were always on the victim country’s border whereas here, they are almost always inland. Next, we compute the minimum possible mid-attack RTT for this optimal attack (${\tau }_{mid}^{\ast }$), based on the speed of light. We designate this lower bound RTT as the absolute threshold of our changepoint detector: HiDe flags an attack whenever the observed minRTT reaches ${\tau }_{mid}^{\ast }$, guaranteeing zero false negatives (see §5.6 for more on false positives). While we could choose a less conservative bound – e.g., the ${25}^{th}$-percentile estimated latency in 200 km buckets (§3.5) – we opt for the most conservative threshold to guarantee protection, at the expense of coverage, as is our design goal (§4).

BGP attacks target prefixes, with a $/24$ prefix being the smallest possible target. All flows to an attacked prefix experience the same change in propagation delay, but noisy RTTs in individual flows can obscure this change. We aggregate RTT samples by prefix before computing the minimum RTT per window (discussed next), as at least one flow per window is likely to produce a sample representative of the true propagation delay. Figure 7 demonstrates this for a US-based destination prefix with one noisy and one stable flow. Also, prefix-level aggregation reduces switch memory requirements from per-flow to per-prefix, which is significant.

We divide streams of per-prefix RTT samples into non-overlapping time windows of a fixed size (i.e., tumbling windows) and compute the minRTT in each window. This filters out short-term RTT spikes due to benign confounding factors like queuing delay from short-lived congestion, end-host processing delays, and TCP oddities like delayed ACKs [36]. We select a sub-second (e.g., 100 ms) time window – while minRTT can be measured more reliably with longer time windows, it would delay mitigation allowing an attacker more time to complete their attack. Finally, tracking minRTTs in non-overlapping tumbling windows requires only per-flow state, as opposed to overlapping sliding windows, making it more suitable for a switch implementation.

BGP interception attacks primarily target prefixes that host sensitive services – government sites, banking portals, cryptocurrency nodes, and the like – because such websites handle sensitive data from users around the world. The attacker places itself between the user and the server – intercepting valuable data. HiDe prioritizes these vulnerable server prefixes for protection. By default, it excludes prefixes used exclusively by WiFi or cellular access networks – since they rarely host critical services – unless the operator explicitly includes them.

Some destination prefixes have RTTs that are consistently noisy or high even under benign conditions, making it nearly impossible to detect interception attacks on them from certain threat regions without excessive false positives. For example, Figure 8 shows a prefix where the minRTT often exceeds 100 ms, making it impractical to defend against a threat region that causes a small deviation in comparison. Further analysis of such prefixes reveals that often, they tend to be associated with client-side access networks, such as cellular or WiFi, which HiDe does not defend by default anyway. Concretely, during a profiling phase independent of the detection phase, we monitor the max. of min. RTTs in tumbling time windows for each destination prefix. Later, we defend a prefix against a threat region only if ${\tau }_{mid}^{\ast }-max(RT{T}_{min})>\lambda$, where $\lambda$ is called the surge threshold. $\lambda$ defines the min. increase in $RT{T}_{min}$ required between two consecutive windows to flag an attack, and can be set to a constant (e.g., 10 ms) or a fraction of $max(RT{T}_{min})$ (e.g., 10%). A lower $\lambda$ provides broader coverage but increases susceptibility to false positives, and vice-versa. Users can adjust $\lambda$ based on their desired trade-offs. We evaluate the false positive rate for different values of $\lambda$ in §8.

The user configures the control plane by providing the network prefix of their home network (source prefix) and specifying the threat regions (optionally, per prefix).

The auto-tuning component translates the user inputs and destination prefixes read from the data plane into corresponding geographic coordinates and computes the ${\tau }_{mid}^{\ast }$. It also retrieves traffic statistics (specifically, $min(RT{T}_{min})$ and $max(RT{T}_{min})$) from the data plane for each prefix. Combining this information, the component identifies which prefixes can be effectively protected and generates changepoint detection parameters for those prefixes, which it then sends to the switch controller. For prefixes that cannot be protected, it provides the user with a summary listing each prefix and their corresponding $RT{T}_{min}$ statistics.

The switch controller translates the received parameters into corresponding match-action rules and installs them on the data plane. It also receives attack alarms from the data plane and (optionally) notifies the user.

We leverage Dart, an existing system, to generate accurate RTT measurements per flow from all the traffic observed by the switch. Dart achieves this at scale, handling a large number of flows without missing any RTT samples by efficiently managing switch resources [36].

The next component aggregates RTT samples per destination prefix, breaks them down into time windows, and calculates the minimum RTT per window. Additionally, it computes traffic statistics like $min(RT{T}_{min}$ and $max(RT{T}_{min})$ per prefix to share with the control plane.

The data plane performs changepoint detection using our two-window algorithm, minRTTs per window per prefix, and parameters installed by the control plane.

When an attack is detected, the data plane rate limits or blocks the corresponding prefix – depending on operator configuration – and raises an alarm.

Optionally, if the operator seeks more confidence in the detection result, and HiDe is not being used in conjunction with control-plane-based approaches, it can craft and send active probes periodically to determine whether the detection was a false positive. If so, it unblocks the prefix.

The switch control plane installs per-prefix match-action rules specifying the window size ($W$), absolute threshold (${\tau }_{mid}^{\ast }$), and surge threshold ($\lambda$). If enabled, it listens on the CPU port for packets from the data plane containing information about either attack detections or non-coverage, and notifies the user. Additionally, it configures the Tofino’s packet generator to send active probes during the false positive correction phase.

We leverage DART for continuous and accurate per-flow RTT computation [36], and utilize packet mirroring, a native feature that replicates packets, to enhance its functionality. First, the original packet is forwarded without added latency, with the mirrored copy used for RTT computation. Second, RTT samples generated by DART are passed to HiDe-specific data-plane components.

We maintain a prefix table in register memory to support changepoint detection. The table uses a prefix signature, derived by hashing the first 24 bits of the external IP, as the key. The stored values include the prefix’s start timestamp, timestamp of most recent RTT, start timestamp of current window, number of RTT samples in current window, minRTTs for the current and previous windows, attack status, $max(RT{T}_{min})$, and $min(RT{T}_{min})$. The table accommodates up to 65,536 active prefixes, significantly exceeding the peak observed in our 12h campus trace (approx. 5K assuming a 5-second timeout), minimizing hash collisions. For collisions, we use cuckoo hashing [28]: the new prefix replaces the old one, which is loaded into memory, checked for timeout, and recirculated to a new index using a different hash seed if still valid. Each insertion allows up to 3 recirculations.

When an RTT sample is generated for a prefix, HiDe checks the status of the corresponding time window. If the window is not full, it updates the most recent timestamp, increments the RTT count, and replaces the current minimum RTT if the new sample is smaller. If the window is full, the current minimum RTT replaces the previous window’s minimum, and $max(RT{T}_{min})$ and $min(RT{T}_{min})$ are updated. If the window is valid (i.e., it has enough samples), HiDe evaluates the surge conditions and, if those are satisfied, starts the mitigation process by blocking all non-ICMP packets from/to the prefix by adding it to a block table.

Simultaneously, we start crafting and sending ICMP echo packets to the latest IP seen from the prefix and listening to responses to monitor its RTT. If a false positive is detected, the prefix is removed from the block table.

We analyze the resource usage of our prototype by function and find that its low resource consumption leaves ample resources for other concurrent switch functions (Table 1).

As outlined before, we captured 12 hours of production traffic – covering 1 pm to 1 am local time to include global working hours – at the edge of our US-based campus network using a TAP device near the gateway router. Packets – only TCP headers, anonymized at source in a prefix-preserved manner (discussed in Appendix A) – from selected subnets were mirrored and recorded on a collection server with tcpdump.

The dataset comprises $1.1$ TB of trace data representing $5.32$ TB of packet bytes, encompassing $19$ billion packets, $7.5$ million flows, and $238$ million RTT samples. It includes $12$K unique internal IPs and $324$K unique external IPs, distributed across $183$K external prefixes, $23.2$K of which are based in the US.

We set up our experiment using three key components: (1) a host on our campus running iperf3, (2) a high-speed programmable switch on campus where HiDe is deployed to monitor traffic, and (3) a transparent TCP proxy on an Amazon AWS instance. The proxy, which doubles as a PEERING node, advertises a /24 prefix allocated to our experiment. PEERING provides distributed ASes for controlled, real BGP announcements [34]. We use one IP address from the /24 pool, applying iptables rules on components 1 and 3 to masquerade it as the application’s IP. This setup enables HiDe to monitor all experimental traffic while allowing external adversaries to ethically launch BGP interception attacks on the /24 prefix.

We deploy AWS instances in geographically diverse locations, including the US east and west coasts, Europe, and Asia. The iperf3 server runs on the protected host across multiple ports, while clients on the distributed AWS instances connect to the PEERING IP of the transparent proxy, which forwards traffic to the server. The remote hosts are indicated as component 4.

The final step in our setup is launching ethical BGP interception attacks on the PEERING IP. We designate the PEERING node in Amsterdam as the attacker (component 5) and implement a stealthy interception attack using the technique by Birge-Lee et al., which employs BGP communities to control the blast radius of the attack [9]. The attacker advertises the same /24 prefix as the transparent proxy (an equally-specific attack), redirecting traffic from nearby nodes to Amsterdam. The attacker then forwards the intercepted traffic to the transparent proxy, leaving both sender and receiver unaware of the attack.

In accordance with HiDe’s coverage strategy, we only defend vulnerable (CISO) prefixes, i.e., external prefixes associated with a server. As described earlier, we identify such prefixes using a TCP port number-based heuristic. 16.8K US-based external prefixes match this condition in our campus dataset.

From external server prefixes, we further select those that were active for at least 10 minutes out of 12 hours (so we profile on at least 5 mins of data). We determine this by dividing the 12-hour period into buckets of 1 min, and checking which prefixes generated an RTT sample in at least 10 such buckets. 6K US-based external server prefixes are retained after this step. In a real network, the operator could profile prefixes for as long as needed to ensure it covers typical variation of RTT during benign operation – without any excess overhead since the profiling happens directly in the switch. For the following analysis, we make the assumption that the campus data captured during the 12-hour period did not experience any long-distance interception attacks (i.e., no true positives were present), so if HiDe detects a prefix it must be a false positive. We report our results based on optimal attacks from all 257 non-US threat countries.

Some US-based prefixes are not defendable against certain threat regions because during the profiling phase, they exhibit a $min(RT{T}_{min})$ larger than the corresponding ${\tau }_{mid}^{\ast }$ (i.e., measured delay without diversion is always higher than the lower bound with diversion). This could be because the threat region is geographically too close or because the network is always congested. Figure 12(a) shows that, based on this condition, HiDe can cover 99% prefixes against 78% attacks and 75% prefixes against 99% attacks. The covered prefix-threat country pairs are considered in subsequent experiments.

For different values of the surge threshold ($\lambda$), Figure 12(b) shows HiDe’s coverage based on defendability – i.e., whether the mid-attack lower bound RTT clears the pre-attack $max(RT{T}_{min})$ by at least $\lambda$ ms. At $\lambda$ = 5 ms, 25 ms, 50 ms, and 75 ms respectively, HiDe can cover 99% prefixes against 91%, 64%, 31%, and 7% attacks, respectively. This illustrates the trade-off between surge threshold and coverage.

By focusing on defendable prefixes, we achieve a false positive rate of approximately $0.012\%$ at worst, as shown in Figure 12(c). For higher surge thresholds (i.e., 30 ms), it drops to zero.

We estimate the likely downtime from a false positive by measuring how long (in multiples of time window size) it takes for the minRTT to return to normal for a falsely detected prefix. Since our (optional) active probing sends one probe per window, we expect similar results in reality. The median downtime is only 0.75 seconds.

We run the iperf3 server on our campus and the transparent proxy in Ireland, who forwards all traffic it receives to our campus via our prototype. The iperf3 clients are in Virginia (2 flows from the same prefix), Ohio, and Mumbai. The prefix in Virginia is hijacked from Amsterdam, causing Ireland to send traffic to Amsterdam instead of Virginia. Amsterdam then forwards the traffic to Virginia. The traffic takes the following round-trip route before the attack: Virginia (via PEERING infrastructure) to Ireland to our campus to Ireland to Virginia (via PEERING infrastructure) and the following one during the attack: Virginia (via PEERING infrastructure) to Ireland to our campus to Ireland to Amsterdam (via. PEERING infrastructure) to Virginia (via PEERING). Due to limitations of where we can deploy a Tofino switch on live traffic and a lack of diversity in the PEERING topology, we are restricted to this complex setup. The attack takes effect at $25$ seconds, as can be observed from the abrupt rise in RTT (blue dots) in Figure 13.

Using multiple runs of traceroute, we estimate the lower bound of RTT as approx. 190.5 ms before attack and 199 ms during attack (absolute threshold). We set the window size to 250 ms and the surge threshold to 5 ms. Based on the minimum RTTs (orange triangles), we detect the attack in 500 ms (red star).