Five Blind Men and the Internet:
Towards an Understanding of Internet Traffic

Prior work on traffic volumes exchanged at IXPs has typically relied on close collaboration with large exchanges such as AMS-IX and DE-CIX [5, 12, 1]. These studies showed that peak volumes at individual exchanges can reach tens of Tbps [1, 12, 20] and, thanks to fine-grained measurements, revealed distinct diurnal usage patterns in their regions. Using anonymized sFlow records, they also quantified protocol and application-specific contributions to traffic volumes [30, 22, 38]; for example, Feldmann et al. observed increased streaming and collaborative-tool usage during COVID-19 [16]. Overall, this line of work demonstrates that IXP traffic is diverse, representative, and central to Internet connectivity [13, 6].

However, such studies have key limitations: because access often depends on operator agreements, vantage points are concentrated in a few regions, datasets are not easily reproduced or extended by others, and achieving global, longitudinal coverage without proprietary access is difficult. These constraints motivate a complementary approach.

Many IXPs publicly advertise aggregate traffic statistics to attract members. These feeds are mostly volumetric time series updated frequently (typically every 5 minutes) and are available across regions. Because they are public, they are accessible and independently verifiable. Aggregating them can yield a broad, high-resolution, application-agnostic view of Internet usage patterns.

The challenge is practical rather than conceptual: formats vary (JSON APIs, HTML pages, and image-based plots), some feeds are federated, and care is required to avoid double counting and to normalize units, time zones, and sampling intervals. These realities shaped our framework’s design (§3).

It is important to note that we do not claim to see all, or even the most significant share of Internet traffic with this vantage. Like the other blind men, our vantage has its own blind spots (discussed in §7). Our goal is to capture the shape of Internet usage – its daily/weekly cycles, regional differences, and event-driven anomalies – rather than its absolute total. As we will show in §6, with sufficient coverage this vantage consistently captures characteristic regional habits and identifies atypical behavior in response to global events.

PeeringDB is a community-maintained database widely recognized as the authoritative source for information on IXPs. It catalogs publicly registered IXPs, their geographic locations, region classifications, participating networks (ASNs), those networks’ port capacities, and peering policies, serving as a valuable starting point for acquiring traffic volume data. PeeringDB also often provides URLs for IXPs that publicly share traffic statistics; however, we found this field to be sparsely populated and frequently inaccurate in practice. We therefore manually inspected the websites of all IXPs listed in PeeringDB. This multi-week effort yielded a curated list of URLs for publicly accessible traffic statistics pages, enabling continuous monitoring of IXP traffic volumes.

These traffic feeds provide time-series data on outbound and inbound traffic exchanged over an IXP’s public peering fabric, typically at five-minute intervals, consistent with the Open-IX OIX-1 standard [26, 25]. Discussions with 20 IXP operators confirm this practice represents total public peering volume and excludes private network interconnects (PNIs); accordingly, our analysis covers only the public fabric and does not capture PNIs. To avoid double-counting, we use inbound traffic, noting that at the fabric aggregate inbound and outbound volumes are equal (what enters must exit). Where both series were available, we observed they were virtually identical, with small differences attributable to packet loss within the fabric and measurement/reporting noise.

The one-time, labor-intensive process of identifying reliable traffic feeds is essential for enabling continuous and accurate extraction of IXP traffic volumes. Upon establishing these feeds, our system methodically extracts traffic volumes from a diverse array of interfaces and formats. Depending on the traffic feed, we employ one of the following three extraction methods:

API-based.

For some IXPs, we directly obtain their traffic volume data in JSON format at five-minute intervals via IXP Manager [21].

HTML-based.

For IXPs not interfacing with IXP Manager, some embed traffic volume data within webpage HTML or accompanying JSON files. We develop specialized parsers using regular expressions specifically crafted for each individual IXP to extract this data.

OCR-based.

For remaining IXPs, we extract traffic data from graphical plots within images. We use the Tesseract OCR engine [35] with two complementary techniques: digitizing plot axes and curve coordinates, and extracting explicitly rendered text values (e.g. “Current Traffic: X Gbps”), when available (see Figure 2).

While API and HTML methods are generally stable once configured, OCR is inherently more fragile. Potential errors include misinterpreting digits or missing decimal points. To mitigate these, we implement a cross-check: when textual values are available on the plot, we compare them with the values derived from digitizing the plot curve. If only plot digitization is possible, accuracy can be affected by image resolution. We evaluated the accuracy of the OCR approach on a test set of images containing both plots and explicit text values. The results revealed a maximum average deviation of 2.4% between the OCR-extracted text values and the values derived from plot digitization, confirming the method’s viability for our purposes.

The distribution of the 378 traffic feeds in our study across extraction methods is relatively balanced: 146 rely on OCR, 126 use IXP Manager APIs, and 106 employ HTML parsing. However, the share of traffic volume per method is skewed by operator size. Feeds accessed via the standardized IXP Manager API contribute only 7.2% of the total observed traffic volume, whereas OCR-based and HTML-parsed sources – typically bespoke portals used by larger, higher-traffic IXPs – account for roughly equal portions of the remainder. This suggests that larger IXPs more often expose traffic via custom web interfaces rather than default presentation methods.

Continuous, long-term data collection requires a robust and fault-tolerant infrastructure. We designed and deployed a distributed scraping framework to automate data retrieval, validation, and storage.

Our system achieves fault tolerance primarily through redundancy. The system maintains two nodes: the Main Node and the Backup Node. Both nodes orchestrate scraping tasks based on configured frequencies, execute appropriate scrapers (API, HTML, OCR), perform initial data sanity checks, and transmit validated data to a separate Data Node hosting the database. Careful job distribution prevents overload, ensuring smooth operation and consistent data quality.

While the Main Node actively performs these tasks, the Backup Node continuously monitors the Main Node’s health by checking data freshness on the Data Node. If the Main Node becomes unresponsive for longer than one polling interval plus a small grace window, the Backup Node automatically assumes scraping responsibilities. This failover mechanism was triggered 57 times during the operational period, during scheduled maintenance and routine patching of the scraping hosts. The swift automated transition minimizes data gaps, ensuring the continuity of our longitudinal dataset.

The Data Node serves as the central repository for IXP traffic volume data, scraper configurations, historical PeeringDB snapshots [10], and operational logs. It handles weekly data archiving and provides a Grafana-based interface for internal monitoring and visualization [19]. Data storage employs a TimescaleDB database (a time-series optimized PostgreSQL extension) [36]. This combination of redundancy, proactive monitoring, and specialized database architecture ensures robust, reliable, and scalable long-term data collection.

Given the lack of formal documentation for most public IXP traffic statistics, we assessed data quality along two key dimensions:

Resolution.

Most traffic feeds (88%) report data at five-minute intervals, aligning with the MRTG/RRD tool default and the MRTG standard used by IXP Manager. Another 9% report at ten-minute intervals, with the remainder at coarser intervals, 30 minutes being the coarsest we observed.

Timeliness.

Most traffic feeds provide near real-time data, but four exhibit notable delays. The Equinix federated feed (covering 42 IXPs globally) and three IX.br locations (Campo Grande, Cascavel, Boa Vista) are five-minute time series, yet values are published as batch backfills and at uneven intervals rather than continuously. Our inquiries regarding these delays remain unanswered.

Given the evident geographic and traffic volume disparities, how representative is our dataset of IXP infrastructure?

As a measure of representativeness, we divided the total port capacities of all the IXPs we were successfully able to measure in a region by the total port capacities of all the IXPs in that region. We plot these ratios in Figure 5 for each of our seven regions. Since IXPs updated their port capacities during our measurement period, we plot these ratios over the entire measurement period.

As we can see, there is strong evidence of representativeness. Globally, our monitored IXPs account for 87% of all publicly announced IXP port capacity listed in PeeringDB. This high capacity coverage remained remarkably stable over 2023–2024, decreasing by only 1.5 percentage points despite considerable growth in IXP traffic in this period. This high coverage persists regionally, with our framework observing more than 80% available port capacity in all regions. Ranked by coverage, Australia leads (98%), followed by South America (96%) and Africa (92%).

We note that the analysis presented in Figure 5 exclude 76 IXPs (out of 1065) lacking port capacity data in PeeringDB during our study period; manual checks suggest these are typically small, geographically dispersed, or inactive IXPs, with likely negligible impact on aggregate results.

Also, because PeeringDB is self-reported, we assessed the freshness and internal consistency of the capacity data used for this analysis. For all IXPs included in our capacity analysis, the port-capacity-related fields were updated at least once per year during 2023–2024, including IXPs without public traffic feeds, indicating that these records are actively maintained rather than stale. As a plausibility check, where both traffic and capacity were available we verified that observed peaks did not exceed announced capacities; and we found no systematic contradictions.

Between January 2023 and December 2024, aggregate IXP traffic exhibited substantial and consistent growth. We illustrate this trend in Figure 6, which sums the daily mean, peak (${95}^{th}$ percentile), and trough ($5^{th}$ percentile) traffic volumes across all traffic profiles to provide a comprehensive view of aggregate IXP traffic volume. The daily mean traffic rose from approximately 138 Tbps to 200 Tbps, a 49.2% increase over the two-year period. Peak and trough traffic experienced similar increases of 47.6% and 54.3%, respectively.

While substantial, growth was not uniform: average traffic increased by 23.4% in 2023 but by a more moderate 16.9% in 2024, based on a linear fit applied to each year. These figures align closely with independent observations (Cloudflare reporting 25.0% and 17.2% global increases, respectively) [2, 3], suggesting that IXP traffic is a strong proxy for studying overall Internet-traffic growth. Peak traffic followed a similar deceleration pattern (20.0% in 2023 vs. 13.6% in 2024), while trough traffic slowed less dramatically (27.7% vs. 22.2%).

Beyond growth rates, the day-to-day volatility of these metrics reveals distinct characteristics. Peak traffic exhibits the largest absolute daily fluctuations, with typical swings of $\pm$6.0 Tbps (standard deviation). Conversely, trough traffic is proportionally the most volatile, with daily changes averaging 5.1% relative to its mean – more than double the relative volatility of peak (2.4%) or average (2.0%) traffic. Average traffic is the most stable metric, with absolute fluctuations of $\pm$3.4 Tbps and the lowest relative volatility, making it ideal for tracking overall trends reliably.

Traffic growth varies significantly within each year of the study period. As evident in Figure 6, aggregate traffic growth remains relatively stagnant in the first half of the year (shaded in gray) but accelerates in the second half. For example, in 2023, average traffic grew by only 3.4% in the first half of the year, compared to 14.3% in the second half.

Regional analysis clarifies this trend. As illustrated in Figure 7, northern hemisphere regions closely follow the aggregate pattern, whereas South America exhibits the opposite trend. For instance, in 2023, South America experienced accelerated traffic growth of 42.5% in the first half of the year, compared to only 17.1% in the second half. Australia, another southern hemisphere region, showed balanced growth throughout the year. A common pattern emerges: traffic growth tends to accelerate during cooler months of the year, possibly due to increased indoor online activity during autumn and winter.

Short-term slowdowns appear around specific holidays and events, including the Christmas period in Europe and North America, New Year’s Eve/Day in North America, and Eid al-Fitr (April 21-23, 2023, and April 10-12, 2024), marking the end of Ramadan, in the Middle East. These slowdowns likely reflect reduced business activity and shifts in user behavior during holidays. Note that Figure 6 uses a seven-day rolling (weekly) window average for visual clarity, which smooths short-lived, extreme fluctuations; the unsmoothed series shows these holiday dips more prominently.

These examples demonstrate that, while broad seasonal cycles provide a baseline, specific cultural and regional events introduce predictable, short-term variations in aggregated IXP traffic data. Understanding both large-scale seasonal waves and short-term, event-driven ripples is crucial for accurately interpreting traffic dynamics, a theme further explored in §6.

Aggregate averages mask substantial regional heterogeneity (Figure 7). Europe, the largest region by observed volume, shows the most modest growth (15.5%) over two years, while Asia-Pacific (108%), Middle East (82%), North America (72%), Africa (68%), and South America (60%) grow markedly faster, reflecting rapid development, successful interconnection initiatives, or strong demand growth. Australia shows moderate growth (43%).

Given the disparities in observed traffic volumes across regions (§4.1), we investigated whether these different growth rates were genuine or artifacts of varying coverage and observed traffic volumes across regions. To do so, we analyzed how quickly each region’s aggregate growth pattern converges as more IXP data is included. Starting with the IXP contributing the least traffic in a region and progressively adding larger ones by volume, we calculated the Pearson correlation (denoted $R$) between the two-year growth trend of this cumulative partial aggregate and the growth trend of all observed IXPs in that region. The results are plotted in Figure 8.

For all regions except the Middle East and Australia, correlations increase steadily as more traffic is included, indicating high self-similarity within subsets of IXPs. In the Middle East, Manama-IX (Bahrain), a major contributor to regional traffic, experienced an abrupt 50% drop in average traffic (from 300 Gbps to 150 Gbps) around September 2023, likely due to a network disconnection or contract termination. This sudden shift in a single IXP, diverging from smoother regional trends (yellow line in Figure 8), delays cumulative correlation. A similar, less pronounced event occurred in Australia with Megaport Brisbane in early January 2024, possibly due to a contract termination. These disruptions, while skewing convergence metrics for the Middle East and Australia, reinforce confidence in the data’s fidelity, suggesting that, absent such events, convergence would likely occur earlier.

Table 2 summarizes the observed-traffic share needed to achieve $R>0.90$ and $R>0.95$ with the final regional trend. In most regions, growth trends stabilize remarkably quickly. For instance, Asia-Pacific and North America reach $R>0.95$ with less than 10% of observed traffic volume. South America and Europe require 57.8% and 56.8%, respectively, while Africa and Australia achieve $R=0.95$ at 35.0% and 48.1%. The Middle East, due to the Manama-IX disruption, requires 72.6% for both thresholds.

Analysis of IXP traffic at varying scales reveals a fundamental trend: the emergence of clear, repeating structures in usage patterns correlates strongly with the aggregate traffic volume. At lower volumes, typically below 1 Gbps, daily traffic profiles often appear as relatively unstructured fluctuations without a consistently repeating shape (top plot, Figure 9). As traffic volume increases and more individual user activities are aggregated, these random variations tend to average out, and distinct diurnal patterns reflecting collective human activity typically become apparent (middle plot). Once average traffic levels exceed approximately 10–15 Gbps, these daily and weekly patterns generally become well-defined and predictable (bottom plot).

To quantitatively assess when these fluctuations give way to repeating structures, we turn to information theory. We use Shannon entropy (a measure of randomness) to characterize daily traffic patterns. In our context, lower entropy indicates structured traffic patterns and the existence of diurnal cycles, whereas higher entropy reflects erratic and irregular usage throughout the day. More intuitively, lower entropy indicates a more structured traffic pattern, where usage is concentrated in distinct periods, characteristic of clear diurnal cycles. Conversely, higher entropy reflects an erratic distribution of traffic across the day, suggesting flatter usage profiles without pronounced daily peaks and troughs.

Plotting Shannon entropy against average traffic rate for each traffic profile confirms this trend (Figure 10). Entropy generally decreases as traffic volume increases. This demonstrates that, as IXPs aggregate more diverse user activities, the resulting collective behavior smooths into lower-entropy diurnal patterns. Figure 10 also reveals notable outliers: a cluster of vantage points characterized by both high traffic volumes and persistently high entropy. Manual investigation identifies several of these as major research and education (R&E) IXPs (e.g. CERN IX, PacificWave, and VIX) as depicted with red markers. To our knowledge, these are the only R&E IXPs represented in our dataset. This suggests that these profiles may capture significant volumes of machine-driven traffic, such as large-scale scientific data transfers or inter-campus backups. Unlike typical human-driven traffic, such automated flows can result in continuous or irregular bursts of data at any time, leading to a flatter traffic distribution and thereby maintaining higher entropy despite substantial volumes. This suggests that metrics like Shannon entropy could differentiate human-driven and machine-driven traffic in networks, potentially enabling passive detection of botnets or other large-scale malicious activity.

Across our dataset, the majority of our 378 traffic feeds from 472 tracked IXPs consistently exhibit diurnal patterns at sufficiently high traffic volumes, reflecting the rhythmic nature of Internet usage. This subsection examines how diurnal patterns vary across regions in our dataset.

To analyze traffic pattern shapes, we adjust each profile, including single and federated profiles that are clustered by timezone, to their local time. These adjusted profiles are then aggregated to derive regional traffic patterns.

To determine how representative an IXP’s traffic pattern is of its region, we conduct a detailed analysis – analogous to the growth trend stability analysis in §5.3 – assessing the consistency of temporal patterns across IXPs within each region. Specifically, starting with the IXP contributing the least traffic in a region and progressively adding larger ones by volume, we calculate the Pearson correlation between the weekly pattern of the growing partial aggregate and the final pattern derived from all observed IXPs in that region. Figure 11 shows the percentage of observed regional traffic required to achieve near-perfect correlation with the final, fully aggregated regional pattern.

The results reveal a striking contrast to the stabilization of growth trends. Characteristic weekly patterns emerge and converge almost immediately: in most regions, a correlation coefficient of $R>0.95$ is achieved with less than 2% of observed traffic – often with under 0.1%. Even $R>0.99$ typically requires sampling under 10%. This rapid stabilization demonstrates that the weekly “shape” of Internet usage is a strong, consistent signal within a region, captured by observing only a small fraction (approximately 10–15 Gbps) of its traffic in our dataset.

Figure 12 presents the average traffic profile for each day of the week, computed per region and normalized by the regional average traffic. These profiles reveal distinct regional rhythms and highlight consistent contrasts between weekdays and weekends.

The inherent stability of daily and weekly usage patterns, established in the preceding analyses, provides a robust baseline against which significant deviations can be identified. Such deviations can signal anomalous network events driven by impactful real-world occurrences. Building on this observed pattern regularity, we identify anomalous days by detecting substantial variations in their daily traffic shape compared to a region’s typical profile for that day across other weeks. Once an anomalous day is flagged, its intra-day traffic curve is examined against the regional average to pinpoint specific event timings and quantify their impact. For this paper, we validate and select outliers manually; nonetheless, we observe hundreds of such candidates across regions. Work is underway on an automated, scalable pipeline to discover, validate, and characterize these events systematically.

Our dataset reveals numerous instances where aggregated IXP traffic patterns clearly reflect the impact of major societal events. We present a manually curated subset in Figure 13, contrasting the traffic profile on the event day with typical patterns. For instance, in Africa, traffic on July 26, 2024, deviated significantly from the regional norm, coinciding with the Olympic Games opening ceremony, with observed traffic peaking at twice the typical Friday evening level during ceremony hours. A similar impact was observed in South America on August 7, 2024, during the Olympics Women’s football semi-final (Spain vs. Brazil), a popular derby, where traffic during the match hours rose to nearly twice the usual volume for that period. The A-League Men’s Grand Final in Australia (May 24, 2024) also produced a noticeable, albeit more moderate, traffic increase of 20-25% during match time (dashed line).

Gaming events also leave a clear imprint. In Australia, a major Fortnite update on November 30, 2024, led to sustained traffic elevation on December 1, 2024, with peak download and gaming hours showing traffic levels over twice the typical Sunday afternoon level. This event also had a significant impact in Europe on December 1, 2024, with observed traffic during peak gaming periods approximately 10% above typical Sunday levels. The re-release of Fortnite’s “OG” mode on December 6, 2024, had a further widespread impact, visibly elevating traffic in the Middle East (approximately 25% above the typical baseline during evening gaming) and similarly affecting North America (around 20% increase during peak engagement hours). E-commerce events, such as Singles’ Day on November 13, 2024, in Asia-Pacific, resulted in sustained traffic elevation throughout the day, approximately 30-40% above the regional average for a comparable weekday. Reality television, such as the Big Brother Brasil finale on April 4, 2024, also drove a significant traffic increase in our dataset.

These diverse examples demonstrate that aggregated IXP traffic patterns are highly sensitive and responsive to a wide range of real-world events that drive collective online activity. The ability to detect and quantify these deviations validates the richness of IXP traffic data beyond simple volume metrics. Such pattern deviation analysis not only provides insights into the network-level impact of different societal phenomena but also highlights the potential for using this approach in near-real-time monitoring to identify widespread service-demand shifts or other significant network anomalies. Understanding the shape of traffic, therefore, offers a complementary and often more nuanced view of Internet dynamics compared to volumetric analysis alone.