Document Open Access Logo

Simulate Before Sending: Rethinking Transport in Datacenter Networks

Authors Dan Straussman , Isaac Keslassy , Alexander Shpiner , Liran Liss

PDF

File

Thumbnail PDF
PDF
OASIcs.NINeS.2026.19.pdf
  • Filesize: 4.46 MB
  • 22 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Networks → Data center networks
  • Networks → Transport protocols
Keywords
  • Datacenter networks
  • transport protocols
  • AI training
  • lossless networks

Metrics

Abstract

Existing transport protocols in commodity datacenter networks struggle to provide low collective completion times (CCTs) to AI training collectives, as packet losses and retransmissions significantly degrade performance.
We propose dcSim, an efficient transport that achieves low CCTs and practically lossless performance with commodity switches. In dcSim, each packet first employs a small simulation probe to traverse the network and explore congestion along a candidate path. Only packets whose simulation probes succeed are then transmitted, expecting to succeed as well. Evaluations confirm that dcSim achieves faster CCTs than existing schemes, with small queues and virtually zero packet loss. Finally, dcSim also excels in adverse conditions, including oversubscribed topologies.

Cite As Get BibTex

Dan Straussman, Isaac Keslassy, Alexander Shpiner, and Liran Liss. Simulate Before Sending: Rethinking Transport in Datacenter Networks. In 1st New Ideas in Networked Systems (NINeS 2026). Open Access Series in Informatics (OASIcs), Volume 139, pp. 19:1-19:22, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/OASIcs.NINeS.2026.19

Author Details

Dan Straussman
  • Technion, Haifa, Israel
Isaac Keslassy
  • Technion, Haifa, Israel
  • University of California Berkeley, CA, USA
Alexander Shpiner
  • Nvidia, Yokneam, Israel
Liran Liss
  • Nvidia, Yokneam, Israel

Funding

  • Keslassy, Isaac: This work was partly supported by the Louis and Miriam Benjamin Chair in Computer-Communication Networks.

References

  1. D. Abts and J. Kim. High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities. Synthesis Lectures on Computer Architecture. Springer International Publishing, 2022. URL: https://books.google.ca/books?id=NYZyEAAAQBAJ.
  2. Sharon Adarlo. Amazon Is Building a Gigantic Computing Facility to Match the Human Brain. https://futurism.com/the-byte/amazon-anthropic-ai-data-center, 2025.
  3. Saksham Agarwal, Qizhe Cai, Rachit Agarwal, David Shmoys, and Amin Vahdat. Harmony: A congestion-free datacenter architecture. In Usenix NSDI, pages 329-343, 2024. URL: https://www.usenix.org/conference/nsdi24/presentation/agarwal-saksham.
  4. Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. A scalable, commodity data center network architecture. In ACM SIGCOMM, pages 63-74, 2008. URL: https://doi.org/10.1145/1402958.1402967.
  5. Mohammad Alizadeh, Shuang Yang, Milad Sharif, Sachin Katti, Nick McKeown, Balaji Prabhakar, and Scott Shenker. pFabric: Minimal Near-Optimal Datacenter Transport. In SIGCOMM, 2013. Google Scholar
  6. Wei Bai, Shanim Sainul Abdeen, Ankit Agrawal, Krishan Kumar Attre, Paramvir Bahl, Ameya Bhagat, Gowri Bhaskara, Tanya Brokhman, Lei Cao, Ahmad Cheema, et al. Empowering Azure Storage with RDMA. In Usenix NSDI, 2023. Google Scholar
  7. Matthias Bastian. OpenAI’s Stargate pivot highlights rift with Microsoft over future AI computing needs. https://the-decoder.com/openais-stargate-pivot-highlights-rift-with-microsoft-over-future-ai-computing-needs/, 2025.
  8. Tommaso Bonato, Abdul Kabbani, Daniele De Sensi, Rong Pan, Yanfang Le, Costin Raiciu, Mark Handley, Timo Schneider, Nils Blach, Ahmad Ghalayini, et al. FASTFLOW: Flexible Adaptive Congestion Control for High-Performance Datacenters. arXiv preprint arXiv:2404.01630, 2024. Google Scholar
  9. Tommaso Bonato, Abdul Kabbani, Ahmad Ghalayini, Michael Papamichael, Mohammad Dohadwala, Lukas Gianinazzi, Mikhail Khalilov, Elias Achermann, Daniele De Sensi, and Torsten Hoefler. REPS: Recycled entropy packet spraying for adaptive load balancing and failure mitigation. In EuroSys, 2026. Google Scholar
  10. Qizhe Cai, Mina Tahmasbi Arashloo, and Rachit Agarwal. dcPIM: Near-optimal proactive datacenter transport. In ACM SIGCOMM, pages 53-65, 2022. URL: https://doi.org/10.1145/3544216.3544235.
  11. Inho Cho, Keon Jang, and Dongsu Han. Credit-scheduled delay-bounded congestion control for datacenters. In ACM SIGCOMM, pages 239-252, 2017. URL: https://doi.org/10.1145/3098822.3098840.
  12. Advait Dixit, Pawan Prakash, Y Charlie Hu, and Ramana Rao Kompella. On the impact of packet spraying in data center networks. In IEEE Infocom, pages 2130-2138, 2013. Google Scholar
  13. Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. RDMA over Ethernet for distributed training at Meta scale. In ACM SIGCOMM, pages 57-70, 2024. Google Scholar
  14. Peter X. Gao, Akshay Narayan, Gautam Kumar, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker. pHost: Distributed near-optimal datacenter transport over commodity network fabric. In ACM CoNEXT, pages 1-12. ACM, 2015. URL: https://doi.org/10.1145/2716281.2836086.
  15. Barak Gerstein, Mark Silberstein, and Isaac Keslassy. Making congestion control robust to per-packet load balancing in datacenters, 2025. arXiv:2509.07907. URL: https://doi.org/10.48550/arXiv.2509.07907.
  16. Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn. RDMA over commodity ethernet at scale. In ACM SIGCOMM, pages 202-215, 2016. URL: https://doi.org/10.1145/2934872.2934908.
  17. Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew W Moore, Gianni Antichi, and Marcin Wójcik. Re-architecting datacenter networks and stacks for low latency and high performance. In ACM SIGCOMM, pages 29-42, 2017. URL: https://doi.org/10.1145/3098822.3098825.
  18. Jinbin Hu, Jiawei Huang, Wenjun Lv, Yutao Zhou, Jianxin Wang, and Tian He. CAPS: Coding-based adaptive packet spraying to reduce flow completion time in data center. IEEE/ACM Transactions on Networking, 27(6):2338-2353, 2019. URL: https://doi.org/10.1109/TNET.2019.2945863.
  19. Shuihai Hu, Wei Bai, Gaoxiong Zeng, Zilong Wang, Baochen Qiao, Kai Chen, Kun Tan, and Yi Wang. Aeolus: A building block for proactive transport in datacenters. In ACM SIGCOMM, pages 422-434, 2020. URL: https://doi.org/10.1145/3387514.3405878.
  20. Gadi Hutt and Bob Evans. AWS Launches Project Rainier: Massive AI Supercomputing Cluster for Anthropic. https://podcasts.apple.com/bb/podcast/aws-launches-project-rainier-massive-ai-supercomputing/id1437752008?i=1000717513092, 2025.
  21. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. Google Scholar
  22. Zewen Jin, Shengnan Wang, Jiaan Zhu, Hongrui Zhan, Youhui Bai, Lin Zhang, Zhenyu Ming, and Cheng Li. Bigmac: A communication-efficient mixture-of-experts model structure for fast training and inference, 2025. URL: https://doi.org/10.48550/arXiv.2502.16927.
  23. Naga Katta, Aditi Ghag, Mukesh Hira, Isaac Keslassy, Aran Bergman, Changhoon Kim, and Jennifer Rexford. Clove: Congestion-aware load balancing at the virtual edge. In ACM CoNEXT, pages 323-335, 2017. Google Scholar
  24. Mikhail Khalilov, Siyuan Shen, Marcin Chrapek, Tiancheng Chen, Kenji Nakano, Peter-Jan Gootzen, Salvatore Di Girolamo, Rami Nudelman, Gil Bloch, Sreevatsa Anantharamu, et al. SDR-RDMA: Software-defined reliability architecture for planetary scale RDMA communication. arXiv preprint arXiv:2505.05366, 2025. Google Scholar
  25. Yanfang Le, Rong Pan, Peter Newman, Jeremias Blendin, Abdul Kabbani, Vipin Jain, Raghava Sivaramu, and Francis Matus. Strack: A reliable multipath transport for AI/ML clusters. arXiv preprint arXiv:2407.15266, 2024. URL: https://doi.org/10.48550/arXiv.2407.15266.
  26. Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE training and inference with lina. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 945-959, 2023. URL: https://www.usenix.org/conference/atc23/presentation/li-jiamin.
  27. Wenxue Li, Xiangzhou Liu, Yunxuan Zhang, Zihao Wang, Wei Gu, Tao Qian, Gaoxiong Zeng, Shoushou Ren, Xinyang Huang, Zhenghang Ren, et al. Revisiting RDMA reliability for lossy fabrics. In ACM SIGCOMM, pages 85-98, 2025. Google Scholar
  28. Yuliang Li et al. Firefly: Scalable, ultra-accurate clock synchronization for datacenters. In ACM SIGCOMM, pages 434-452, 2025. Google Scholar
  29. Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, and Minlan Yu. HPCC: High precision congestion control. In ACM SIGCOMM, pages 44-58, 2019. URL: https://doi.org/10.1145/3341302.3342085.
  30. Xudong Liao, Yijun Sun, Han Tian, Xinchen Wan, Yilun Jin, Zilong Wang, Zhenghang Ren, Xinyang Huang, Wenxue Li, Kin Fai Tse, et al. MixNet: A runtime reconfigurable optical-electrical fabric for distributed mixture-of-experts training. In ACM SIGCOMM, pages 554-574, 2025. Google Scholar
  31. Hwijoon Lim, Jaehong Kim, Inho Cho, Keon Jang, Wei Bai, and Dongsu Han. Flexpass: A case for flexible credit-based transport for datacenter networks. In EuroSys, pages 606-622, 2023. URL: https://doi.org/10.1145/3552326.3587453.
  32. Wei Liu, Kun Qian, Zhenhua Li, Tianyin Xu, Yunhao Liu, Weicheng Wang, Yun Zhang, Jiakang Li, Shuhong Zhu, Xue Li, et al. SkeletonHunter: Diagnosing and localizing network failures in containerized large model training. In ACM SIGCOMM, pages 527-540, 2025. Google Scholar
  33. Xinyi Liu, Yujie Wang, Fangcheng Fu, Xupeng Miao, Shenhan Zhu, Xiaonan Nie, and Bin Cui. NetMoE: Accelerating MoE training through dynamic sample placement. In ICML, 2025. Google Scholar
  34. Jie Lu, Jiaqi Gao, Fei Feng, Zhiqiang He, Menglei Zheng, Kun Liu, Jun He, Binbin Liao, Suwei Xu, Ke Sun, et al. Alibaba Stellar: A new generation RDMA network for cloud AI. In ACM SIGCOMM, pages 453-466, 2025. Google Scholar
  35. Saf Malik. Zuckerberg: Meta to spend 'hundreds of billions' on AI data centres for superintelligence push . https://www.capacitymedia.com/article-zuckerberg-meta-ai-data-centres, 2025.
  36. Sarah McClure, Sylvia Ratnasamy, and Scott Shenker. Load balancing for AI training workloads. arXiv preprint arXiv:2507.21372, 2025. URL: https://doi.org/10.48550/arXiv.2507.21372.
  37. Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. Homa: A receiver-driven low-latency transport protocol using network priorities. In ACM SIGCOMM, pages 221-235, 2018. URL: https://doi.org/10.1145/3230543.3230564.
  38. Yang Nie, Zheng Shi, Xinyi Chen, and Liguo Qian. An out-of-order packet processing algorithm of RoCE based on improved SACK. In IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), pages 1402-1408, 2022. Google Scholar
  39. NVIDIA. NVIDIA InfiniBand Adaptive Routing Technology Accelerating HPC and AI Applications. https://www.amax.com/content/files/2023/12/NVIDIA_InfiniBand_Adaptive_Routing_Technology_Insights_Whitepaper.pdf, 2023.
  40. NVIDIA. Connectx-8 supernic datasheet. https://nvdam.widen.net/s/pxsjzhgw6j/connectx-datasheet-connectx-8-supernic-3231505, 2024. Datasheet.
  41. NVIDIA. NVIDIA Spectrum-X Network Platform Architecture. https://resources.nvidia.com/en-us-networking-ai/nvidia-spectrum-x, 2024.
  42. Vladimir Olteanu, Haggai Eran, Dragos Dumitrescu, Adrian Popa, Cristi Baciu, Mark Silberstein, Georgios Nikolaidis, Mark Handley, and Costin Raiciu. An edge-queued datagram service for all datacenter traffic. In Usenix NSDI, pages 761-777, 2022. Google Scholar
  43. OpenAI. Stargate advances with 4.5 GW partnership with Oracle. https://openai.com/index/stargate-advances-with-partnership-with-oracle/, 2025.
  44. PA Governor’s Press Office. Governor Josh Shapiro Announces Amazon Plans to Invest $20 Billion in Pennsylvania for AI Infrastructure. https://dced.pa.gov/newsroom/governor-josh-shapiro-announces-amazon-plans-to-invest-20-billion-in-pennsylvania-for-ai-infrastructure-in-largest-capital-investment-in-commonwealth-history/, 2025.
  45. Dylan Patel, Daniel Nishball, and Jeremie Eliahou Ontiveros. Multi-Datacenter Training: OpenAI’s Ambitious Plan To Beat Google’s Infrastructure. https://semianalysis.com/2024/09/04/multi-datacenter-training-openais/, 2024.
  46. Chenchen Qi, Wenfei Wu, Yongcan Wang, Keqiang He, Yu-Hsiang Kao, Zongying He, Chen-Yu Yen, Zhuo Jiang, Feng Luo, Surendra Anubolu, et al. SGLB: Scalable and robust global load balancing in commodity AI clusters. In ACM SIGCOMM, pages 626-644, 2025. Google Scholar
  47. Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, et al. Alibaba HPN: A data center network for large language model training. In ACM SIGCOMM, pages 691-706, 2024. Google Scholar
  48. Costin Raiciu and Gianni Antichi. NDP: Rethinking datacenter networks and stacks two years after. ACM SIGCOMM Computer Communication Review, 49(5):112-114, 2019. URL: https://doi.org/10.1145/3371934.3371968.
  49. Sudarsanan Rajasekaran, Manya Ghobadi, and Aditya Akella. CASSINI: Network-aware job scheduling in machine learning clusters. In Usenix NSDI, pages 1403-1420, 2024. Google Scholar
  50. Peter Rizk. Turbocharging Generative AI Workloads with NVIDIA Spectrum-X Networking Platform. https://developer.nvidia.com/blog/turbocharging-ai-workloads-with-nvidia-spectrum-x-networking-platform/, 2023.
  51. Chenchen Shou, Guyue Liu, Hao Nie, Huaiyu Meng, Yu Zhou, Yimin Jiang, Wenqing Lv, Yelong Xu, Yuanwei Lu, Zhang Chen, et al. InfiniteHBD: Building datacenter-scale high-bandwidth domain for LLM with optical circuit switching transceivers. In ACM SIGCOMM, pages 1-23, 2025. Google Scholar
  52. Cha Hwan Song, Xin Zhe Khooi, Raj Joshi, Inho Choi, Jialin Li, and Mun Choon Chan. Network load balancing with in-network reordering support for rdma. In ACM SIGCOMM, pages 816-831, 2023. URL: https://doi.org/10.1145/3603269.3604849.
  53. Dan Straussman et al. dcSim repository. https://github.com/danstr1/dcsim, 2025.
  54. Lide Suo, Yiren Pang, Wenxin Li, Renjie Pei, Keqiu Li, Xiulong Liu, Xin He, Yitao Hu, and Guyue Liu. PPT: A pragmatic transport for datacenters. In ACM SIGCOMM, pages 954-969, 2024. URL: https://doi.org/10.1145/3651890.3672235.
  55. Terabit Ethernet. dcPIM repository. https://github.com/Terabit-Ethernet/dcPIM, 2025.
  56. Ultra Ethernet Consortium. Ultra EthernetTM Specification v1.0. https://ultraethernet.org/wp-content/uploads/sites/20/2025/06/UE-Specification-6.11.25.pdf, 2025.
  57. Moshe Voloshin. Introduction to Congestion Control for RoCE. Technical report, Broadcom Inc., 2023. URL: "https://docs.broadcom.com/doc/NCC-WP1XX".
  58. Yongji Wu, Yechen Xu, Jingrong Chen, Zhaodong Wang, Ying Zhang, Matthew Lentz, and Danyang Zhuo. MCCS: A service-based approach to collective communication for multi-tenant cloud. In ACM SIGCOMM, pages 679-690, 2024. URL: https://doi.org/10.1145/3651890.3672252.
  59. Jie Zhang, Dafang Zhang, and Kun Huang. Improving datacenter throughput and robustness with Lazy TCP over packet spraying. Computer Communications, 62:23-33, 2015. URL: https://doi.org/10.1016/J.COMCOM.2015.02.011.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact