Document Open Access Logo

Flooding with Absorption: An Efficient Protocol for Heterogeneous Bandits over Complex Networks

Authors Junghyun Lee , Laura Schmid , Se-Young Yun

PDF
Thumbnail PDF

File

LIPIcs.OPODIS.2023.20.pdf
  • Filesize: 3.29 MB
  • 25 pages

Document Identifiers

Author Details

Junghyun Lee
  • Kim Jaechul Graduate School of AI, KAIST, Seoul, Republic of Korea
Laura Schmid
  • Kim Jaechul Graduate School of AI, KAIST, Seoul, Republic of Korea
Se-Young Yun
  • Kim Jaechul Graduate School of AI, KAIST, Seoul, Republic of Korea

Funding

JH and SY were supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korean government(MSIT) (No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)). LS was supported by the Stochastic Analysis and Application Research Center (SAARC) under National Research Foundation of Korea grant NRF-2019R1A5A1028324.

Acknowledgements

The authors thank Ulrich Schmid (TU Wien) and the anonymous reviewers for helpful comments and suggestions.

Cite AsGet BibTex

Junghyun Lee, Laura Schmid, and Se-Young Yun. Flooding with Absorption: An Efficient Protocol for Heterogeneous Bandits over Complex Networks. In 27th International Conference on Principles of Distributed Systems (OPODIS 2023). Leibniz International Proceedings in Informatics (LIPIcs), Volume 286, pp. 20:1-20:25, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024)
https://doi.org/10.4230/LIPIcs.OPODIS.2023.20

Abstract

Multi-armed bandits are extensively used to model sequential decision-making, making them ubiquitous in many real-life applications such as online recommender systems and wireless networking. We consider a multi-agent setting where each agent solves their own bandit instance endowed with a different set of arms. Their goal is to minimize their group regret while collaborating via some communication protocol over a given network. Previous literature on this problem only considered arm heterogeneity and networked agents separately. In this work, we introduce a setting that encompasses both features. For this novel setting, we first provide a rigorous regret analysis for a standard flooding protocol combined with the classic UCB policy. Then, to mitigate the issue of high communication costs incurred by flooding in complex networks, we propose a new protocol called Flooding with Absorption (FwA). We provide a theoretical analysis of the resulting regret bound and discuss the advantages of using FwA over flooding. Lastly, we experimentally verify on various scenarios, including dynamic networks, that FwA leads to significantly lower communication costs despite minimal regret performance loss compared to other network protocols.

Subject Classification

ACM Subject Classification
  • Theory of computation → Multi-agent learning
  • Theory of computation → Sequential decision making
  • Theory of computation → Regret bounds
  • Networks → Network protocol design
Keywords
  • multi-armed bandits
  • multi-agent systems
  • collaborative learning
  • network protocol
  • flooding

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Supplementary Materials

References

  1. Animashree Anandkumar, Nithin Michael, and Ao Tang. Opportunistic Spectrum Access with Multiple Users: Learning under Competition. In 2010 Proceedings IEEE INFOCOM, pages 1-9, 2010. URL: https://doi.org/10.1109/INFCOM.2010.5462144.
  2. Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning, 47(2):235-256, 2002. URL: https://doi.org/10.1023/A:1013689704352.
  3. Orly Avner and Shie Mannor. Multi-user lax communications: A multi-armed bandit approach. In IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications, pages 1-9, 2016. URL: https://doi.org/10.1109/INFOCOM.2016.7524557.
  4. Albert-László Barabási and Réka Albert. Emergence of Scaling in Random Networks. Science, 286(5439):509-512, 1999. URL: https://doi.org/10.1126/science.286.5439.509.
  5. Albert-László Barabási and Réka Albert. Statistical mechanics of complex networks. Reviews of Modern Physics, 74(1):47-97, 2002. URL: https://doi.org/10.1103/RevModPhys.74.47.
  6. Oded Berger-Tal, Jonathan Nathan, Ehud Meron, and David Saltz. The Exploration-Exploitation Dilemma: A Multidisciplinary Framework. PLOS ONE, 9:1-8, apr 2014. URL: https://doi.org/10.1371/journal.pone.0095693.
  7. Béla Bollobás. Modern Graph Theory, volume 184 of Graduate Texts in Mathematics. Springer, 2002. URL: https://doi.org/10.1007/978-1-4612-0619-4.
  8. Sébastien Bubeck. Bandits Games and Clustering Foundations. PhD thesis, INRIA Nord Europe, jun 2010. Google Scholar
  9. Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems. Foundations and Trends® in Machine Learning, 5(1):1-122, 2012. URL: https://doi.org/10.1561/2200000024.
  10. Swapna Buccapatnam, Jian Tan, and Li Zhang. Information sharing in distributed stochastic bandits. In 2015 IEEE Conference on Computer Communications (INFOCOM), pages 2605-2613, 2015. URL: https://doi.org/10.1109/INFOCOM.2015.7218651.
  11. Keren Censor-Hillel, Bernhard Haeupler, Jonathan Kelner, and Petar Maymounkov. Global Computation in a Poorly Connected World: Fast Rumor Spreading with No Dependence on Conductance. In Proceedings of the Forty-Fourth Annual ACM Symposium on Theory of Computing, STOC '12, pages 961-970. Association for Computing Machinery, 2012. URL: https://doi.org/10.1145/2213977.2214064.
  12. Nicolò Cesa-Bianchi, Tommaso Cesari, and Claire Monteleoni. Cooperative Online Learning: Keeping your Neighbors Updated. In Proceedings of the 31st International Conference on Algorithmic Learning Theory, volume 117 of Proceedings of Machine Learning Research, pages 234-250. PMLR, 08 February-11 February 2020. URL: http://proceedings.mlr.press/v117/cesa-bianchi20a.html.
  13. Nicholas B Chang and Mingyan Liu. Controlled Flooding Search in a Large Network. IEEE/ACM Transactions on Networking, 15(2):436-449, 2007. URL: https://doi.org/10.1145/1279660.1279675.
  14. Ronshee Chawla, Abishek Sankararaman, Ayalvadi Ganesh, and Sanjay Shakkottai. The Gossiping Insert-Eliminate Algorithm for Multi-Agent Bandits. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3471-3481. PMLR, 26-28 August 2020. URL: http://proceedings.mlr.press/v108/chawla20a.html.
  15. Ronshee Chawla, Daniel Vial, Sanjay Shakkottai, and R. Srikant. Collaborative Multi-Agent Heterogeneous Multi-Armed Bandits. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 4189-4217. PMLR, 23-29 July 2023. URL: https://proceedings.mlr.press/v202/chawla23a.html.
  16. Yu-Zhen Janice Chen, Lin Yang, Xuchuang Wang, Xutong Liu, Mohammad Hajiesmaili, John C. S. Lui, and Don Towsley. On-Demand Communication for Asynchronous Multi-Agent Bandits. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 3903-3930. PMLR, 25-27 April 2023. URL: https://proceedings.mlr.press/v206/chen23c.html.
  17. Andrea Clementi, Pierluigi Crescenzi, Carola Doerr, Pierre Fraigniaud, Francesco Pasquale, and Riccardo Silvestri. Rumor Spreading in Random Evolving Graphs. Random Structures & Algorithms, 48(2):290-312, 2016. URL: https://doi.org/10.1002/RSA.20586.
  18. Andrea Clementi, Angelo Monti, Francesco Pasquale, and Riccardo Silvestri. Information Spreading in Stationary Markovian Evolving Graphs. IEEE Transactions on Parallel and Distributed Systems, 22(9):1425-1432, 2011. URL: https://doi.org/10.1109/TPDS.2011.33.
  19. Andrea Clementi, Riccardo Silvestri, and Luca Trevisan. Information spreading in dynamic graphs. Distributed Computing, 28(1):55-73, feb 2015. URL: https://doi.org/10.1007/S00446-014-0219-2.
  20. Andrea E. F. Clementi, Claudio Macci, Angelo Monti, Francesco Pasquale, and Riccardo Silvestri. Flooding Time of Edge-Markovian Evolving Graphs. SIAM Journal on Discrete Mathematics, 24(4):1694-1712, 2010. URL: https://doi.org/10.1137/090756053.
  21. Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms. The MIT Press, 4 edition, 2022. Google Scholar
  22. Abhimanyu Dubey and Alex Pentland. Cooperative Multi-Agent Bandits with Heavy Tails. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2730-2739. PMLR, 13-18 July 2020. URL: http://proceedings.mlr.press/v119/dubey20a.html.
  23. Paul Erdős and Alfréd Rényi. On random graphs, I. Publicationes Mathematicae Debrecen, 6:290-297, 1959. Google Scholar
  24. P.T. Eugster, R. Guerraoui, A.-M. Kermarrec, and L. Massoulié. Epidemic Information Dissemination in Distributed Systems. Computer, 37(5):60-67, 2004. URL: https://doi.org/10.1109/MC.2004.1297243.
  25. Alan Frieze and Colin McDiarmid. Algorithmic Theory of Random Graphs. Random Structures & Algorithms, 10(1–2):5-42, feb 1997. URL: https://doi.org/10.1002/(SICI)1098-2418(199701/03)10:1/2%3C5::AID-RSA2%3E3.0.CO;2-Z.
  26. E. N. Gilbert. Random Graphs. The Annals of Mathematical Statistics, 30(4):1141-1144, 1959. Google Scholar
  27. A. Gyárfás. Problems from the world surrounding perfect graphs. Applicationes Mathematicae, 19(3-4):413-441, 1987. Google Scholar
  28. András Gyárfás, András Sebő, and Nicolas Trotignon. The chromatic gap and its extremes. Journal of Combinatorial Theory, Series B, 102(5):1155-1178, 2012. URL: https://doi.org/10.1016/J.JCTB.2012.06.001.
  29. Bernhard Haeupler. Simple, Fast and Deterministic Gossip and Rumor Spreading. Journal of the ACM, 62(6), dec 2015. URL: https://doi.org/10.1145/2767126.
  30. Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. Exploring Network Structure, Dynamics, and Function using NetworkX. In Proceedings of the 7th Python in Science Conference, pages 11-15, 2008. Google Scholar
  31. Paul W. Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social Networks, 5(2):109-137, 1983. URL: https://doi.org/10.1016/0378-8733(83)90021-7.
  32. Long Jin, Shuai Li, Lin Xiao, Rongbo Lu, and Bolin Liao. Cooperative Motion Generation in a Distributed Network of Redundant Robot Manipulators With Noises. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 48(10):1715-1724, 2018. URL: https://doi.org/10.1109/TSMC.2017.2693400.
  33. Andrea Kadović, Sebastian M. Krause, Guido Caldarelli, and Vinko Zlatic. Bond and site color-avoiding percolation in scale-free networks. Physical Review E, 98:062308, dec 2018. URL: https://doi.org/10.1103/PhysRevE.98.062308.
  34. Ravi Kumar Kolla, Krishna Jagannathan, and Aditya Gopalan. Collaborative Learning of Stochastic Bandits Over a Social Network. IEEE/ACM Transactions on Networking, 26(4):1782-1795, 2018. URL: https://doi.org/10.1109/TNET.2018.2852361.
  35. Sebastian M. Krause, Michael M. Danziger, and Vinko Zlatić. Hidden Connectivity in Networks with Vulnerable Classes of Nodes. Phys. Rev. X, 6:041022, oct 2016. URL: https://doi.org/10.1103/PhysRevX.6.041022.
  36. Sebastian M. Krause, Michael M. Danziger, and Vinko Zlatić. Color-avoiding percolation. Physical Review E, 96:022313, aug 2017. URL: https://doi.org/10.1103/PhysRevE.96.022313.
  37. T.L Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4-22, 1985. URL: https://doi.org/10.1016/0196-8858(85)90002-8.
  38. Peter Landgren, Vaibhav Srivastava, and Naomi Ehrich Leonard. Distributed cooperative decision making in multi-agent multi-armed bandits. Automatica, 125:109445, 2021. URL: https://doi.org/10.1016/J.AUTOMATICA.2020.109445.
  39. Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020. URL: https://doi.org/10.1017/9781108571401.
  40. Marc Lelarge, Alexandre Proutière, and M. Sadegh Talebi. Spectrum bandit optimization. In 2013 IEEE Information Theory Workshop (ITW), pages 1-5, 2013. URL: https://doi.org/10.1109/ITW.2013.6691221.
  41. Daniel A Levinthal and James G March. The myopia of learning. Strategic management journal, 14(S2):95-112, 1993. URL: https://doi.org/10.1002/smj.4250141009.
  42. Feng Li, Dongxiao Yu, Huan Yang, Jiguo Yu, Holger Karl, and Xiuzhen Cheng. Multi-Armed-Bandit-Based Spectrum Scheduling Algorithms in Wireless Networks: A Survey. IEEE Wireless Communications, 27(1):24-30, 2020. URL: https://doi.org/10.1109/MWC.001.1900280.
  43. Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A Contextual-Bandit Approach to Personalized News Article Recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW '10, pages 661-670. Association for Computing Machinery, 2010. URL: https://doi.org/10.1145/1772690.1772758.
  44. Shuai Li, Ruofan Kong, and Yi Guo. Cooperative Distributed Source Seeking by Multiple Robots: Algorithms and Experiments. IEEE/ASME Transactions on Mechatronics, 19(6):1810-1820, 2014. URL: https://doi.org/10.1109/TMECH.2013.2295036.
  45. H Lim and C Kim. Flooding in wireless ad hoc networks. Computer Communications, 24(3):353-363, 2001. URL: https://doi.org/10.1016/S0140-3664(00)00233-4.
  46. Keqin Liu and Qing Zhao. Distributed Learning in Multi-Armed Bandit With Multiple Players. IEEE Transactions on Signal Processing, 58(11):5667-5681, 2010. URL: https://doi.org/10.1109/TSP.2010.2062509.
  47. Qin Lv, Pei Cao, Edith Cohen, Kai Li, and Scott Shenker. Search and Replication in Unstructured Peer-to-Peer Networks. In Proceedings of the 16th International Conference on Supercomputing, ICS '02, pages 84-95, New York, NY, USA, 2002. Association for Computing Machinery. URL: https://doi.org/10.1145/514191.514206.
  48. Qin Lv, Sylvia Ratnasamy, and Scott Shenker. Can Heterogeneity Make Gnutella Scalable? In Revised Papers from the First International Workshop on Peer-to-Peer Systems, IPTPS '01, pages 94-103, Berlin, Heidelberg, 2002. Springer-Verlag. URL: https://doi.org/10.1007/3-540-45748-8_9.
  49. Udari Madhushani, Abhimanyu Dubey, Naomi Leonard, and Alex Pentland. One More Step Towards Reality: Cooperative Bandits with Imperfect Communication. In Advances in Neural Information Processing Systems, volume 34, pages 7813-7824. Curran Associates, Inc., 2021. URL: https://proceedings.neurips.cc/paper/2021/hash/40cb228987243c91b2dd0b7c9c4a0856-Abstract.html.
  50. Udari Madhushani and Naomi Ehrich Leonard. Distributed Bandits: Probabilistic Communication on d-regular Graphs. In 2021 European Control Conference (ECC), pages 830-835, 2021. URL: https://doi.org/10.23919/ECC54610.2021.9655031.
  51. James G. March. Exploration and Exploitation in Organizational Learning. Organization Science, 2(1):71-87, 1991. URL: https://doi.org/10.1287/orsc.2.1.71.
  52. Katja Mehlhorn, Ben R Newell, Peter M Todd, Michael D Lee, Kate Morgan, Victoria A Braithwaite, Daniel Hausmann, Klaus Fiedler, and Cleotilde Gonzalez. Unpacking the exploration-exploitation tradeoff: A synthesis of human and animal literatures. Decision, 2(3):191-215, 2015. URL: https://doi.org/10.1037/dec0000033.
  53. Andrea Munaro. Bounded clique cover of some sparse graphs. Discrete Mathematics, 340(9):2208-2216, 2017. URL: https://doi.org/10.1016/J.DISC.2017.04.004.
  54. Konstantinos Oikonomou, George Koufoudakis, Sonia Aïssa, and Ioannis Stavrakakis. Probabilistic Flooding Performance Analysis Exploiting Graph Spectra Properties. IEEE/ACM Transactions on Networking, 31(1):133-146, 2023. URL: https://doi.org/10.1109/TNET.2022.3192310.
  55. Ashikur Rahman, Wlodek Olesinski, and Pawel Gburzynski. Controlled flooding in wireless ad-hoc networks. In International Workshop on Wireless Ad-Hoc Networks, 2004., pages 73-78. IEEE, 2004. URL: https://doi.org/10.1109/IWWAN.2004.1525544.
  56. Abishek Sankararaman, Ayalvadi Ganesh, and Sanjay Shakkottai. Social Learning in Multi Agent Multi Armed Bandits. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 3(3), dec 2019. URL: https://doi.org/10.1145/3366701.
  57. Devavrat Shah. Gossip Algorithms. Foundations and Trends® in Networking, 3(1):1-125, 2009. URL: https://doi.org/10.1561/1300000014.
  58. K. Sugawara, T. Kazama, and T. Watanabe. Foraging behavior of interacting robots with virtual pheromone. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), volume 3, pages 3074-3079 vol.3, 2004. URL: https://doi.org/10.1109/IROS.2004.1389878.
  59. Balazs Szorenyi, Robert Busa-Fekete, Istvan Hegedus, Robert Ormandi, Mark Jelasity, and Balazs Kegl. Gossip-based distributed stochastic bandit algorithms. In Proceedings of the 30th International Conference on Machine Learning, volume 28(3) of Proceedings of Machine Learning Research, pages 19-27. PMLR, 17-19 June 2013. URL: http://proceedings.mlr.press/v28/szorenyi13.html.
  60. Andrew S. Tanenbaum, Nick Feamster, and David Wetherall. Computer Networks. Pearson, 6 edition, 2021. Google Scholar
  61. Yu-Chee Tseng, Sze-Yao Ni, Yuh-Shyan Chen, and Jang-Ping Sheu. The Broadcast Storm Problem in a Mobile Ad Hoc Network. Wireless Networks, 8(2):153-167, mar 2002. URL: https://doi.org/10.1023/A:1013763825347.
  62. Amin Vahdat and David Becker. Epidemic Routing for Partially-Connected Ad Hoc Networks. Technical Report CS-200006, Duke University, 2000. Google Scholar
  63. Daniel Vial, Sanjay Shakkottai, and R. Srikant. Robust Multi-Agent Multi-Armed Bandits. In Proceedings of the Twenty-Second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, MobiHoc '21, pages 161-170. Association for Computing Machinery, 2021. URL: https://doi.org/10.1145/3466772.3467045.
  64. Milan Vojnović and Alexandre Proutière. Hop limited flooding over dynamic networks. In 2011 Proceedings IEEE INFOCOM, pages 685-693, 2011. URL: https://doi.org/10.1109/INFCOM.2011.5935249.
  65. Po-An Wang, Alexandre Proutière, Kaito Ariu, Yassir Jedra, and Alessio Russo. Optimal Algorithms for Multiplayer Multi-Armed Bandits. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 4120-4129. PMLR, 26-28 August 2020. URL: http://proceedings.mlr.press/v108/wang20m.html.
  66. N. Wisitpongphan, O.K. Tonguz, J.S. Parikh, P. Mudalige, F. Bai, and V. Sadekar. Broadcast storm mitigation techniques in vehicular ad hoc networks. IEEE Wireless Communications, 14(6):84-94, 2007. URL: https://doi.org/10.1109/MWC.2007.4407231.
  67. Wenchao Xia, Tony Q. S. Quek, Kun Guo, Wanli Wen, Howard H. Yang, and Hongbo Zhu. Multi-Armed Bandit-Based Client Scheduling for Federated Learning. IEEE Transactions on Wireless Communications, 19(11):7108-7123, 2020. URL: https://doi.org/10.1109/TWC.2020.3008091.
  68. Lin Yang, Yu-Zhen Janice Chen, Mohammad H. Hajiemaili, John C. S. Lui, and Don Towsley. Distributed Bandits with Heterogeneous Agents. In IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, pages 200-209, 2022. URL: https://doi.org/10.1109/INFOCOM48880.2022.9796901.
  69. Lin Yang, Yu-Zhen Janice Chen, Stephen Pasteris, Mohammad H. Hajiesmaili, John C. S. Lui, and Don Towsley. Cooperative Stochastic Bandits with Asynchronous Agents and Constrained Feedback. In Advances in Neural Information Processing Systems, volume 34, pages 8885-8897. Curran Associates, Inc., 2021. URL: https://proceedings.neurips.cc/paper/2021/hash/4a5876b450b45371f6cfe5047ac8cd45-Abstract.html.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact