Document Open Access Logo

Mean-Payoff-Parity and Lifting Strategies from MDPs to 2-Player Stochastic Games

Authors Mohan Dantam , Richard Mayr

PDF

File

Thumbnail PDF
PDF
LIPIcs.CONCUR.2026.30.pdf
  • Filesize: 1.03 MB
  • 18 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Computing methodologies → Stochastic games
Keywords
  • MDPs
  • Stochastic Games
  • Parity
  • Mean-payoff
  • Strategy complexity

Metrics

Abstract

We consider the strategy complexity (i.e., memory and randomization) of optimal strategies in turn-based 2-player zero-sum stochastic games. Results in [Gimbert and Kelmendi, 2023; Richard Mayr et al., 2021] show how to lift optimal memoryless strategies for shift-invariant inverse-submixing objectives from MDPs to 2-player stochastic games with an exponential increase in the number of memory modes. We show the corresponding lower bound, i.e., the extra exponential memory is required in general, even for randomized strategies.
Moreover, we solve the strategy complexity of the well-studied mean-payoff-parity objective (MP > 0 ∩ EPAR) in 2-player stochastic games. This objective is also shift-invariant inverse-submixing, but easier than the worst case for this class. In MDPs, Maximizer has optimal memoryless randomized strategies, while optimal deterministic strategies require exponential memory. However, in stochastic games, optimal randomized strategies require, at least and at most, linear memory (equal to the number of even colors).
Finally, we show that the different construction in [Gimbert and Zielonka, 2009; Patricia Bouyer et al., 2023] for lifting memoryless (resp. finite-memory) deterministic strategies from MDPs (resp. 1-player games) to 2-player games cannot be generalized even to memoryless randomized strategies. We construct a shift-invariant objective where Max and Min each have optimal memoryless randomized strategies in all MDPs, but optimal (randomized) Max strategies still require infinite memory in deterministic 2-player games.

Cite As Get BibTex

Mohan Dantam and Richard Mayr. Mean-Payoff-Parity and Lifting Strategies from MDPs to 2-Player Stochastic Games. In 37th International Conference on Concurrency Theory (CONCUR 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 391, pp. 30:1-30:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.CONCUR.2026.30

Author Details

Mohan Dantam
  • Department of Computer Science, University of Oxford, UK
Richard Mayr
  • School of Informatics, University of Edinburgh, UK

References

  1. Rajeev Alur, Thomas A. Henzinger, and Orna Kupferman. Alternating-time temporal logic. Journal of the ACM, 49(5):672-713, 2002. URL: http://dx.doi.org/10.1145/585265.585270.
  2. Patrick Billingsley. Probability and measure. John Wiley & Sons, 2008. Google Scholar
  3. Patricia Bouyer, Youssouf Oualhadj, Mickael Randour, and Pierre Vandenhove. Arena-independent finite-memory determinacy in stochastic games. Logical Methods in Computer Science, 19, 2023. URL: http://dx.doi.org/10.46298/lmcs-19(4:18)2023.
  4. Tomás Brázdil, Václav Brožek, Krishnendu Chatterjee, Vojtěch Forejt, and Antonín Kučera. Markov decision processes with multiple long-run average objectives. Logical Methods in Computer Science, 10, 2014. Google Scholar
  5. Tomás Brázdil, Václav Brozek, and Kousha Etessami. One-Counter Stochastic Games. In Kamal Lodaya and Meena Mahajan, editors, IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS 2010), volume 8 of Leibniz International Proceedings in Informatics (LIPIcs), pages 108-119, Dagstuhl, Germany, 2010. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. Full version at http://arxiv.org/abs/1009.5636. URL: http://dx.doi.org/10.4230/LIPIcs.FSTTCS.2010.108.
  6. Tomás Brázdil, Václav Brozek, and Kousha Etessami. One-counter stochastic games. CoRR, abs/1009.5636, 2010. (Cited for good reason. LIPIcs version of this paper does not contain the relevant material.). URL: http://arxiv.org/abs/1009.5636.
  7. Krishnendu Chatterjee. Optimal strategy synthesis in stochastic Müller games. In International Conference on Foundations of Software Science and Computational Structures (FoSSaCS), volume 4423 of Lecture Notes in Computer Science, pages 138-152. Springer, 2007. URL: http://dx.doi.org/10.1007/978-3-540-71389-0_11.
  8. Krishnendu Chatterjee and Laurent Doyen. Energy and mean-payoff parity Markov decision processes. In International Symposium on Mathematical Foundations of Computer Science (MFCS), volume 6907, pages 206-218, 2011. URL: http://dx.doi.org/10.1007/978-3-642-22993-0_21.
  9. Krishnendu Chatterjee and Laurent Doyen. Games and Markov decision processes with mean-payoff parity and energy parity objectives. In Mathematical and Engineering Methods in Computer Science (MEMICS), volume 7119 of LNCS, pages 37-46. Springer, 2011. URL: http://dx.doi.org/10.1007/978-3-642-25929-6_3.
  10. Krishnendu Chatterjee, Laurent Doyen, Hugo Gimbert, and Thomas A. Henzinger. Randomness for free. Information and Computation, 245:3-16, 2015. URL: http://dx.doi.org/10.1016/J.IC.2015.06.003.
  11. Krishnendu Chatterjee, Laurent Doyen, Hugo Gimbert, and Youssouf Oualhadj. Perfect-information stochastic mean-payoff parity games. In International Conference on Foundations of Software Science and Computational Structures (FoSSaCS), volume 8412 of LNCS, 2014. URL: http://dx.doi.org/10.1007/978-3-642-54830-7_14.
  12. Krishnendu Chatterjee, Thomas A. Henzinger, and Marcin Jurdziński. Mean-payoff parity games. In Logic in Computer Science (LICS), pages 178-187, 2005. URL: http://dx.doi.org/10.1109/LICS.2005.26.
  13. Krishnendu Chatterjee, Marcin Jurdziński, and Thomas A. Henzinger. Simple stochastic parity games. In Computer Science Logic (CSL), volume 2803 of LNCS, pages 100-113. Springer, 2003. URL: http://dx.doi.org/10.1007/978-3-540-45220-1_11.
  14. Krishnendu Chatterjee, Marcin Jurdziński, and Thomas A. Henzinger. Quantitative stochastic parity games. In ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 121-130. SIAM, 2004. URL: http://dl.acm.org/citation.cfm?id=982792.982808.
  15. Krishnendu Chatterjee, Mickael Randour, and Jean-François Raskin. Strategy synthesis for multi-dimensional quantitative objectives. Acta Informatica, 51(3-4):129-163, 2014. URL: http://dx.doi.org/10.1007/S00236-013-0182-6.
  16. E.M. Clarke, O. Grumberg, and D. Peled. Model Checking. MIT Press, December 1999. Google Scholar
  17. Thomas Colcombet and Damian Niwinski. On the positional determinacy of edge-labeled games. Theoretical Computer Science, 352(1-3):190-196, 2006. URL: http://dx.doi.org/10.1016/J.TCS.2005.10.046.
  18. Anne Condon. The complexity of stochastic games. Information and Computation, 96(2):203-224, 1992. URL: http://dx.doi.org/10.1016/0890-5401(92)90048-K.
  19. Mohan Dantam and Richard Mayr. Mean-payoff-parity and lifting strategies from MDPs to 2-player stochastic games, 2026. URL: http://arxiv.org/abs/2606.19324.
  20. Laure Daviaud, Marcin Jurdziński, and Ranko Lazić. A pseudo-quasi-polynomial algorithm for mean-payoff parity games. In Logic in Computer Science (LICS), pages 325-334, 2018. Google Scholar
  21. Luca De Alfaro. Formal verification of probabilistic systems. PhD thesis, Stanford University, 1997. Google Scholar
  22. Luca De Alfaro and Thomas A. Henzinger. Interface automata. ACM SIGSOFT Software Engineering Notes, 26(5):109-120, 2001. URL: http://dx.doi.org/10.1145/503209.503226.
  23. David L. Dill. Trace theory for automatic hierarchical verification of speed-independent circuits, volume 24. MIT press Cambridge, 1989. Google Scholar
  24. J. Filar and K. Vrieze. Competitive Markov Decision Processes. Springer, 1997. Google Scholar
  25. H. Gimbert and E. Kelmendi. Submixing and shift-invariant stochastic games. International Journal of Game Theory, 52:1179-1214, 2023. URL: http://dx.doi.org/10.1007/s00182-023-00860-5.
  26. Hugo Gimbert, Youssouf Oualhadj, and Soumya Paul. Computing optimal strategies for Markov decision processes with parity and positive-average conditions. Working paper or preprint, 2011. URL: https://hal.science/hal-00559173/en/.
  27. Hugo Gimbert and Wiesław Zielonka. Games where you can play optimally without any memory. In International Conference on Concurrency Theory (CONCUR), volume 3653 of Lecture Notes in Computer Science, pages 428-442. Springer, 2005. URL: http://dx.doi.org/10.1007/11539452_33.
  28. Hugo Gimbert and Wiesław Zielonka. Pure and Stationary Optimal Strategies in Perfect-Information Stochastic Games. Working paper or preprint, December 2009. URL: https://hal.archives-ouvertes.fr/hal-00438359.
  29. Kristoffer Arnsfelt Hansen, Rasmus Ibsen-Jensen, and Abraham Neyman. The big match with a clock and a bit of memory. Mathematics of Operations Research, 48, 2022. Google Scholar
  30. Florian Horn. Random fruits on the Zielonka tree. In International Symposium on Theoretical Aspects of Computer Science (STACS), volume 3 of LIPIcs, pages 541-552. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Germany, 2009. URL: http://dx.doi.org/10.4230/LIPIcs.STACS.2009.1848.
  31. James C. A. Main and Mickael Randour. Different strokes in randomised strategies: Revisiting Kuhn’s theorem under finite-memory assumptions. In CONCUR, volume 243 of LIPIcs, pages 22:1-22:18. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2022. URL: http://dx.doi.org/10.4230/LIPIcs.CONCUR.2022.22.
  32. A. Maitra and W. Sudderth. Stochastic games with Borel payoffs. In Stochastic Games and Applications, pages 367-373. Kluwer, Dordrecht, 2003. Google Scholar
  33. Donald A. Martin. The determinacy of Blackwell games. Journal of Symbolic Logic, 63(4):1565-1581, 1998. URL: http://dx.doi.org/10.2307/2586667.
  34. Richard Mayr, Sven Schewe, Patrick Totzke, and Dominik Wojtczak. Simple stochastic games with almost-sure energy-parity objectives are in NP and coNP. In Proc. of Fossacs, volume 12650 of LNCS, 2021. Extended version on arXiv. URL: http://arxiv.org/abs/2101.06989.
  35. George H. Mealy. A method for synthesizing sequential circuits. The Bell System Technical Journal, 34(5):1045-1079, 1955. URL: http://dx.doi.org/10.1002/j.1538-7305.1955.tb03788.x.
  36. Amir Pnueli and Roni Rosner. On the synthesis of a reactive module. In Annual Symposium on Principles of Programming Languages (POPL), pages 179-190, 1989. URL: http://dx.doi.org/10.1145/75277.75293.
  37. Peter J. Ramadge and W. Murray Wonham. Supervisory control of a class of discrete event processes. SIAM journal on control and optimization, 25(1):206-230, 1987. Google Scholar
  38. Lloyd S. Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095-1100, 1953. Google Scholar
  39. Pierre Vandenhove. Strategy complexity of zero-sum games on graphs. PhD thesis, Université Paris-Saclay; Université de Mons, 2023. URL: https://theses.hal.science/tel-04095220.
  40. Wiesław Zielonka. Infinite games on finitely coloured graphs with applications to automata on infinite trees. Theoretical Computer Science, 200(1-2):135-183, 1998. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact