Document Open Access Logo

Risk-Averse Optimization of Total Rewards in Markovian Models Using Deviation Measures

Authors Christel Baier , Jakob Piribauer , Maximilian Starke

PDF
Thumbnail PDF

File

LIPIcs.CONCUR.2024.9.pdf
  • Filesize: 0.79 MB
  • 20 pages

Document Identifiers

Author Details

Christel Baier
  • Technische Universität Dresden, Germany
Jakob Piribauer
  • Technische Universität Dresden, Germany
  • Universität Leipzig, Germany
Maximilian Starke
  • Technische Universität Dresden, Germany

Funding

This work was partly funded by the DFG Grant 389792660 as part of TRR 248 (Foundations of Perspicuous Software Systems), the Cluster of Excellence EXC 2050/1 (CeTI, project ID 390696704, as part of Germany’s Excellence Strategy), and the DFG project BA 1679/11-1.

Cite AsGet BibTex

Christel Baier, Jakob Piribauer, and Maximilian Starke. Risk-Averse Optimization of Total Rewards in Markovian Models Using Deviation Measures. In 35th International Conference on Concurrency Theory (CONCUR 2024). Leibniz International Proceedings in Informatics (LIPIcs), Volume 311, pp. 9:1-9:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024)
https://doi.org/10.4230/LIPIcs.CONCUR.2024.9

Abstract

This paper addresses objectives tailored to the risk-averse optimization of accumulated rewards in Markov decision processes (MDPs). The studied objectives require maximizing the expected value of the accumulated rewards minus a penalty factor times a deviation measure of the resulting distribution of rewards. Using the variance in this penalty mechanism leads to the variance-penalized expectation (VPE) for which it is known that optimal schedulers have to minimize future expected rewards when a high amount of rewards has been accumulated. This behavior is undesirable as risk-averse behavior should keep the probability of particularly low outcomes low, but not discourage the accumulation of additional rewards on already good executions. The paper investigates the semi-variance, which only takes outcomes below the expected value into account, the mean absolute deviation (MAD), and the semi-MAD as alternative deviation measures. Furthermore, a penalty mechanism that penalizes outcomes below a fixed threshold is studied. For all of these objectives, the properties of optimal schedulers are specified and in particular the question whether these objectives overcome the problem observed for the VPE is answered. Further, the resulting algorithmic problems on MDPs and Markov chains are investigated.

Subject Classification

ACM Subject Classification
  • Theory of computation → Logic and verification
Keywords
  • Markov decision processes
  • risk-aversion
  • deviation measures
  • total reward

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Related Versions

Supplementary Materials

References

  1. Mohamadreza Ahmadi, Anushri Dixit, Joel W. Burdick, and Aaron D. Ames. Risk-averse stochastic shortest path planning. In 2021 60th IEEE Conference on Decision and Control (CDC), Austin, TX, USA, December 14-17, 2021, pages 5199-5204. IEEE, 2021. URL: https://doi.org/10.1109/CDC45484.2021.9683527.
  2. James Aspnes and Maurice Herlihy. Fast randomized consensus using shared memory. Journal of Algorithms, 11(3):441-461, 1990. URL: https://doi.org/10.1016/0196-6774(90)90021-6.
  3. Christel Baier, Krishnendu Chatterjee, Tobias Meggendorfer, and Jakob Piribauer. Entropic risk for turn-based stochastic games. In Jérôme Leroux, Sylvain Lombardy, and David Peleg, editors, 48th International Symposium on Mathematical Foundations of Computer Science, MFCS 2023, August 28 to September 1, 2023, Bordeaux, France, volume 272 of LIPIcs, pages 15:1-15:16. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2023. URL: https://doi.org/10.4230/LIPIcs.MFCS.2023.15.
  4. Christel Baier, Marcus Daum, Clemens Dubslaff, Joachim Klein, and Sascha Klüppelholz. Energy-utility quantiles. In Julia M. Badger and Kristin Yvonne Rozier, editors, NASA Formal Methods - 6th International Symposium, NFM 2014, Houston, TX, USA, April 29 - May 1, 2014. Proceedings, volume 8430 of Lecture Notes in Computer Science, pages 285-299. Springer, 2014. URL: https://doi.org/10.1007/978-3-319-06200-6_24.
  5. Christel Baier, Joachim Klein, Sascha Klüppelholz, and Sascha Wunderlich. Maximizing the conditional expected reward for reaching the goal. In Axel Legay and Tiziana Margaria, editors, 23rd International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), volume 10206 of Lecture Notes in Computer Science, pages 269-285. Springer, 2017. URL: https://doi.org/10.1007/978-3-662-54580-5_16.
  6. Christel Baier, Jakob Piribauer, and Maximilian Starke. Risk-averse optimization of total rewards in markovian models using deviation measures, 2024. URL: https://arxiv.org/abs/2407.06887.
  7. Dimitri P. Bertsekas and John N. Tsitsiklis. An analysis of stochastic shortest path problems. Math. Oper. Res., 16:580-595, 1991. URL: https://doi.org/10.1287/moor.16.3.580.
  8. Tomás Brázdil, Krishnendu Chatterjee, Vojtech Forejt, and Antonín Kucera. Trading performance for stability in markov decision processes. J. Comput. Syst. Sci., 84:144-170, 2017. URL: https://doi.org/10.1016/j.jcss.2016.09.009.
  9. T. Chen, V. Forejt, M. Kwiatkowska, D. Parker, and A. Simaitis. PRISM-games: A model checker for stochastic multi-player games. In N. Piterman and S. Smolka, editors, Proc. 19th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'13), volume 7795 of LNCS, pages 185-191. Springer, 2013. URL: https://doi.org/10.1007/978-3-642-36742-7_13.
  10. Taolue Chen, Vojtech Forejt, Marta Z. Kwiatkowska, David Parker, and Aistis Simaitis. Automatic verification of competitive stochastic systems. Formal Methods Syst. Des., 43(1):61-92, 2013. URL: https://doi.org/10.1007/S10703-013-0183-7.
  11. EJ Collins. Finite-horizon variance penalised Markov decision processes. Operations-Research-Spektrum, 19(1):35-39, 1997. Google Scholar
  12. Luca de Alfaro. Computing minimum and maximum reachability times in probabilistic systems. In 10th International Conference on Concurrency Theory (CONCUR), volume 1664 of Lecture Notes in Computer Science, pages 66-81, 1999. URL: https://doi.org/10.1007/3-540-48320-9_7.
  13. Jerzy A Filar, Lodewijk CM Kallenberg, and Huey-Miin Lee. Variance-penalized Markov decision processes. Mathematics of Operations Research, 14(1):147-161, 1989. URL: https://doi.org/10.1287/moor.14.1.147.
  14. John Gill. Computational complexity of probabilistic Turing machines. SIAM Journal on Computing, 6(4):675-695, 1977. URL: https://doi.org/10.1137/0206049.
  15. Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL: https://www.gurobi.com.
  16. Christoph Haase and Stefan Kiefer. The odds of staying on budget. In Magnús M. Halldórsson, Kazuo Iwama, Naoki Kobayashi, and Bettina Speckmann, editors, Automata, Languages, and Programming - 42nd International Colloquium, ICALP 2015, Kyoto, Japan, July 6-10, 2015, Proceedings, Part II, volume 9135 of Lecture Notes in Computer Science, pages 234-246. Springer, 2015. URL: https://doi.org/10.1007/978-3-662-47666-6_19.
  17. A. Itai and M. Rodeh. Symmetry breaking in distributed networks. Information and Computation, 88(1), 1990. URL: https://doi.org/10.1016/0890-5401(90)90004-2.
  18. Hiroshi Konno and Hiroaki Yamazaki. Mean-absolute deviation portfolio optimization model and its applications to tokyo stock market. Management Science, 37(5):519-531, 1991. URL: http://www.jstor.org/stable/2632458.
  19. Jan Kretínský and Tobias Meggendorfer. Conditional value-at-risk for reachability and mean payoff in Markov decision processes. In 33rd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS), pages 609-618. ACM, 2018. URL: https://doi.org/10.1145/3209108.3209176.
  20. Xiaoteng Ma, Shuai Ma, Li Xia, and Qianchuan Zhao. Mean-semivariance policy optimization via risk-averse reinforcement learning. J. Artif. Intell. Res., 75:569-595, 2022. URL: https://doi.org/10.1613/jair.1.13833.
  21. Petr Mandl. On the variance in controlled Markov chains. Kybernetika, 7(1):1-12, 1971. URL: http://www.kybernetika.cz/content/1971/1/1.
  22. Shie Mannor and John N. Tsitsiklis. Mean-variance optimization in Markov decision processes. In Lise Getoor and Tobias Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning, ICML'11, pages 177-184, Madison, WI, USA, 2011. Omnipress. URL: https://icml.cc/2011/papers/156_icmlpaper.pdf.
  23. Harry Markowitz. Portfolio selection. The Journal of Finance, 7(1):77-91, 1952. URL: http://www.jstor.org/stable/2975974.
  24. Harry M. Markowitz. Portfolio Selection: Efficient Diversification of Investments. Yale University Press, 1959. URL: http://www.jstor.org/stable/j.ctt1bh4c8h.
  25. Jakob Piribauer, Ocan Sankur, and Christel Baier. The variance-penalized stochastic shortest path problem. In Mikolaj Bojanczyk, Emanuela Merelli, and David P. Woodruff, editors, 49th International Colloquium on Automata, Languages, and Programming, ICALP 2022, July 4-8, 2022, Paris, France, volume 229 of LIPIcs, pages 129:1-129:19. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2022. URL: https://doi.org/10.4230/LIPIcs.ICALP.2022.129.
  26. Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994. URL: https://doi.org/10.1002/9780470316887.
  27. Mickael Randour, Jean-François Raskin, and Ocan Sankur. Percentile queries in multi-dimensional markov decision processes. Formal Methods Syst. Des., 50(2-3):207-248, 2017. URL: https://doi.org/10.1007/s10703-016-0262-7.
  28. Seinosuke Toda. PP is as Hard as the Polynomial-Time Hierarchy. SIAM Journal on Computing, 20(5):865-877, 1991. URL: https://doi.org/10.1137/0220053.
  29. Michael Ummels and Christel Baier. Computing quantiles in Markov reward models. In Frank Pfenning, editor, 16th International Conference on Foundations of Software Science and Computation Structures (FoSSaCS), volume 7794 of Lecture Notes in Computer Science, pages 353-368. Springer, 2013. URL: https://doi.org/10.1007/978-3-642-37075-5_23.
  30. Tom Verhoeff. Reward variance in Markov chains: A calculational approach. In Proceedings of Eindhoven FASTAR Days. Technische Universiteit Eindhoven, 2004. Google Scholar
  31. Li Xia. Risk-sensitive Markov decision processes with combined metrics of mean and variance. Production and Operations Management, 29(12):2808-2827, 2020. URL: https://doi.org/10.1111/poms.13252.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact