Document Open Access Logo

Score Design for Multi-Criteria Incentivization

Authors Anmol Kabra, Mina Karzand, Tosca Lechner, Nati Srebro, Serena Wang

PDF

File

Thumbnail PDF
PDF
LIPIcs.FORC.2024.8.pdf
  • Filesize: 0.79 MB
  • 22 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Algorithmic game theory and mechanism design
  • Theory of computation → Computational geometry
Keywords
  • Multi-criteria incentives
  • Score-based incentives
  • Incentivizing improvement
  • Computational geometry

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

We present a framework for designing scores to summarize performance metrics. Our design has two multi-criteria objectives: (1) improving on scores should improve all performance metrics, and (2) achieving pareto-optimal scores should achieve pareto-optimal metrics. We formulate our design to minimize the dimensionality of scores while satisfying the objectives. We give algorithms to design scores, which are provably minimal under mild assumptions on the structure of performance metrics. This framework draws motivation from real-world practices in hospital rating systems, where misaligned scores and performance metrics lead to unintended consequences.

Cite As Get BibTex

Anmol Kabra, Mina Karzand, Tosca Lechner, Nati Srebro, and Serena Wang. Score Design for Multi-Criteria Incentivization. In 5th Symposium on Foundations of Responsible Computing (FORC 2024). Leibniz International Proceedings in Informatics (LIPIcs), Volume 295, pp. 8:1-8:22, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024) https://doi.org/10.4230/LIPIcs.FORC.2024.8

Author Details

Anmol Kabra
  • Toyota Technological Institute at Chicago, IL, USA
Mina Karzand
  • University of California at Davis, CA, USA
Tosca Lechner
  • University of Waterloo, Canada
Nati Srebro
  • Toyota Technological Institute at Chicago, IL, USA
Serena Wang
  • University of California at Berkeley, CA, USA
  • Google, Palo Alto, CA, USA

Funding

  • Kabra, Anmol: Supported in part through the NSF-TRIPODS Institute on Data, Economics, Algorithms and Learning (IDEAL).
  • Lechner, Tosca: Supported by a Vector Research Grant and a Apple Waterloo PhD fellowship for machine learning and data science.

Acknowledgements

Anmol Kabra thanks Naren Sarayu Manoj and Max Ovsiankin for pointers on convex analysis and geometry.

References

  1. Rahul Aggarwal, J Gmerice Hammond, Karen E Joynt Maddox, Robert W Yeh, and Rishi K Wadhera. Association between the proportion of black patients cared for at hospitals and financial penalties under value-based payment programs. Journal of American Medical Association, 325(12):1219-1221, 2021. Google Scholar
  2. Saba Ahmadi, Hedyeh Beyhaghi, Avrim Blum, and Keziah Naggita. Setting fair incentives to maximize improvement. arXiv preprint arXiv:2203.00134, 2022. Google Scholar
  3. Diane Alexander. How do doctors respond to incentives? unintended consequences of paying doctors to reduce costs. Journal of Political Economy, 128(11):4046-4096, 2020. Google Scholar
  4. Tal Alon, Magdalen Dobson, Ariel Procaccia, Inbal Talgam-Cohen, and Jamie Tucker-Foltz. Multiagent evaluation mechanisms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34 (02), pages 1774-1781, 2020. Google Scholar
  5. Arlene S Ash, Stephen F Fienberg, Thomas A Louis, Sharon-Lise T Normand, Therese A Stukel, and Jessica Utts. Statistical issues in assessing hospital performance. https://www.cms.gov/medicare/quality-initiatives-patient-assessment-instruments/hospitalqualityinits/downloads/statistical-issues-in-assessing-hospital-performance.pdf, 2012. Accessed: 2024-01-09.
  6. Deborah L Bandalos. Measurement theory and applications for the social sciences. Guilford Publications, 2018. Google Scholar
  7. Heski Bar-Isaac and Joel Shapiro. Credit ratings accuracy and analyst incentives. American Economic Review, 101(3):120-124, 2011. Google Scholar
  8. Matthew E Barclay, Mary Dixon-Woods, and Georgios Lyratzopoulos. Concordance of hospital ranks and category ratings using the current technical specification of us hospital star ratings and reasonable alternative specifications. In JAMA Health Forum, volume 3(5), pages e221006-e221006. American Medical Association, 2022. Google Scholar
  9. Kim C. Border. Caltech ec 181, lecture notes: Convex analysis and economic theory. https://healy.econ.ohio-state.edu/kcb/Ec181/, 2020.
  10. Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. Google Scholar
  11. Chicago Booth Review. Hospital ratings are deeply flawed. Can they be fixed? https://www.chicagobooth.edu/review/hospital-ratings-are-deeply-flawed-can-they-be-fixed, 2020. Accessed: 2024-01-09.
  12. Jeffrey Clemens and Joshua D Gottlieb. Do physicians' financial incentives affect medical treatment and patient health? American Economic Review, 104(4):1320-1349, 2014. Google Scholar
  13. CMS.gov. Hospital Value Based Purchasing (VBP) Program. https://qualitynet.cms.gov/inpatient/hvbp. Accessed: 2024-01-15.
  14. CMS.gov. Medicare 2024 Part C & D Star Ratings Technical Notes. https://www.cms.gov/files/document/2024-star-ratings-technical-notes.pdf. Accessed: 2024-04-10.
  15. CMS.gov. Overall Hospital Quality Star Ratings. https://qualitynet.cms.gov/inpatient/public-reporting/overall-ratings. Accessed: 2024-01-15.
  16. CMS.gov. Quality Payment Program: Merit-based Incentive Payment System (MIPS). https://qpp.cms.gov/mips/reporting-options-overview. Accessed: 2024-04-10.
  17. CMS.gov. Report to Congress: Risk Adjustment in Medicare Advantage. https://www.cms.gov/files/document/report-congress-risk-adjustment-medicare-advantage-december-2021.pdf. Accessed: 2024-04-10.
  18. Douglas A Conrad. The theory of value-based payment incentives and their application to health care. Health Services Research, 50:2057-2089, 2015. Google Scholar
  19. David Donoho and Victoria Stodden. When does non-negative matrix factorization give a correct decomposition into parts? Advances in neural information processing systems, 16, 2003. Google Scholar
  20. David Dranove and Paul Wehner. Physician-induced demand for childbirths. Journal of health economics, 13(1):61-73, 1994. Google Scholar
  21. José H Dulá, Richard V Helgason, and N Venugopal. An algorithm for identifying the frame of a pointed finite conical hull. INFORMS Journal on Computing, 10(3):323-330, 1998. Google Scholar
  22. David Gale. On inscribing n-dimensional sets in a regular n-simplex. Proceedings of the American Mathematical Society, 4(2):222-225, 1953. Google Scholar
  23. Nicolas Gillis. Nonnegative matrix factorization. SIAM, 2020. Google Scholar
  24. Nicolas Gillis and Stephen A Vavasis. Fast and robust recursive algorithmsfor separable nonnegative matrix factorization. IEEE transactions on pattern analysis and machine intelligence, 36(4):698-714, 2013. Google Scholar
  25. Charles AE Goodhart and CAE Goodhart. Problems of monetary management: the UK experience. Springer, 1984. Google Scholar
  26. Andrew S Grove. High output management. Vintage, 2015. Google Scholar
  27. Luke Guerdan, Amanda Coston, Kenneth Holstein, and Zhiwei Steven Wu. Counterfactual prediction under outcome measurement error. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1584-1598, 2023. Google Scholar
  28. Nika Haghtalab, Nicole Immorlica, Brendan Lucier, and Jack Z Wang. Maximizing welfare with incentive-aware evaluation mechanisms. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 160-166, 2021. Google Scholar
  29. Jason D Hartline, Yingkai Li, Liren Shan, and Yifan Wu. Optimization of scoring rules. arXiv preprint arXiv:2007.02905, 2020. Google Scholar
  30. Jason D Hartline, Liren Shan, Yingkai Li, and Yifan Wu. Optimal scoring rules for multi-dimensional effort. arXiv preprint arXiv:2211.03302, 2022. Google Scholar
  31. Bengt Holmstrom and Paul Milgrom. The firm as an incentive system. The American Economic Review, 84(4):972-991, 1994. URL: http://www.jstor.org/stable/2118041.
  32. Anmol Kabra, Mina Karzand, Tosca Lechner, Nati Srebro, and Serena Wang. Score design for multi-criteria incentivization. To appear on arXiv. Google Scholar
  33. Hyunmin Kim, Asos Mahmood, Noah E Hammarlund, and Cyril F Chang. Hospital value-based payment programs and disparity in the united states: A review of current evidence and future perspectives. Frontiers in Public Health, 10:882715, 2022. Google Scholar
  34. Jon Kleinberg and Manish Raghavan. How do classifiers induce agents to invest effort strategically? ACM Transactions on Economics and Computation (TEAC), 8(4):1-23, 2020. Google Scholar
  35. Daniel Koretz. The Testing Charade: Pretending to Make Schools Better. The University of Chicago Press, 2017. Google Scholar
  36. Abhishek Kumar, Vikas Sindhwani, and Prabhanjan Kambadur. Fast conical hull algorithms for near-separable non-negative matrix factorization. In International Conference on Machine Learning, pages 231-239. PMLR, 2013. Google Scholar
  37. Nisha Kurian, Jyotsna Maid, Sharoni Mitra, Lance Rhyne, Michael Korvink, and Laura H Gunn. Predicting hospital overall quality star ratings in the usa. In Healthcare, volume 9(4), page 486. MDPI, 2021. Google Scholar
  38. Lydia T Liu, Solon Barocas, Jon Kleinberg, and Karen Levy. On the actionability of outcome prediction. arXiv preprint arXiv:2309.04470, 2023. Google Scholar
  39. Francisco J López. An algorithm to find the lineality space of the positive hull of a set of vectors. Journal of Mathematical Modelling and Algorithms, 10(1):1-30, 2011. Google Scholar
  40. Jerry Muller. The tyranny of metrics. Princeton University Press, 2018. Google Scholar
  41. Committee on Quality of Health Care in America. Crossing the quality chasm: a new health system for the 21st century. National Academies Press, 2001. Google Scholar
  42. Joseph O'Rourke, Alok Aggarwal, Sanjeev Maddila, and Michael Baldwin. An optimal algorithm for finding minimal enclosing triangles. Journal of Algorithms, 7(2):258-269, 1986. Google Scholar
  43. Esther Rolf, Max Simchowitz, Sarah Dean, Lydia T Liu, Daniel Bjorkegren, Moritz Hardt, and Joshua Blumenstock. Balancing competing objectives with noisy data: Score-based classifiers for welfare-aware machine learning. In International Conference on Machine Learning, pages 8158-8168. PMLR, 2020. Google Scholar
  44. Kirsten Schardt, Lorraine Hutzler, Joseph Bosco, Casey Humbyrd, and Matt DeCamp. Increase in healthcare disparities: The unintended consequences of value-based medicine, lessons from the total joint bundled payments for care improvement. Bulletin of the NYU Hospital for Joint Diseases, 78(2):93-97, 2020. Google Scholar
  45. Alexander Schrijver. Theory of linear and integer programming. John Wiley & Sons, 1998. Google Scholar
  46. Marilyn Strathern. `improving ratings': audit in the british university system. European review, 5(3):305-321, 1997. Google Scholar
  47. The New York Times. The Hype over Hospital Rankings. https://www.nytimes.com/2013/07/28/sunday-review/the-hype-over-hospital-rankings.html, 2013. Accessed: 2024-01-09.
  48. Godfried T Toussaint. Solving geometric problems with the rotating calipers. In Proc. IEEE Melecon, volume 83, page A10, 1983. Google Scholar
  49. U.S. News and World Report. FAQ: How and Why We Rank and Rate Hospitals. https://health.usnews.com/health-care/best-hospitals/articles/faq-how-and-why-we-rank-and-rate-hospitals, 2023. Accessed: 2024-01-15.
  50. Stephen A Vavasis. On the complexity of nonnegative matrix factorization. SIAM Journal on Optimization, 20(3):1364-1377, 2010. Google Scholar
  51. Rishi K Wadhera, Jose F Figueroa, Karen E Joynt Maddox, Lisa S Rosenbaum, Dhruv S Kazi, and Robert W Yeh. Quality measure development and associated spending by the centers for medicare & medicaid services. JAMA, 323(16):1614-1616, 2020. Google Scholar
  52. Serena Wang, Stephen Bates, PM Aronow, and Michael I Jordan. Operationalizing counterfactual metrics: Incentives, ranking, and information asymmetry. arXiv preprint arXiv:2305.14595, 2023. Google Scholar
  53. Roger J-B Wets and Christoph Witzgall. Algorithms for frames and lineality spaces of cones. Journal of Research of the National Bureau of Standards, 71:1-7, 1967. Google Scholar
  54. Lawrence J White. Credit rating agencies: An overview. Annu. Rev. Financ. Econ., 5(1):93-122, 2013. Google Scholar
  55. Lofti Zadeh. Optimality and non-scalar-valued performance criteria. IEEE transactions on Automatic Control, 8(1):59-60, 1963. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact