Document Open Access Logo

More Dominantly Truthful Multi-Task Peer Prediction with a Finite Number of Tasks

Author Yuqing Kong

PDF
Thumbnail PDF

File

LIPIcs.ITCS.2022.95.pdf
  • Filesize: 1.61 MB
  • 20 pages

Document Identifiers

Author Details

Yuqing Kong
  • The Center on Frontiers of Computing Studies, Peking University, Beijing, China

Funding

  • Kong, Yuqing: This work is supported by National Natural Science Foundation of China award number 62002001.

Cite AsGet BibTex

Yuqing Kong. More Dominantly Truthful Multi-Task Peer Prediction with a Finite Number of Tasks. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022). Leibniz International Proceedings in Informatics (LIPIcs), Volume 215, pp. 95:1-95:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2022)
https://doi.org/10.4230/LIPIcs.ITCS.2022.95

Abstract

In the setting where we ask participants multiple similar possibly subjective multi-choice questions (e.g. Do you like Bulbasaur? Y/N; do you like Squirtle? Y/N), peer prediction aims to design mechanisms that encourage honest feedback without verification. A series of works have successfully designed multi-task peer prediction mechanisms where reporting truthfully is better than any other strategy (dominantly truthful), while they require an infinite number of tasks. A recent work proposes the first multi-task peer prediction mechanism, Determinant Mutual Information (DMI)-Mechanism, where not only is dominantly truthful but also works for a finite number of tasks (practical). However, the existence of other practical dominantly-truthful multi-task peer prediction mechanisms remains to be an open question. This work answers the above question by providing - a new family of information-monotone information measures: volume mutual information (VMI), where DMI is a special case; - a new family of practical dominantly-truthful multi-task peer prediction mechanisms, VMI-Mechanisms. To illustrate the importance of VMI-Mechanisms, we also provide a tractable effort incentive optimization goal. We show that DMI-Mechanism may not be not optimal but we can construct a sequence of VMI-Mechanisms that are approximately optimal. The main technical highlight in this paper is a novel geometric information measure, Volume Mutual Information, that is based on a simple idea: we can measure an object A’s information amount by the number of objects that is less informative than A. Different densities over the object lead to different information measures. This also gives Determinant Mutual Information a simple geometric interpretation.

Subject Classification

ACM Subject Classification
  • Theory of computation → Algorithmic game theory and mechanism design
Keywords
  • Information elicitation
  • information theory

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Related Versions

References

  1. Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Working draft, November, 3, 2005. Google Scholar
  2. Y. Cai, C. Daskalakis, and C. H. Papadimitriou. Optimum statistical estimation with strategic data sources. Eprint Arxiv, 42(5):590-595, 2014. Google Scholar
  3. Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous proficiency. In Proceedings of the 22nd international conference on World Wide Web, pages 319-330. International World Wide Web Conferences Steering Committee, 2013. Google Scholar
  4. Boi Faltings, Radu Jurca, Pearl Pu, and Bao Duy Tran. Incentives to counter bias in human computation. In Second AAAI Conference on Human Computation and Crowdsourcing, 2014. Google Scholar
  5. Rafael M. Frongillo and Jens Witkowski. A geometric perspective on minimal peer prediction. ACM Trans. Economics and Comput., 5(3):17:1-17:27, 2017. URL: https://doi.org/10.1145/3070903.
  6. Jason D. Hartline, Yingkai Li, Liren Shan, and Yifan Wu. Optimization of scoring rules. CoRR, abs/2007.02905, 2020. URL: http://arxiv.org/abs/2007.02905.
  7. Vijay Kamble, Nihar Shah, David Marn, Abhay Parekh, and Kannan Ramachandran. Truth serums for massively crowdsourced evaluation tasks. arXiv preprint arXiv:1507.07045, 2015. Google Scholar
  8. Yuqing Kong. Dominantly truthful multi-task peer prediction with a constant number of tasks. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 2398-2411. SIAM, 2020. Google Scholar
  9. Yuqing Kong and Grant Schoenebeck. Equilibrium selection in information elicitation without verification via information monotonicity. In 9th Innovations in Theoretical Computer Science Conference, ITCS 2018, January 11-14, 2018, Cambridge, MA, USA, pages 13:1-13:20, 2018. URL: https://doi.org/10.4230/LIPIcs.ITCS.2018.13.
  10. Yuqing Kong and Grant Schoenebeck. An information theoretic framework for designing information elicitation mechanisms that reward truth-telling. ACM Trans. Econ. Comput., 7(1):2:1-2:33, January 2019. URL: https://doi.org/10.1145/3296670.
  11. Y. Liu, J. Wang, and Y. Chen. Surrogate scoring rules. In EC '20: The 21st ACM Conference on Economics and Computation, 2020. Google Scholar
  12. E. C. Merkle and M. Steyvers. Choosing a strictly proper scoring rule. Decision Analysis, 10(4):292-304, 2013. Google Scholar
  13. N. Miller, P. Resnick, and R. Zeckhauser. Eliciting informative feedback: The peer-prediction method. Management Science, pages 1359-1373, 2005. Google Scholar
  14. Eric Neyman, Georgy Noarov, and S. Matthew Weinberg. Binary scoring rules that incentivize precision. In Proceedings of the 22nd ACM Conference on Economics and Computation, EC '21, pages 718-733, New York, NY, USA, 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3465456.3467639.
  15. Kent Osband. Optimal forecasting incentives. Journal of Political Economy, 97(5):1091-1112, 1989. Google Scholar
  16. D. Prelec. A Bayesian Truth Serum for subjective data. Science, 306(5695):462-466, 2004. Google Scholar
  17. Goran Radanovic and Boi Faltings. Incentives for truthful information elicitation of continuous signals. In Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014. Google Scholar
  18. Goran Radanovic and Boi Faltings. Incentive schemes for participatory sensing. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1081-1089. International Foundation for Autonomous Agents and Multiagent Systems, 2015. Google Scholar
  19. Goran Radanovic, Boi Faltings, and Radu Jurca. Incentives for effort in crowdsourcing using the peer truth serum. ACM Trans. Intell. Syst. Technol., 7(4), March 2016. URL: https://doi.org/10.1145/2856102.
  20. C. E. Shannon. A mathematical theory of communication. SIGMOBILE Mob. Comput. Commun. Rev., 5(1):3-55, January 2001. URL: https://doi.org/10.1145/584091.584093.
  21. Victor Shnayder, Arpit Agarwal, Rafael Frongillo, and David C Parkes. Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation, pages 179-196. ACM, 2016. Google Scholar
  22. Leon Simon et al. Lectures on geometric measure theory. The Australian National University, Mathematical Sciences Institute, 1983. Google Scholar
  23. J. Witkowski and D. Parkes. A robust Bayesian Truth Serum for small populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI 2012), 2012. Google Scholar
  24. Luis Zermeno. A principal-expert model and the value of menus. unpublished paper, Massachusetts Institute of Technology, 4, 2011. Google Scholar
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact