Document Open Access Logo

Limitations on Accurate, Trusted, Human-Level Reasoning

Authors Rina Panigrahy, Vatsal Sharan

PDF

File

Thumbnail PDF
PDF
LIPIcs.FORC.2026.11.pdf
  • Filesize: 0.8 MB
  • 21 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Computing methodologies → Philosophical/theoretical foundations of artificial intelligence
  • Computing methodologies → Machine learning
  • Theory of computation → Computational complexity and cryptography
Keywords
  • Accuracy
  • Safety
  • Trust
  • Complexity-theoretic limitations

Metrics

Abstract

We identify a fundamental incompatibility between the goals of accuracy, trust, and human-level reasoning in artificial intelligence (AI) systems, for strict mathematical definitions of these notions. We define accuracy of a system as the property that it never makes any false claims when it has the ability to abstain from making a prediction on any input, and trust as the assumption that the system is accurate. We define human-level reasoning as the property of an AI system always matching or exceeding human capability. Our core finding is that - for our formal definitions of these notions - an accurate and trusted AI system cannot be a human-level reasoning system: for such an accurate, trusted system there are task instances which are easily and provably solvable by a human but not by the system. Our proofs draw parallels to Gödel’s incompleteness theorems and Turing’s proof of the undecidability of the halting problem, and can be regarded as interpretations of Gödel’s and Turing’s results. Key to our proof is the formalization of the notion of trust, which allows us to separate the intrinsic property of a system (being accurate) from its epistemic status (being trusted).

Cite As Get BibTex

Rina Panigrahy and Vatsal Sharan. Limitations on Accurate, Trusted, Human-Level Reasoning. In 7th Symposium on Foundations of Responsible Computing (FORC 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 368, pp. 11:1-11:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.FORC.2026.11

Author Details

Rina Panigrahy
  • Google Research, Mountain View, CA, USA
Vatsal Sharan
  • University of Southern California, Los Angeles, CA, USA

Funding

  • Sharan, Vatsal: VS was supported by NSF awards 2239265, 2346058 and an Okawa Foundation Research Grant. The work was done in part while VS was visiting the Simons Institute for the Theory of Computing. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any funding agencies such as the National Science Foundation.

Acknowledgements

We thank Prabhakar Raghavan, Mukund Raghothaman and anonymous reviewers for insightful discussions and helpful feedback.

References

  1. Manuel Alfonseca, Manuel Cebrian, Antonio Fernandez Anta, Lorenzo Coviello, Andrés Abeliuk, and Iyad Rahwan. Superintelligence cannot be contained: Lessons from computability theory. Journal of Artificial Intelligence Research, 70:65-76, 2021. URL: https://doi.org/10.1613/JAIR.1.12202.
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint, 2016. URL: https://arxiv.org/abs/1606.06565.
  3. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. Managing extreme AI risks amid rapid progress. Science, 384(6698):842-845, 2024. Google Scholar
  4. Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, et al. International AI safety report. arXiv preprint, 2025. URL: https://arxiv.org/abs/2501.17805.
  5. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint, 2021. URL: https://arxiv.org/abs/2108.07258.
  6. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014. Google Scholar
  7. Mario Brcic and Roman V Yampolskiy. Impossibility results in AI: A survey. ACM computing surveys, 56(1):1-24, 2023. URL: https://doi.org/10.1145/3603371.
  8. The Center for AI Safety. Statement on AI risk. https://aistatement.com/, 2025. Accessed: 2025-07-23.
  9. David J Chalmers. Minds, machines, and mathematics. Psyche, 2(9):117-18, 1995. Google Scholar
  10. Chen Chen, Xueluan Gong, Ziyao Liu, Weifeng Jiang, Si Qi Goh, and Kwok-Yan Lam. Trustworthy, responsible, and safe AI: A comprehensive architectural framework for AI safety with challenges and mitigations. arXiv preprint, 2024. URL: https://arxiv.org/abs/2408.12935.
  11. Jaymari Chua, Yun Li, Shiyi Yang, Chen Wang, and Lina Yao. AI safety in generative AI large language models: A survey. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2407.18369.
  12. Michael Chui, Eric Hazan, Roger Roberts, Alex Singla, and Kate Smaje. The economic potential of generative AI, 2023. Google Scholar
  13. Edmund M Clarke, Thomas A Henzinger, Helmut Veith, and Roderick Bloem. Handbook of Model Checking. Springer, 2018. Google Scholar
  14. A Philip Dawid. The well-calibrated bayesian. Journal of the American statistical Association, 77(379):605-610, 1982. Google Scholar
  15. Peter Doggers. Will this position help understand human consciousness? https://www.chess.com/news/view/will-this-position-help-to-understand-human-consciousness-4298, 2017. Accessed: 2025-07-21.
  16. Eyal Even-Dar, Shie Mannor, and Yishay Mansour. PAC bounds for multi-armed bandit and markov decision processes. In International Conference on Computational Learning Theory, pages 255-270. Springer, 2002. URL: https://doi.org/10.1007/3-540-45435-7_18.
  17. Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. How far are we from AGI: Are LLMs all we need? Transactions on Machine Learning Research, 2024. Google Scholar
  18. Future of Life Institute. AI safety index 2024. https://futureoflife.org/wp-content/uploads/2024/12/AI-Safety-Index-2024-Full-Report-27-May-25.pdf, 2024. Accessed: 2025-07-23.
  19. Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017. Google Scholar
  20. Ben Goertzel and Cassio Pennachin. Artificial General Intelligence. Springer, 2007. Google Scholar
  21. Kurt Gödel. Über formal unentscheidbare sätze der principia mathematica und verwandter systeme i. Monatshefte für Mathematik und Physik, 38(1):173-198, 1931. Google Scholar
  22. C. A. R. Hoare. An axiomatic basis for computer programming. Communications of the ACM, 12(10):576-580, 1969. URL: https://doi.org/10.1145/363235.363259.
  23. Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 624-635, 2021. URL: https://doi.org/10.1145/3442188.3445923.
  24. Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck. lil’UCB: An optimal exploration algorithm for multi-armed bandits. In Conference on Learning Theory, pages 423-439. PMLR, 2014. URL: http://proceedings.mlr.press/v35/jamieson14.html.
  25. Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed bandits. In International conference on machine learning, pages 1238-1246. PMLR, 2013. Google Scholar
  26. Manfred Kerber. Why is the Lucas-Penrose argument invalid? In Annual Conference on Artificial Intelligence, pages 380-393. Springer, 2005. URL: https://doi.org/10.1007/11551263_30.
  27. Geoffrey LaForte, Patrick J Hayes, and Kenneth M Ford. Why Gödel’s theorem cannot refute computationalism. Artificial Intelligence, 104(1-2):265-286, 1998. URL: https://doi.org/10.1016/S0004-3702(98)00052-6.
  28. Steven M LaValle. Planning algorithms. Cambridge university press, 2006. Google Scholar
  29. Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and machines, 17(4):391-444, 2007. URL: https://doi.org/10.1007/S11023-007-9079-X.
  30. Shane Legg, Marcus Hutter, et al. A collection of definitions of intelligence. Frontiers in Artificial Intelligence and applications, 157:17, 2007. Google Scholar
  31. John R Lucas. Minds, machines and Gödel. Philosophy, 36(137):112-127, 1961. Google Scholar
  32. Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, et al. Artificial intelligence index report 2025. arXiv preprint, 2025. URL: https://arxiv.org/abs/2504.07139.
  33. John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon. A proposal for the Dartmouth summer research project on artificial intelligence. http://jmc.stanford.edu/articles/dartmouth/dartmouth.pdf, 1955.
  34. Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8-30, 1961. Google Scholar
  35. Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Position: Levels of AGI for operationalizing progress on the path to AGI. In Forty-first International Conference on Machine Learning, 2024. Google Scholar
  36. Roger Penrose and Martin Gardner. The Emperor’s New Mind: Concerning Computers, Minds, and The Laws of Physics. Oxford University Press, 1989. Google Scholar
  37. Henry Gordon Rice. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical society, 74(2):358-366, 1953. Google Scholar
  38. David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. Tackling climate change with machine learning. ACM Computing Surveys (CSUR), 55(2):1-96, 2022. URL: https://doi.org/10.1145/3485128.
  39. Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Viking, 2019. Google Scholar
  40. Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Pearson, 3rd edition, 2016. Google Scholar
  41. Herbert A Simon. Models of Man: Social and Rational. Wiley, 1957. Google Scholar
  42. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, 31(3):943-950, 2025. Google Scholar
  43. Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable AGI. arXiv preprint, 2023. URL: https://doi.org/10.48550/arXiv.2309.01933.
  44. Alan M. Turing. On computable numbers, with an application to the entscheidungsproblem. Proceedings of the London Mathematical Society, s2-42(1):230-265, 1937. URL: https://doi.org/10.1112/PLMS/S2-42.1.230.
  45. Alan M. Turing. Computing machinery and intelligence. Mind, LIX(236):433-460, 1950. URL: https://doi.org/10.1093/MIND/LIX.236.433.
  46. Alan M. Turing. Intelligent machinery, a heretical theory. https://uberty.org/wp-content/uploads/2015/02/intelligent-machinery-a-heretical-theory.pdf, 1951. Accessed: 2025-07-21.
  47. Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131, 1974. Google Scholar
  48. Ben Van Calster, David J McLernon, Maarten van Smeden, Laure Wynants, Ewout W Steyerberg, et al. Calibration: the achilles heel of predictive analytics. BMC Medicine, 17:230, 2019. Google Scholar
  49. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60, 2023. URL: https://doi.org/10.1038/S41586-023-06221-2.
  50. Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S Yu, and Qingsong Wen. Large language models for education: A survey and outlook. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2403.18105.
  51. Norbert Wiener. The Human Use of Human Beings: Cybernetics and Society. Houghton Mifflin, Boston, 1950. Google Scholar
  52. Wikipedia. https://en.wikipedia.org/wiki/Penrose%E2%80%93Lucas_argument. Accessed: 2025-07-21.
  53. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2401.11817.
  54. Tom Zahavy, Vivek Veeriah, Shaobo Hou, Kevin Waugh, Matthew Lai, Edouard Leurent, Nenad Tomasev, Lisa Schut, Demis Hassabis, and Satinder Singh. Diversifying AI: Towards creative chess with AlphaZero. arXiv preprint, 2023. URL: https://doi.org/10.48550/arXiv.2308.09175.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact