2026-07-26T16:44:53Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:23149 2025-10-02T12:35:19Z ddc:004 open_access

When Does a Predictor Know Its Own Loss? Gollakota, Aravind Gopalan, Parikshit Karan, Aayush Peale, Charlotte Wieder, Udi loss prediction multicalibration active learning algorithmic fairness calibration predictive uncertainty uncertainty estimation machine learning theory Given a predictor and a loss function, how well can we predict the loss that the predictor will incur on an input? This is the problem of loss prediction, a key computational task associated with uncertainty estimation for a predictor. In a classification setting, a predictor will typically predict a distribution over labels and hence have its own estimate of the loss that it will incur, given by the entropy of the predicted distribution. Should we trust this estimate? In other words, when does the predictor know what it knows and what it does not know? In this work we study the theoretical foundations of loss prediction. Our main contribution is to establish tight connections between nontrivial loss prediction and certain forms of multicalibration [Ursula Hébert-Johnson et al., 2018], a multigroup fairness notion that asks for calibrated predictions across computationally identifiable subgroups. Formally, we show that a loss predictor that is able to improve on the self-estimate of a predictor yields a witness to a failure of multicalibration, and vice versa. This has the implication that nontrivial loss prediction is in effect no easier or harder than auditing for multicalibration. We support our theoretical results with experiments that show a robust positive correlation between the multicalibration error of a predictor and the efficacy of training a loss predictor. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Aravind Gollakota and Parikshit Gopalan and Aayush Karan and Charlotte Peale and Udi Wieder 2025 Is Part Of LIPIcs, Volume 329, 6th Symposium on Foundations of Responsible Computing (FORC 2025) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.FORC.2025.22 urn:nbn:de:0030-drops-231490 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.FORC.2025.22 eng https://creativecommons.org/licenses/by/4.0/legalcode