2026-07-23T17:25:41Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:12022 2024-03-06T10:49:06Z ddc:004 open_access

Bias In, Bias Out? Evaluating the Folk Wisdom Rambachan, Ashesh Roth, Jonathan fairness selective labels discrimination training data We evaluate the folk wisdom that algorithmic decision rules trained on data produced by biased human decision-makers necessarily reflect this bias. We consider a setting where training labels are only generated if a biased decision-maker takes a particular action, and so "biased" training data arise due to discriminatory selection into the training data. In our baseline model, the more biased the decision-maker is against a group, the more the algorithmic decision rule favors that group. We refer to this phenomenon as bias reversal. We then clarify the conditions that give rise to bias reversal. Whether a prediction algorithm reverses or inherits bias depends critically on how the decision-maker affects the training data as well as the label used in training. We illustrate our main theoretical results in a simulation study applied to the New York City Stop, Question and Frisk dataset. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Ashesh Rambachan and Jonathan Roth 2020 Is Part Of LIPIcs, Volume 156, 1st Symposium on Foundations of Responsible Computing (FORC 2020) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.FORC.2020.6 urn:nbn:de:0030-drops-120225 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.FORC.2020.6 eng https://creativecommons.org/licenses/by/3.0/legalcode