Oh that’s interesting! So you can have your “creditworthiness AI” \hat Y) predict that disproportionately more black (A) people are not creditworthy, as long as that’s (roughly) in line with the actual creditworthiness (Y) as determined by the “supervisor”.
That's also a good point -- evaluating fairness against labelled data breaks down if the labels themselves are biased in some way. IMO (and I am not familiar with the entire fair ML literature), we still lack technical tools to address biased data well.
I didn’t mean that they’re biased, just that being “black” correlates with “being poor”, which is not biased in any way but might still appear so based on superficial view(well, the data/AI isn’t biased, though the society might be but that’s not the bank’s problem...)
That correlation is a "bias", or maybe better an "unfairness".
Such a system would end up requiring black people have higher credit scores than otherwise equal white counterparts, in order to be granted whatever thing it is. That's, as modern feminist/race theory might define it, an implicit bias. Nowhere is anyone explicitly stating "black implies bad", but the way the model is built creates unequal representation.
No no, what I mean, and is AFAIK visible in the data, is that the poor/low credit people are disproportionately black. So even if the AI/statistics ignore race (as they should), the result will be that blacks have lower credit scores (on average, because they poorer).
That part is to be expected, the important thing is that you don't then recorrelate, either explicity or implicitly, race with the lower credit scores, that can essentially double count race (via income and then via the second check) and that's what you want to avoid.
Ok yeah then we agree. You definitely shouldn’t feed “irrelevant” features to the AI (i.e. those you don’t want it to discriminate/regress upon) otherwise it will overfit, in the best case to the noise, in the worst to the bias.
