In recent years, organizations in both the public and private sphere have made widespread use of predictive analytics and machine learning for use cases such as college admissions, loan applications, airport screenings, and of course, advertising. These applications not only drive speed and efficiency, but there is an underlying assumption that decisions with social justice implications are best made by data-driven algorithms, because they are inherently impartial.
If only that were true. As it turns out, data is socially constructed, and inherits our human imperfections and biases with startling fidelity. So too do algorithms trained on these biased datasets, and the effects are very difficult to detect. Instead of curbing the potential for systemic discrimination against disadvantaged groups, many researchers believe that the use of algorithms has actually expanded it.
Consider the criminal justice system in the United States. In recent years, courts have been relying on a variety of third-party predictive algorithms to quantify the risk that a convicted criminal will commit a future crime (known as recidivism). Historically, judges have made these subjective determinations based on personal experience and professional expertise; the introduction of an objective, data-driven algorithm into these settings seems like a sensible thing to do. Indeed, it sounds like a marquee application for the field of machine learning.
Here’s the problem: in 2016, ProPublica published an analysis of the COMPAS Recidivism Risk Score algorithm showing that it was racially discriminatory towards Black defendants. According to the article, Black defendants were “77 percent more likely to be pegged as at higher risk of committing a future violent crime and 45 percent more likely to be predicted to commit a future crime of any kind”. Given that these risk scores were being shown to judges minutes before they presided over sentencing hearings, the implications are quite troubling; as of 2016, at least 9 US state courts were actively using COMPAS.
COMPAS is a proprietary algorithm and its publisher has declined to release the exact model specification; however, we do know that it is based on a questionnaire that includes criminal history as well as a set of behavioral questions. For example, it asks defendants questions such as “How many of your friends/acquaintances are taking drugs illegally?” and “How often did you get in fights at school?”. It also defendants to agree/disagree with statements such as “A hungry person has a right to steal” and “When people do minor offenses or use drugs they don’t hurt anyone except themselves”.
Notably, race is not referenced in the questionnaire; however, that does not mean it isn’t correlated with the above questions. These hidden correlations allow race to influence the model just as effectively as if it were included as an explicit variable. Predictive models that are “race-blinded” are simply blind to the fact that they are, in fact, deeply racist.
One might object here on philosophical grounds. After all, if a “protected” attribute is truly correlated with an important outcome, then by definition, the rest of us are worse-off for not being able to take advantage of this information. The fundamental principle behind establishing “protected attributes” is the axiomatic notion that most observed differences in racial/ethnic/gender groups are either the result of historical imbalances in opportunity, or reduced data quality due to smaller sample size. Absent those imbalances, we posit that we would not see, for example, differences in high school completion rates among White and Black students, or SAT scores between Hispanic and Asian students. By reading into these differences and using them to make important decisions, we are perpetuating the cycle of unfairness.
Thus far, we have seen that simply ignoring protected attributes is not a viable strategy for guarding against discrimination in our algorithms. An alternative is to control for the effects by establishing different threshold-criteria for various disadvantaged groups. For example, a bank granting loan applications on the basis of a predictive model may notice that Hispanic applicants have an average score of 45 (on a hypothetical scale of 1-100), whereas White applicants have an average score of 52. As a result, it observes that, on average, White applicants are approved 62% of the time, whereas Hispanic applicants only receive a loan in 48% of cases.
In this case, the bank can curb the discriminatory behavior of the algorithm by adjusting the decision-making threshold based on demographic criteria so as to bring the two acceptance rates into alignment. In a sense, this is reverse discrimination, but with the explicit intent to harmonize acceptance rates among the two populations.
Yet there are problems with this approach. First, there is the obvious fact that acceptance criteria varies based on a protected attribute, i.e., all other things being equal, under this scenario a less qualified Hispanic applicant has the same chance at getting a loan as a more qualified White applicant (due to the manipulation of thresholds). Moreover, there is a significant cost borne by society or the private enterprise by deviating from the “optimal solution”, which in this fictional scenario would accept White applications at a higher rate than Hispanic ones. Can we do better than this?
It turns out we can. In a journal article titled Equality of Opportunity in Supervised Learning, researchers Moritz Hardt, Eric Price, and Nathan Srebro propose a framework for post-processing any learned predictor to apply a condition known as “equality of opportunity”. Equality of opportunity (illustrated here) is the idea that, instead of harmonizing acceptance rates by demographic across the entire body of loan applicants (“positive rate”), we only need to ensure that the acceptance rate is equal for applicants who would actually pay back a loan (“true positive rate”).
Equality of opportunity provides an interesting alternative to the status quo. The best feature is that it is a post-processing step, meaning that the principle can be applied to existing predictive models. This is especially important in the scenario where organizations do not have access to the source code for a third-party algorithm, but still need to make use of it. It will be interesting to see how institutions will come to adopt equality of opportunity in the months and years ahead.