Why Explainable AI Matters

Why Explainable AI Matters
By Severin Perez | March 2, 2022

When justifying a decision or rule for their kids, parents sometimes resort to an old standby: “because I said so.” It’s an unsatisfying, but effective response–especially in light of the uneven power dynamic between parents and children. In essence, many artificial intelligence (AI) systems, and the organizations that deploy them, are now using the same strategy. Why were you denied a bank loan? Because the AI said so. Why did the police arrest you? Because the AI said so. Why are you getting ads for extremist political causes? Because the AI said so.

We don’t accept the “because I said so” argument from judges, police officers, doctors, or executives, and we shouldn’t accept it from AI systems either. In order to verify that an AI is making a valid decision, it should be an explainable AI. If the AI is explainable, then we can ensure that its processes align with our laws, regulations, and social norms. Further, an explainable AI is one that we can interrogate to confirm that no privacy harms are inherent in the system itself, or the data that feeds it.

Explainable AI

“AI and Machine Learning” by [Mike MacKenzie](https://www.flickr.com/photos/152824664@N07/30212411048/) is licensed under CC-BY-2.0.

An explainable AI is one that behaves in a way that most users can understand. When the system produces an output, we should be able to say why and how. In order to be explainable, the AI must also be interpretable, meaning that we have an idea about how the internal technology operates. In other words, the system is not a “black box”, where data goes in and decisions come out, with no clear connection between the two. [1]

Consider an AI used by a bank to decide which customers qualify for a loan. If the AI is explainable, then we should be able to identify which variables, weights, and mechanisms it uses to make decisions. For example, the AI might take the customer’s savings account balance (SAB) and requested loan amount (RLA) as variables and make a decision based on the formula “if RLA is less than 2 * SAB, then approve the loan, otherwise deny the loan.”

Of course, the above example is a drastic oversimplification. Realistically, an AI for approving bank loans will consider a vast amount of data about bank customers, put the data through a neural network optimized for accuracy, and output decisions that even the system designers may not fully understand. This raises serious questions about fairness, bias, and transparency.

Potential for Harm

AI systems are now making decisions in a variety of fields that were previously the exclusive purview of expert human decision-makers, including justice, law enforcement, health, and commerce. This is problematic not only because we want such systems to be accurate, but also because they can have baked-in bias that perpetuates patterns of discrimination and inequity–even in cases when the designers and users are not themselves bad actors. [2]

The good news is that explainable AI can provide us with the means to identify the sources of bias in a system. Since an explainable AI system is transparent by nature, we can evaluate what data it is using, and how. This is an important mechanism for identifying privacy harms at the information processing stage of data usage. [3] Further, if we can see what data a system is using, we can ask follow-on questions about where the data came from, whether it’s accurate, and whether we feel it is sensitive and merits special protections. In other words, we can hold the AI and its designers accountable.

Never Trust, Always Verify

“Paying with your face” by [Eliza Galstyan](https://commons.wikimedia.org/wiki/File:Paying-with-your-face.jpg) is licensed under CC-BY-SA-4.0.

As much as we might like to live in a world where we can universally trust governments and corporations to use our data responsibly, that world doesn’t yet exist. Companies like Clearview AI are harvesting photos from social media and using them to feed a facial recognition system that is popular with law enforcement. [4] Similarly, in 2018 we learned that Cambridge Analytica had been improperly acquiring data from Facebook to build voter profiles for use in election campaigns. [5]

If Clearview AI and Cambridge Analytica had followed the principles of explainable AI, either by virtue of social norms or regulatory requirement, then we would have had earlier and more frequent opportunities to raise questions about possible abuse of our data. Not only could we have asked whether they had our consent to use our data, but we could also have evaluated the mechanisms their systems used to make decisions about us. As it stands now though, such companies are unaccountable to their data subjects.

In order to avoid such abuses, one argument is to employ robust privacy policies so that consumers can make informed choices about how and where they generate data. Although this is a worthy goal, it’s not enough on its own. In reality, regulations like the European Union’s General Data Protection Regulation (GDPR) have helped to drive an increase in the length and difficulty of privacy policies, making it even harder for the average consumer to understand them. [6] Explainable AI would provide additional insights into the ways in which systems are using our data, making it easier for us to identify and mitigate potential harms.

Trade Offs

Of course, explainable AI isn’t free–there are trade offs that we must consider, including with overall accuracy and performance. [1] “Black box” AI systems have become popular not because designers prefer opacity, but because they’re more effective than fully transparent systems. Somewhat paradoxically, explainable AI may also introduce new privacy risks by opening additional attack vectors for malicious actors to steal data from a system. [7]

As AI usage spreads to new areas of human life, it’s up to us to decide how such tools should be used. It may be that we don’t need to understand how many AI systems work, because the decisions they make may not be all that sensitive. In other cases though, as in justice and health, the trade-off between explainability and accuracy merits deeper consideration. The choices we make in these cases will have long-lasting implications, not only for our social values, but for how we live our lives.

Sources

1. Bundy, A., Crowcroft, J., Ghahramani, Z., Reid, N., Weller, A., McCarthy, N., & Montgomery, J. (2019). Explainable AI: the basics. The Royal Society. https://royalsociety.org/topics-policy/projects/explainable-ai/
2. Hoffman, Anna L. (2019). Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse. Information, Communication & Society, Volume 22, No. 7, 900-915. https://doi.org/10.1080/1369118X.2019.1573912
3. Solove, Daniel. (2006). A Taxonomy of Privacy. University of Pennsylvania Law Review, Vol. 154, No. 3, p. 477, January 2006, GWU Law School Public Law Research Paper No. 129. https://ssrn.com/abstract=667622
4. Hill, Kashmir. (2020, January 18). The Secretive Company That Might End Privacy as We Know It. The New York Times. https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html
5. Confessore, Nicholas. (2018, April 4). Cambridge Analytica and Facebook: The Scandal and the Fallout So Far. The New York Times. https://www.nytimes.com/2018/04/04/us/politics/cambridge-analytica-scandal-fallout.html
6. Amos, R., Acar, G., Lucherini, E., Kshirsagar, M., Narayanan, A., & Mayer, J. (2020). Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset. arXiv. https://arxiv.org/abs/2008.09159
7. Zhao, X., Zhang, W., Xiao, X., & Lim, B. (2021). Exploiting Explanations for Model Inversion Attacks. arXiv. https://arxiv.org/abs/2104.12669
8. Kerry, Cameron F. (2020). Protecting privacy in an AI-driven world. Brookings. https://www.brookings.edu/research/protecting-privacy-in-an-ai-driven-world/