Why AI Should advance Slow and Fix Things

Joy Buolamwini‘s AI research was attracting attention years before she received her Ph.D. from the MIT Media Lab in 2022. As a graduate student, she made waves with a 2016 TED talk about algorithmic bias that has received more than 1.6 million views to date. In the talk, Buolamwini, who is Black, showed that standard facial detection systems didn’t recognize her face unless she put on a white mask. During the talk, she also brandished a shield emblazoned with the logo of her new organization, the Algorithmic Justice League, which she said would fight for people harmed by AI systems, people she would later come to call the excoded.

In her new book, Unmasking AI: My Mission to Protect What Is Human in a World of Machines, Buolamwini describes her own awakenings to the clear and present dangers of today’s AI. She explains her research on facial recognition systems and the Gender Shades research project, in which she showed that commercial gender classification systems consistently misclassified dark-skinned women. She also narrates her stratospheric rise—in the years since her TED talk, she has presented at the World Economic Forum, testified before Congress, and participated in President Biden’s roundtable on AI.

While the book is an interesting read on a autobiographical level, it also contains useful prompts for AI researchers who are ready to question their assumptions. She reminds engineers that default settings are not neutral, that convenient datasets may be rife with ethical and legal problems, and that benchmarks aren’t always assessing the right things. Via email, she answered IEEE Spectrum‘s questions about how to be a principled AI researcher and how to change the status quo.

One of the most interesting parts of the book for me was your detailed description of how you did the research that became Gender Shades: how you figured out a data collection method that felt ethical to you, struggled with the inherent subjectivity in devising a classification scheme, did the labeling labor yourself, and so on. It seemed to me appreciate the opposite of the Silicon Valley “advance fast and break things” ethos. Can you visualize a world in which every AI researcher is so scrupulous? What would it take to get to such a state of affairs?

Joy Buolamwini: When I was earning my academic degrees and learning to code, I did not have examples of ethical data collection. Basically if the data were available online it was there for the taking. It can be difficult to visualize another way of doing things, if you never see an alternative pathway. I do believe there is a world where more AI researchers and practitioners exercise more caution with data-collection activities, because of the engineers and researchers who accomplish out to the Algorithmic Justice League looking for a better way. Change starts with conversation, and we are having important conversations today about data provenance, classification systems, and AI harms that when I started this work in 2016 were often seen as insignificant.

What can engineers do if they’re concerned about algorithmic bias and other issues regarding AI ethics, but they work for a typical big tech company? The kind of place where nobody questions the use of convenient datasets or asks how the data was collected and whether there are problems with consent or bias? Where they’re expected to produce results that measure up against standard benchmarks? Where the choices seem to be: Go along with the status quo or find a new job?

Buolamwini: I cannot stress enough the importance of documentation. In conducting algorithmic audits and approaching well-known tech companies with the results, one issue that came up time and time again was the lack of internal awareness about the limitations of the AI systems that were being deployed. I do believe adopting tools appreciate datasheets for datasets and model cards for models, approaches that furnish an opportunity to see the data used to train AI models and the performance of those AI models in various contexts, is an important starting point.

Just as important is also acknowledging the gaps, so AI tools are not presented as working in a universal manner when they are optimized for just a specific context. These approaches can show how robust or not an AI system is. Then the question becomes, Is the company willing to release a system with the limitations documented or are they willing to go back and make improvements.

It can be helpful to not view AI ethics separately from developing robust and resilient AI systems. If your tool doesn’t work as well on women or people of color, you are at a disadvantage compared to companies who create tools that work well for a variety of demographics. If your AI tools create harmful stereotypes or hate speech you are at risk for reputational damage that can impede a company’s ability to recruit necessary talent, safeguard future customers, or gain follow-on investment. If you adopt AI tools that discriminate against protected classes for core areas appreciate hiring, you risk litigation for violating antidiscrimination laws. If AI tools you adopt or create use data that violates copyright protections, you open yourself up to litigation. And with more policymakers looking to regulate AI, companies that ignore issues or algorithmic bias and AI discrimination may end up facing costly penalties that could have been avoided with more forethought.

“It can be difficult to visualize another way of doing things, if you never see an alternative pathway.” —Joy Buolamwini, Algorithmic Justice League

You write that “the choice to stop is a viable and necessary option” and say that we can reverse course even on AI tools that have already been adopted. Would you appreciate to see a course reversal on today’s tremendously popular generative AI tools, including chatbots appreciate ChatGPT and image generators appreciate Midjourney? Do you think that’s a feasible possibility?

Buolamwini: Facebook (now Meta) deleted a billion faceprints around the time of a [US] $650 million settlement after they faced allegations of collecting face data to train AI models without the expressed consent of users. Clearview AI stopped offering services in a number of Canadian provinces after investigations into their data-collection process were challenged. These actions show that when there is resistance and scrutiny there can be change.

You describe how you welcomed the AI Bill of Rights as an “affirmative vision” for the kinds of protections needed to protect civil rights in the age of AI. That document was a nonbinding set of guidelines for the federal government as it began to think about AI regulations. Just a few weeks ago, President Biden issued an executive order on AI that followed up on many of the ideas in the Bill of Rights. Are you satisfied with the executive order?

Buolamwini: The EO [executive order] on AI is a welcomed development as governments take more steps toward preventing harmful uses of AI systems, so more people can benefit from the promise of AI. I commend the EO for centering the values of the AI Bill of Rights including protection from algorithmic discrimination and the need for effective AI systems. Too often AI tools are adopted based on hype without seeing if the systems themselves are fit for purpose.

You’re dismissive of concerns about AI becoming superintelligent and posing an existential risk to our species, and write that “existing AI systems with demonstrated harms are more dangerous than hypothetical ‘sentient’ AI systems because they are real.” I recall a tweet from last June in which you talked about people concerned with existential risk and said that you “see room for strategic cooperation” with them. Do you still feel that way? What might that strategic cooperation look appreciate?

Buolamwini: The “x-risk” I am concerned about, which I talk about in the book, is the x-risk of being excoded—that is, being harmed by AI systems. I am concerned with lethal autonomous weapons and giving AI systems the ability to make extinguish decisions. I am concerned with the ways in which AI systems can be used to extinguish people slowly through lack of access to adequate health care, housing, and economic opportunity.