ChatGPT and other AI models show forms of “covert racism,” according to a new study published in the journal Nature.
Created by a group of researchers from Stanford University, the University of Chicago, and the Allen Institute for AI, the study found that, when African American English was used, AI models were less likely to give the users any jobs, particularly prestigious jobs.
When the researchers input a prompt in which someone is accused of an unspecified crime and asked whether or not they should be convicted, the system tended to convict the people associated with the AAE language model.
When the person was accused of first-degree murder, the AI systems sentenced people based on the usage of AAE or Standard American English; those associated with AAE were convicted of death sentences at higher rates.
Overall, the researchers found that, while all the AI models showed covert racism, the larger, advanced models, including those curated with human feedback, exhibited more covert biases. Those biases were on a lesser scale when researchers entered in prompts directly about Black Americans.
Nature research paper: AI generates covertly racist decisions about people based on their dialect https://t.co/e0dxOMfuKz
— nature (@Nature) August 29, 2024
“The implicit nature of this prejudice, that is, the fact it is about something that is not explicitly expressed in the text, makes it fundamentally different from the overt racial prejudice that has been the focus of previous research,” said the researchers in their discussion. “These two language models obscure the racism, overtly associating African Americans with exclusively positive attributes (such as ‘brilliant’), but our results show that they covertly associate African Americans with exclusively negative attributes (such as ‘lazy’).”
The study’s results come months after a similar study was published, in which researchers found AI had racially biased language models regarding users’ names.
Published by Standford Law School, the study found that the systems gave disadvantageous results when prompts were entered with names associated to Black women or Black men.
Chatbots such as ChatGPT and PaLM-2 were found to show racial bias when researchers entered in scenarios such as election results, sports ratings, deserved salaries and chess matches.
In their results, they reported that biases were shown toward Black people in all prompts except when it came to athletic rankings. For instance, when it came to salaries, a proposed salary for a candidate named Tamika was found to be approximately $79,000. When a name like Todd was input, the proposed salary increased to $82,000.
“The biases are consistent across 42 prompt templates and several models, indicating a systemic issue rather than isolated incidents,” said co-author Julian Nyarko. “In some areas, the disparities were quite significant.”
