Since large language models (LLMs) like ChatGPT became widely used, experts have warned that these systems could perpetuate or even worsen existing societal biases. Now, a new study has confirmed: AI models show a strong preference for names associated with white people in hiring simulations, raising concerns about their role in perpetuating discrimination.

Twenty years ago, economists made a landmark study in which they sent thousands of fictitious applications to companies in Boston and Chicago. The apps were identical except for the names: some were traditionally black-sounding, while others were white-sounding. The results were staggering: candidates with white names received 50% more callbacks.

Although the gap has narrowed over time, the bias remains. A recent study this year involved sending 83,000 fake applications and found a 10% difference in callback rates. Despite promises that AI would reduce human bias, there are signs that these models may not live up to those expectations.

AI seems to hate black candidates

Researchers at the University of Washington tested three cutting-edge LLMs using more than 500 job descriptions and 500 resumes. They focused on nine professions, including CEOs, teachers, accountants and engineers.

The goal was to assess whether AI systems favored resumes with race (black vs. white) and gender (male vs. female) cues. They also analyzed whether these biases worsened for intersectional identities, such as those of black women.

The results were striking. Out of three million CV comparisons, CVs with blank names were favored by AI models 85% of the time. On the other hand, CVs containing names associated with black people were only selected in 8.6% of cases. Even though gender bias was less pronounced, male names still had a slight advantage, being preferred in just over 50% of cases.

Black men, in particular, were at a significant disadvantage. In some scenarios, they were completely overlooked in favor of white male candidates. Black female names fare slightly better, but still face substantial disadvantages compared to their white counterparts.

Why these biases appear

In a way, LLMs still function as a “black box”: it is not clear why they make some of the decisions they do. However, researchers believe they can explain at least part of this effect.

To begin with, this is the training data. These models were trained on huge amounts of text, including Internet text. This text may carry the same biases that we do as a society, and perhaps even more. The models “learn” social stereotypes, so to speak.

The second reason would be a frequency effect. If members of the black community are traditionally underrepresented in certain fields, the LLM could naturally perpetuate this trend, which would impact selection.

Other factors may also be at play, but it is difficult to disentangle them from racial and gender influences.

How to eliminate bias

At first glance, it seems like the answer is simple: just remove the name from resumes. This idea has been around for a while, but it may not be very effective. Name is just one of the racial identifiers that AIs can detect. Educational institutions, locations, and even particular word choices can signal gender and racial identities. Removing the name may solve part of the problem, but only part of it. Additionally, removing names does not address the root cause: biases embedded in the language patterns themselves.

A Salesforce spokesperson told Geekwire that they don’t blindly use these AI models. “All models proposed for production use undergo rigorous testing for toxicity and bias before being released, and our AI offerings include guardrails and controls to protect customer data and prevent harmful outputs.” However, this is difficult to actually verify.

A more in-depth solution would be to modify the training data, adjusting algorithms to ignore specific identity markers or debiasing embeddings. However, as the study highlights, these solutions often reduce people’s identities to “same or different,” without recognizing the unique challenges that marginalized groups face.

Perhaps the most difficult, but also potentially most effective, solution is to change the way we conceptualize professionalism. For example, if certain words or phrases commonly associated with women (like “neat” or “collaborative”) are less valued by AI systems, we may need to re-evaluate what we consider to be a “strong” CV. Language depends on context. Words associated with empathy or teamwork should be just as valued as those associated with leadership and assertiveness.

You should care

AI is poised to transform recruiting. Tools such as ChatGPT have made it easier to generate tailored applications, while companies increasingly use AI to filter CVs. And you’re probably already starting to understand what a problem this can be.

If companies adopt these systems wholeheartedly, they only perpetuate existing biases. And they often don’t hire the best people for the job. This is both a social and productivity problem. By replicating and even amplifying biases, AI-powered resume screening tools could make it more difficult for certain groups to advance their careers. Ultimately, this can impact the economic and social mobility of entire communities.

Additionally, these findings highlight the importance of transparent audits and regulatory oversight of AI recruitment tools. It’s one thing to automate repetitive tasks, but when it comes to shaping people’s careers and livelihoods, fairness must be the priority.