GEPI’s How AI Fails Women Report Examines How Algorithms Amplify Gender Bias

A hiring score, a credit assessment, or a clinical recommendation can look impartial while carrying forward the inequalities in the data behind it. That is the central warning of the Gender Equity Policy Institute’s new report, How AI Fails Women: When Bias Becomes Amplified, which brings together research on how automated systems can reproduce—and sometimes intensify—gender discrimination.

Written by machine learning scientist and GEPI senior fellow Myriam Shiran, PhD, the report reviews academic and gray literature across economic opportunity, healthcare, criminal justice, generated content, consumer AI design, and AI-enabled image-based sexual abuse. It is a synthesis of existing evidence, rather than a new experiment measuring every AI system against a single benchmark.

Its most consequential argument is about what happens after a biased result leaves a model. Recommendations influence decisions, generated content shapes expectations, and some outputs become material for future training. An unfair pattern can therefore become harder to identify as it moves through successive systems.

Removing a gender field is an intuitive response to discrimination. GEPI’s review explains why that step alone is insufficient: names, education, employment histories, and other details can still carry gender-related signals. Historical outcomes may also encode unequal treatment before a model ever encounters them.

One hiring audit cited in the report, by Wilson and Caliskan in 2024, involved more than three million simulated résumé comparisons across nine occupations. The tested screening models significantly favored male-associated names in 51.9% of comparisons and female-associated names in 11.1%. The remaining 37% did not show a significant preference. These are results from controlled model tests, not percentages of real applicants denied jobs.

The evidence also varies by test design. The report describes a separate 2025 audit in which several language models showed no gender difference when researchers changed names alone, while changes to education details affected scores. A narrow test can miss information that matters elsewhere in the same workflow.

In lending, the review points to research on roughly 80,000 unsecured personal loans from a major U.S. fintech lender. Women’s loans generated higher returns for the lender, a finding the study linked to a model overestimating their default risk and pricing loans accordingly. Here, the issue is the relationship between predicted risk, actual repayment, and the price a borrower pays—not simply whether gender appears in the application.

Healthcare offers a particularly clear example of why population-wide performance can conceal important differences. A model may perform well overall while making more errors for people underrepresented in its training or validation data.

GEPI cites a 2024 study by Zack and colleagues that presented GPT-4 with otherwise identical chest-pain cases described as involving a man or a woman. Across 200 runs for each description, the model rated cardiac stress testing as highly important in 71% of the male cases and 58% of the female cases—a 13-percentage-point difference.

That finding concerns model recommendations in a controlled evaluation, not observed treatment decisions or patient outcomes. The distinction matters, particularly because the report includes exploratory studies of different models, diseases, and clinical tasks conducted at different times.

The review also makes an essential qualification: clinically meaningful sex differences should not simply be erased. The challenge is to determine whether a difference in a model’s advice reflects relevant biology or an unsupported assumption. That requires representative evaluation data and careful examination of errors, rather than a demand that every recommendation be identical.

The report distinguishes harm to an individual’s access to resources—such as a loan or medical care—from harm in how groups are portrayed. Generated recommendation letters, news articles, and occupational images can reinforce stereotypes even when no immediate allocation decision follows.

Research summarized by GEPI found language models describing women more through interpersonal qualities and men more through leadership or ability. Studies of image generators found exaggerated gender segregation across occupations. Those results apply to the models and versions examined; they should not be read as a fresh audit of every currently available system.

Consumer assistants and companions raise a different question. Their voices, personalities, and responses to harassment are deliberate product choices. Evaluating only the statistical balance of their answers can overlook the assumptions embedded in what a product is designed to be.

The review also examines nonconsensual synthetic sexual imagery, where AI can make abuse easier to produce and distribute. This is more than an inaccurate depiction: the use of a real pers