When AI Sees Gender: The Hidden Bias in Vision-Language Models

Vision-language models (VLMs) are supposed to be the unbiased translators between the visual world and language. But when confronted with ambiguous images, a worker in full gear or a figure from behind, do they truly avoid gender bias? Not quite.

Unmasking the Gender Defaults

In practice, these models often lean heavily towards male defaults, even in traditionally female-dominated professions. It seems the models' outputs don't always reflect their internal biases. So, what gives? This is where the Latent Association Leaning Score (LALS) comes into play. It's a new metric that projects what the models internally associate with gender onto their text embeddings. Across fifteen occupations and more than 800 gender-ambiguous images, the models show their true colors.

Decoding the Internal Conflict

Despite what they output, VLMs frequently encode female associations internally. This internal-external mismatch is systematic. As layers process data, male signals gain strength, while female signals peak mid-network and then get crushed before the image is generated. Why's that happening?

There’s a filter at work. It seems that VLMs are wired to amplify male cues and suppress female ones, especially in final outputs. Even cultural markers like clothing color, which you’d think could help balance the gender scale, often modulate these biases further.

What This Means for Real-World Applications

With AI systems increasingly used in hiring and other decision-making processes, this bias isn't just a technical quirk, it’s a real-world problem. When AI sees a construction worker as male, it’s shaping our digital society based on outdated gender norms. The benchmark doesn't capture what matters most. Whose data trains these models? Whose labor? Whose benefit?

This isn’t just about accuracy. it's about representation. The paper buries the most important finding in the appendix: our models have a gender bias problem, and we need to do something about it. If you're building an AI system, ask yourself: who funded the study? Who benefits from these biases? Because if we don’t examine these questions, we’re just letting AI systems perpetuate the same stereotypes we’re trying to dismantle.