Bias in AI Training: 4 Real Sources and Why It Matters Now

Understanding Bias in AI Training

Bias in AI training is the tendency of a model to favor certain outcomes, groups, or perspectives because of patterns in the data, labels, or feedback it learned from — not because anyone told it to. OpenAI's GPT models, Google's Gemini, and Anthropic's Claude are each trained on hundreds of billions of words pulled from the public web, books, and licensed datasets, and the web itself skews toward English-language, Western, and male-dominated sources.

That's why two AI tools can give you noticeably different answers to the same question. One learned its defaults from a slightly different slice of the internet than the other. If an answer sounds more confident, more hedged, or more Western-centric than you expected, bias in AI training is usually part of the reason.

"Bias in AI training isn't intentional — it's inherited. A model trained on unbalanced data learns unbalanced defaults, then defends them with total confidence."

The Stanford Institute for Human-Centered AI has pointed to training-data transparency as one of the clearest ways to hold AI systems accountable, and I agree with that framing — knowing how a model learned is the first real step toward trusting what it says. That's the through-line for the rest of this post: not just where bias in AI training comes from, but what's now legally required to address it in the US.

The 4 Real Sources of Bias in AI Training

Bias in AI training comes from four places: the data itself, the humans who label and filter it, the feedback loops that reinforce popular answers, and the design choices baked into what the model is optimized to predict. Each one compounds the others.

1. The Training Data Itself

Every model is only as balanced as the text, images, and interactions it was fed. If most of that material comes from a specific region, language, or income bracket, the model treats those patterns as the default — not because they're objectively "normal," but because they showed up most often.

2. Human Labeling and Filtering

Behind every model, annotation teams decide what counts as "safe," "positive," or "relevant." If those teams share a narrow cultural background or work from unclear guidelines, their judgment calls — which slang to prioritize, which content to filter — quietly become the model's judgment calls too.

3. Feedback Loops From User Behavior

Once a model launches, it keeps learning from what users click, upvote, or engage with. That reinforces whatever answer already won the most attention — which is a popularity signal, not an accuracy one, and it can harden existing stereotypes over time.

4. Model Design and Objective Functions

Even the metric a model is trained to optimize can introduce bias. A hiring model trained to predict "who gets promoted" instead of "who is qualified" will faithfully reproduce whatever promotion patterns already existed — including unfair ones.

Here's how the four sources of bias in AI training break down side by side:

A model doesn't know it's biased. It just knows which pattern won the most during training — and repeats it with total confidence.

Together, these four sources are what most people mean when they talk about bias in AI training. If you want the mechanics behind how these models learn patterns in the first place, I break that down separately in machine learning vs. generative AI.

Real-World Consequences: Hiring, Healthcare & Facial Recognition

Bias in AI training stops being theoretical the moment a model makes a call that affects someone's job, health, or freedom. These three documented cases show the pattern at full scale.

Hiring: When a Résumé Screener Learns the Wrong Lesson

Amazon famously scrapped an internal AI recruiting tool in 2018 after finding it downgraded résumés containing the word "women's" — the system had trained on a decade of past hiring data that skewed heavily male, and it faithfully learned to prefer that pattern — a textbook case of bias in AI training turning into a legal liability.

Facial Recognition: The Gender Shades Study

MIT Media Lab researcher Joy Buolamwini's Gender Shades project tested commercial facial-analysis systems on 1,270 unique faces and found the error rate for darker-skinned women topped 34.7% — against an error rate near 0% for lighter-skinned men — on benchmark datasets that were overwhelmingly composed of lighter-skinned subjects to begin with. You can read the full methodology at the Gender Shades project page.

Healthcare: When Historical Spending Stands In for Need

A widely cited 2019 study found a healthcare-risk algorithm used across US hospitals systematically underestimated how sick Black patients were, because it used past healthcare spending as a proxy for medical need — and spending itself already reflected unequal access to care. The algorithm wasn't reading race directly; it was reading a biased stand-in for it — a quieter but no less real case of bias in AI training.

These aren't hypothetical edge cases. Amazon, MIT, and peer-reviewed healthcare research all documented the same pattern: bias in AI training turns into unbalanced outcomes at scale, for real people.

The Law Is Catching Up: US Rules to Know in 2026

Bias in AI training used to be a purely technical conversation. In the US, it's now a compliance one too — and most employers aren't tracking it closely enough yet.

NYC Local Law 144: The Only Mandatory Audit (So Far)

As of September 2026, New York City's Local Law 144 remains the only US law that requires an independent, third-party bias audit before an employer can use an automated hiring tool on NYC-based candidates. It's been in effect since July 2023, and enforcement has tightened through 2026 after a December 2025 city audit found the original process too weak to catch violations. You can read the requirements directly on NYC's official AEDT page.

Illinois, Colorado, and What's Really Required Elsewhere

Rules on bias in AI training vary sharply once you leave New York. Illinois's HB 3773, effective January 1, 2026, requires employers to notify candidates when AI plays a role in an employment decision — it doesn't require an audit. Colorado's original AI Act never took effect; it was repealed and replaced by SB 26-189, signed May 14, 2026, which applies from January 1, 2027 with narrower rules: pre-use notice and a right to an explanation after an adverse decision.

Federally, the EEOC has confirmed that Title VII and the ADA apply to AI-assisted hiring decisions the same way they apply to human ones — there's no "the algorithm did it" exemption. You can see the agency's own guidance on AI and the ADA.

Only one US city currently requires proof your hiring AI isn't biased. That's likely to change — but as of today, most companies still aren't required to check.

I'm not a lawyer — treat this section as a starting point, not legal advice, and confirm your state's current requirements before making a hiring decision.

How to Detect and Reduce Bias in AI Training Yourself

You don't need a machine learning background to catch bias in AI training before it affects a decision that matters to you. Developers have free tools for the technical side, and there's a fast manual check for everyone else.

Free Tools Researchers and Developers Rely On

These three open-source toolkits are what fairness researchers most often reach for to measure bias in AI training before a model ships:

A 4-Point Checklist for Spotting Bias in AI Training Yourself

• Ask the same question twice with one detail swapped — a name, a gender, a location — and compare the two answers.
• Ask the model to explain its reasoning. Vague or circular justifications are a flag.
• Cross-check anything about a person or group against a second AI tool or a primary source.
• Watch for unearned confidence — a skewed answer delivered with total certainty is a classic bias-plus-hallucination combo. I go deeper on that pattern in why AI sounds confident when it's wrong.

The fastest way I know to catch bias in AI training in the wild: ask the same question twice with one detail changed, and see if the answer changes with it.

Who's Responsible for Fair AI?

Bias in AI training isn't only a technical problem — it's an accountability one. Every time a model makes a decision, someone is responsible for how it was trained and who it affects.

OpenAI, Anthropic, and Google DeepMind have each published research and frameworks aimed at making model behavior more transparent — Anthropic's Constitutional AI approach, for instance, trains models against an explicit set of written principles rather than opaque trial and error. None of this makes a model bias-free, but it does make the process more inspectable than it was even two years ago.

If bias in AI training genuinely interests you past the surface level, The Alignment Problem by Brian Christian is the single book I'd point you to — the hiring, bail, and healthcare examples above all get a much deeper treatment in it. I've put the full recommendation below.