Understanding where bias in AI systems originates.

Training data bias:

• AI systems are trained on data reflecting the real world, including its biases
• If training data overrepresents men, the model will overrepresent male patterns
• If research literature is geographically biased (more studies from wealthy countries), AI trained on that literature is geographically biased
• If funding historically favored certain research directions, AI follows that direction
• Example: Language models trained on internet text reflect biases in that text (gender stereotypes, racial biases, economic assumptions)

Data imbalance:

• Training data may have unequal representation of different groups
• An imaging model trained on 80% light-skinned patients and 20% dark-skinned patients performs worse on dark-skinned patients
• A literature AI trained on more recent papers than historical papers may overweight recent trends
• Solutions require explicit balancing, which is often not done

Annotation bias:

• Humans label training data, and humans have biases
• How ambiguous cases are labeled depends on who labels them
• Medical imaging labeled by radiologists may reflect radiologist biases
• Literature importance assessed by domain experts may reflect expert consensus or expert biases

Algorithmic bias:

• Beyond training data, algorithms themselves can introduce bias
• Optimization for accuracy on the overall dataset may produce worse performance on minority subgroups
• Some machine learning algorithms are more prone to bias than others
• Historical data used to train models perpetuates historical inequities

Specific ways AI bias affects researchers using these tools.

Literature bias amplification:

• Publication bias (positive findings published more than negative) is already in the literature
• Literature AI trained on published work amplifies this bias
• AI recommends papers similar to highly-cited papers, amplifying citation bias
• Result: Literature review based on AI recommendations may overestimate effect sizes, underrepresent negative findings, miss studies from underrepresented communities

Population representation bias:

• Most biomedical research enrolls primarily from wealthy countries
• AI trained on this literature is biased toward findings from those populations
• When researchers ask "What does the literature say about X condition?" the AI response is skewed toward literature on privileged populations
• Result: Research designed based on AI summary may not be applicable to other populations

Outcome measurement bias:

• Some outcomes are studied more than others
• AI pattern-matches on what has been studied; unstudied outcomes are invisible
• Example: Depression in women studied extensively, depression in men less studied, so AI emphasis may not reflect true research needs
• Result: Research questions chosen with AI input may focus on studied outcomes while ignoring important unstudied ones

Search algorithm bias:

• AI literature search tools rank results by relevance, but relevance is defined algorithmically
• Papers similar to what you initially search for rank higher (filter bubble effect)
• Less-cited papers from underrepresented communities rank lower
• Result: Literature search misses important work from underrepresented communities

Bias in data analysis:

• AI-generated analysis code may not appropriately handle data from diverse populations
• Algorithms may perform differently across demographic groups
• Visualizations can be misleading in ways that amplify biases
• Result: Analysis conducted with AI assistance may miss important subgroup differences

Recognizing where bias is most likely to harm research.

Key vulnerable populations:

• Racial and ethnic minorities: Often underrepresented in biomedical research; AI amplifies this underrepresentation
• Women: Some health research historically underrepresented women; AI biases reflect this
• Low-income populations: Research more often conducted in wealthy populations; AI biases reflect this
• Global South: Research concentrated in wealthy countries; AI trained on that literature is geographically biased
• Rare populations: Transgender people, people with rare diseases, immigrants, and others underrepresented in data that trains AI

Why this matters: If your research includes or affects these populations, AI biases can systematically produce worse results or recommendations for them.

Practical strategies for protecting against AI bias.

Recognition strategies:

• Ask: "Which populations is this AI trained on?" (Usually some answer is available)
• Ask: "Could this tool have performed worse on minority populations in development?" (Assume yes unless proven otherwise)
• Check: Look at actual recommendations and ask: "Are these diverse? Do they reflect all populations I care about?"
• Verify: When AI makes recommendations, verify them against literature you know (which might be different from what AI recommends)
• Listen: What populations are underrepresented in AI recommendations?

Mitigation strategies:

• Diversify sources: Don't rely solely on AI-recommended papers; actively search for work from underrepresented communities
• Explicit comparison: Ask AI to explicitly compare findings across populations: "What do these studies find in white vs. non-white populations?" or "How do findings differ between men and women?"
• Representation check: In your literature review, count: what proportion of studies included specific populations? Are some overrepresented, others absent?
• Subgroup analysis: In data analysis, explicitly test whether findings generalize across populations: "Does this effect hold for all groups or only some?"
• Consult experts: Talk to people from underrepresented communities in your research area about whether results apply to them
• Pre-commit: Before analyzing data, specify whether you expect effects to generalize across groups or expect group differences; this prevents p-hacking for subgroup effects

How to ensure others can assess bias in your AI use.

Key points:

• Document what AI tools you used and what they were trained on
• Disclose known limitations of AI tools (e.g., "This literature search tool is trained on English-language publications; non-English literature not represented")
• Report results for all populations, not just the overall sample
• If analysis differs by population, report that
• Acknowledge limitations regarding representation and generalizability
• Make it possible for readers to understand where bias might exist

—

Practical Research Use Cases

Scenario: You're using AI to synthesize literature on stroke outcomes to understand what predicts good recovery.

Bias concern: Stroke research is heavily skewed toward white males (who have higher research participation). AI trained on this literature will overrepresent findings from this group.

Mitigation strategy:

1. Ask the AI: "What populations did these papers study? Create a table showing [Author, Year, % male, % white, % over 65]"
2. Review the table: Are all studies >70% male? Mostly white participants?
3. Actively search for papers including diverse populations: "Papers on stroke recovery in women" and "Papers on stroke recovery in Black/African American patients"
4. Ask the AI: "For women vs. men, what do these papers say about recovery predictors? Are they different?"
5. Document in your review: "Our review identified that research is predominantly conducted in male participants; findings may not generalize to women" (if true)

Scenario: You're using AI to generate code for analyzing treatment effects in your dataset of 1,000 patients (60% white, 40% Black/African American).

Bias concern: AI-generated code will likely focus on overall effects without examining whether treatment works differently for different racial groups. Publication bias suggests treatments work better in white populations (due to research emphasis), so AI might not flag potential racial differences.

Mitigation strategy:

1. Ask the AI: "Generate code to test whether treatment effects differ by race. I want to see: overall effect, effect for white participants, effect for Black/African American participants, and p-value for interaction."
2. Review the code: Does it test treatment x race interaction?
3. Run the code and examine results: Are effects the same across groups?
4. If effects differ: investigate why. Is it truly a racial difference or is something else different between groups?
5. Document findings: "We found treatment effects were [describe], with effects [same/different] across racial groups"

Scenario: You're using Semantic Scholar to find papers on depression in men. The AI returns papers, but they mostly discuss men's depression in the context of suicide.

Bias concern: The training data reflects publication patterns where men's depression is studied more in suicide contexts than other contexts. AI amplifies this pattern.

Mitigation strategy:

1. Recognize: "My search results are heavily weighted toward suicide. Is that because depression in men is only studied via suicide, or is that a search bias?"
2. Do supplementary searches: "Depression in men and work performance," "Depression in men and relationships," "Depression in men and physical health"
3. Check: Do these searches return different papers or does the algorithm keep returning the same ones?
4. Conclude: Understand what's true about the literature vs. what's a search bias
5. In your review, acknowledge: "Literature on depression in men emphasizes suicide outcomes; less attention to other functional domains"

Scenario: AI-synthesized literature suggests a cognitive intervention is highly effective (effect size d = 0.8). You're planning to test this in a different population.

Bias concern: The literature showing large effects may come from privileged populations with good resources. Your population may have different baseline abilities, resources, or other factors affecting effect size.

Mitigation strategy:

1. Ask the AI: "What populations did the papers showing large effects study? What was their demographic profile?"
2. Ask: "Did any papers test this intervention in [your specific population]? What were the effect sizes?"
3. Compare: How similar is the literature population to your planned population?
4. Adjust expectations: If literature is from very different populations, expect smaller effects in your population
5. Explicitly test: Build subgroup analysis into your design to test whether the intervention works similarly in your population as in the literature

—

Hands-On Exercise

Objective: Identify bias in AI literature tool recommendations and develop mitigation strategies.

Steps (Time: 90-120 minutes):

1. Conduct an AI literature search (30 minutes):

• Choose a research topic you care about
• Use Semantic Scholar or Elicit to search
• Review the top 20-30 results
• Take notes on the papers returned

1. Analyze demographic representation (30 minutes):

• For at least 10 papers, extract: authors' affiliations (note: wealthy vs. lower-resource countries), study populations (note demographic representation if described), sample sizes
• Create a table: Author | Country of Origin | Population Studied | Notes on Representation
• What patterns do you see?
• What populations are overrepresented? Underrepresented? Absent?

1. Compare to alternate searches (30 minutes):

• Do additional targeted searches for underrepresented populations
• Example: If search 1 is "depression treatment," search 2 could be "depression treatment in low-income populations" or "depression treatment in Southeast Asia"
• Compare results: Does the algorithm return different papers?
• Are the underrepresented populations easier to find with targeted searches?

1. Identify specific biases (15 minutes):

• Based on your analysis, what biases exist in these search results?
• Which populations are over- or under-emphasized?
• How might this bias affect a researcher using only these results?

1. Develop mitigation strategies (15 minutes):

• For your research topic, design a literature search process that mitigates biases you identified
• Example: "Use AI search, then supplement with targeted searches for [underrepresented populations]"
• Document: How will you ensure representation? How will you report it?

Time required: 90-120 minutes

—

Common Mistakes and Misconceptions

Algorithms are computational but not objective. They reflect biases in training data and decisions about what to optimize for. Computational does not mean unbiased.

Most AI tools are tested for overall accuracy but not for bias across subgroups. A tool can be 95% accurate overall while being 85% accurate for minority groups. You must check subgroup performance.

Your objectivity doesn't counter AI bias. If the tool amplifies bias, using it amplifies bias in your research regardless of your intentions.

It's the opposite. Acknowledging and mitigating bias demonstrates scientific rigor. Hiding bias or ignoring it is bad science.

Bias undermines validity. If you study only privileged populations and generalize to everyone, your results are wrong. Bias affects both equity and validity.

—

Key Takeaways

• AI systems inherit biases from training data: Publication bias, geographic bias, demographic imbalance in studies, and historical inequities all appear in AI recommendations
• Bias manifests in research as amplification: Literature recommendations overweight published/cited work; analysis recommendations miss subgroup differences; search results bury underrepresented populations
• Vulnerable populations are most at risk: Racial minorities, women (in some fields), low-income populations, Global South, and rare populations are often underrepresented in data and thus underrepresented in AI
• Recognition requires active questioning: Ask which populations AI recommendations are trained on, verify against diverse sources, check whether results generalize across groups
• Mitigation requires supplemental searching and subgroup analysis: Actively search for underrepresented populations, explicitly test for interactions, report results across groups
• Transparency enables others to assess bias: Document which AI tools you used, what populations they're trained on, how you checked for bias, and whether results generalize

—

Reflection Questions

1. Your population of interest: Who does your research serve or affect? What populations are you studying? Are these populations over- or under-represented in research literature more broadly?

1. Bias risks in your field: What biases exist in your research field? Geographically? Demographically? In terms of outcomes studied? In outcome measurement methods?

1. AI tool assessment: For an AI tool you plan to use, what could you learn about biases it might have? How would you verify that it works well for your populations of interest?

1. Mitigation in your work: For your planned research, how would you use AI while protecting against bias? What supplemental checking would you do?