Is AI biased?

The ethical quagmire of modern machine learning

AI bias lurks beneath the surface of our increasingly automated world. As algorithms continue to make decisions that impact human lives, the potential for discrimination grows and grows. It’s important to understand why this is happening and how companies (and individuals) can continue efforts to mitigate.

Otherwise we risk a world where AI’s negative effects on society hit on a deeper level that is less obvious — but more harmful.

Why it matters

A 2022 DataRobot survey found 56% of technology leaders fear AI bias will erode customer trust. This apprehension stems from high-profile cases of algorithmic discrimination.

In 2015, Google's image recognition system labeled photos of Black individuals as "gorillas," exposing racial bias in its training data In 2018, Amazon scrapped an AI hiring system that penalized resumes containing the word "women's" or mentioning all-women's colleges. The system, trained on historical data from a male-dominated tech industry, learned to discriminate against female applicants. Elsewhere, a widely-used system for predicting patient risk has systematically underestimated the health needs of Black patients compared to equally sick white patients. This bias arose from using healthcare costs as a proxy for health needs, failing to account for disparities in healthcare access.

These incidents fuel public skepticism about AI fairness. A 2019 Pew Research survey found 58% of Americans believe computer programs will always reflect human biases. This perception threatens to undermine trust in AI systems across sectors.

Experts warn algorithmic bias could exacerbate existing inequalities. AI researcher Kate Crawford cautions, "The reproduction of harmful ideas is particularly dangerous now that AI has moved from being an experimental discipline used only in laboratories to being tested at scale on millions of people". As algorithms make high-stakes decisions in areas like healthcare, criminal justice, and employment, the societal cost of biased systems grows. Addressing algorithmic fairness has become essential for responsible AI development and deployment.

What research says

Academic research provides valuable insights into the nature, causes, and impacts of AI bias. Recent studies have helped demystify how bias can muddy AI systems and cause irreparable damage.

Causes of bias

Research identifies several key sources of algorithmic bias:

1. Biased training data. AI systems learn from historical data, inheriting and amplifying existing societal biases. One study found commercial facial recognition systems had much high errors rates for darker-skinned females than lighter-skinned males due to unrepresentative training data.

2. Algorithmic design choices. The selection of features, model architectures, and optimization criteria can introduce bias. Researchers at Columbia University developed techniques to detect vulnerabilities in neural networks that could lead to biased outputs.

3. Human bias in development. The personal biases of AI developers can seep into system design. A lack of diversity in tech workforces exacerbates this issue, according to one study.

Types of bias

Studies have uncovered various forms of algorithmic discrimination:

1. Gender bias. Research shows AI systems often perpetuate gender stereotypes. A study found word embeddings used in natural language processing associated "programmer" with male terms and "homemaker" with female terms.

2. Racial bias. Multiple studies have found racial disparities in AI performance. The COMPAS algorithm used in US courts was found to falsely label Black defendants as high-risk at nearly twice the rate of white defendants.

3. Age bias. A study of healthcare algorithms found they systematically underestimated the health needs of older patients.

4. Socioeconomic bias. Research shows AI can amplify existing economic inequalities. A study found LinkedIn's job recommendation algorithm perpetuated gender disparities in high-paying professions.

Impacts of bias

Research reveals far-reaching consequences of biased AI:

1. Healthcare disparities. A study in Science found a widely-used algorithm underestimated the health needs of Black patients, potentially exacerbating racial health inequities.

2. Employment discrimination. Research shows AI hiring tools can perpetuate gender and racial biases, limiting opportunities for marginalized groups.

3. Criminal justice inequities. Studies have found racial bias in predictive policing algorithms and recidivism risk assessment tools used in courts.

4. Amplification of societal biases. Research indicates AI systems can reinforce and magnify existing prejudices, creating feedback loops of discrimination.

Mitigation strategies

Researchers propose various approaches to combat AI bias:

1. Diverse, representative datasets. Studies show using more inclusive training data can significantly reduce bias.

2. Algorithmic fairness techniques. Researchers have developed methods like adversarial debiasing and counterfactual fairness to mathematically enforce equitable outcomes.

3. Interpretability and transparency. Making AI systems more explainable allows for better auditing and bias detection.

4. Interdisciplinary collaboration. Research emphasizes the need for diverse teams including ethicists, social scientists, and domain experts in AI development.

5. Regulatory frameworks. Studies propose legal and policy interventions to ensure responsible AI development and deployment.

How the big guns are mitigating

Major AI companies (claim to) recognize the ethical imperative of addressing algorithmic bias. Various strategies exist within these companies to promote fairness in their systems. Whether they’re functional long term remains to be seen, but there are efforts being made.

Data diversification

Companies are working to create more representative datasets:

• OpenAI emphasizes the need for diverse training data, looking for partners to help create these training sets.

• Microsoft has developed techniques to balance datasets. Reportedly: "Microsoft revised their dataset for training the Face API, resulting in a 20-fold reduction in the recognition error ratio between men and women with darker skin tones and a 9-fold reduction for women by balancing factors such as skin color, age, and gender".

Algorithmic debiasing

Companies employ various technical approaches to mitigate bias:

• Anthropic uses "Constitutional AI" to encode ethical principles into AI systems. They explain: "Our goal is essentially to develop: 1) better techniques for making AI systems safer, 2) better ways of identifying how safe or unsafe AI systems are".

• IBM developed the AI Fairness 360 toolkit, which includes "a set of fairness metrics for datasets and models, including its explanations and algorithms to reduce bias”.

Transparency and explainability

Companies are working to make AI systems more interpretable:

• Google introduced the What-If tool for detecting bias. It "assists designers in identifying the causes of misclassification, determining decision boundaries, and detecting algorithmic fairness through interactive visual interfaces".

• OpenAI emphasizes the need for algorithmic transparency. They state: "We are … exploring partnerships with external organizations to conduct third-party audits of our safety and policy efforts".

Ethical governance

Companies are implementing internal ethics oversight:

• Microsoft formed an AI and Ethical Standards Committee to enforce ethical principles in AI development (and then let them go, whoops).

• Anthropic has begun to experiment with a so-called “Long Term Benefit Trust, what Vox calls “ … a group of people without financial interest in the company who will ultimately have majority control over it”.

• OpenAI states: "We are in the early stages of piloting efforts to solicit public input on topics like system behavior, disclosure mechanisms (such as watermarking), and our deployment policies more broadly".

Regulatory engagement

When it suits them, companies tend to show some favor towards regulation:

• OpenAI typically supports (managed) regulatory oversight: "We appreciate the ChatGPT user community as well as the wider public's vigilance in holding us accountable, and are excited to share more about our work in the three areas above in the coming months".

• Anthropic similarly advocates for some form of regulation (that it deems acceptable): "A robust, third-party testing regime seems like a good way to complement sector-specific regulation as well as develop the muscle for policy approaches that are more general as well”.

While these efforts represent progress, critics argue more robust measures are needed to ensure AI systems truly benefit all of society. The challenge of creating fair, unbiased AI remains an ongoing ethical imperative for the tech industry.

Will we solve it?