Algorithmic bias

Algorithmic bias is systematic, unjust differences in an algorithm’s outputs or effects across groups. It can arise from data, design choices, deployment, or the institutions using the system.

Training data: Examples used to fit a machine-learning model’s parameters. Historical gaps or skewed examples can make a model perform unevenly across groups.
COMPAS: A proprietary risk-assessment system used in United States criminal courts to estimate recidivism risk. Its risk scores became a prominent case in disputes over racial disparities in algorithmic predictions.
Fairness: The principle that people should receive just treatment and that benefits and burdens should be distributed appropriately. Algorithmic bias is judged against a normative account of which group differences are unjust.
Discrimination: Unjust differential treatment of people based on group membership or a protected characteristic. Algorithmic bias can automate or scale discriminatory treatment through consequential decisions.
Proxy variable: A measurable feature used in place of a harder-to-measure attribute. Proxies such as ZIP code can reproduce group disparities without explicitly encoding group identity.
Predictive policing: The use of data analysis to forecast where crime may occur or who may be involved. Feedback from concentrated policing can make predictions reproduce earlier enforcement patterns.
Disparate impact: A policy or practice that disproportionately disadvantages a protected group, even without intentional discrimination. It focuses attention on unequal effects rather than requiring proof of biased intent.
Automation bias: The tendency to over-rely on recommendations produced by automated systems. Human decision-makers may accept biased scores even when other evidence contradicts them.
Target variable: The outcome a model is designed to predict or estimate. A biased target can encode institutional judgments as if they were neutral facts.
Facial recognition system: A system that identifies or verifies people by analyzing facial images. Uneven error rates can expose demographic groups to different risks of misidentification.

Linked from 42 pages