In 2016, a team of journalists at the American investigative-journalism organisation ProPublica published an analysis of a computer system called COMPAS, used in American courts to help judges predict which defendants were likely to commit future crimes. The system was being used, by then, in thousands of sentencing, bail, and parole decisions across the United States. Judges weren’t bound by its predictions, but the predictions influenced how long people went to prison, whether they got bail, and whether they were granted parole.
What ProPublica found was that the system’s errors fell unevenly across racial groups. Among defendants who did not go on to reoffend, Black defendants were nearly twice as likely as white defendants to have been misclassified by COMPAS as high-risk. Among defendants who did reoffend, white defendants were more likely than Black defendants to have been misclassified as low-risk. The system, in other words, was producing more false-positive errors (flagging people as dangerous who weren’t) against Black defendants, and more false-negative errors (flagging people as safe who weren’t) in favour of white defendants.
The company that built COMPAS, Northpointe, pushed back. They argued that the system was calibrated fairly: a given risk score meant roughly the same probability of reoffending regardless of the defendant’s race. By that measure, the system was not biased. The subsequent academic debate established something important: you can’t, in general, satisfy both definitions of fairness at once. If the base rates of the underlying phenomenon differ between groups, any algorithm accurate enough to be useful will fail at least one version of fairness by construction.
The COMPAS controversy became one of the foundational cases in a rapidly growing field now often called algorithmic fairness, or more broadly, the ethics of automated decision-making. As machines make more and more consequential decisions about human lives — in hiring, lending, policing, healthcare, insurance, benefits administration, immigration — the questions raised by COMPAS have become central rather than exotic.
The broader pattern
The COMPAS case isn’t isolated. Similar patterns have been found across many domains.
Facial recognition systems, studied by the American computer scientist Joy Buolamwini and colleagues at MIT Media Lab, have consistently performed worse on women and on darker-skinned people than on men and lighter-skinned people. Buolamwini’s 2018 paper, Gender Shades, documented error rates for commercial facial-recognition systems that differed by 30 per cent or more between demographic groups. The systems had been trained on datasets that over-represented certain populations — typically white men — and had learned to recognise those populations more accurately. When deployed in contexts like law enforcement, security, and employment screening, the differential accuracy produced systematically worse outcomes for the under-represented groups.
Hiring algorithms, the kind used by large companies to screen job applications, have repeatedly been found to discriminate in ways that their designers didn’t intend. One widely-reported case involved a machine-learning system used by Amazon to screen applications for technical roles. The system, trained on patterns in ten years of previous hiring data, effectively learned to penalise resumes that mentioned women’s colleges or the word “women’s” (as in “women’s chess club captain”). The algorithm was encoding the historical gender bias of the human hiring decisions it had been trained on. Amazon reportedly abandoned the system after this was discovered; many similar systems, built by other companies, have never been publicly examined.
Healthcare algorithms have been found to systematically recommend less care to Black patients than to similarly-ill white patients. A 2019 study published in Science, by the researchers Ziad Obermeyer, Sendhil Mullainathan and colleagues, examined a commercial algorithm used to identify patients who would benefit from extra medical attention. The algorithm was using past healthcare spending as a proxy for health need. Because Black patients had historically received less healthcare spending for comparable conditions — a result of structural inequality in the healthcare system — the algorithm systematically underestimated their need for care. The pattern affected an estimated 200 million Americans before it was identified.
What’s actually going wrong
Across these cases, a few common mechanisms recur.
Training data reflects existing biases. Machine learning systems learn from historical data. If the historical data reflects biased decisions — hiring patterns shaped by discrimination, policing patterns shaped by differential enforcement, healthcare patterns shaped by unequal access — then the systems trained on that data will learn and reproduce those biases. The algorithm isn’t introducing bias; it’s absorbing and automating the bias already present in the training data.
Proxy variables smuggle in discrimination. Even when sensitive characteristics (race, gender) are explicitly excluded from an algorithm’s inputs, other variables that correlate with those characteristics can recreate the discrimination. Postal codes correlate with race in many cities. Names correlate with gender. Purchase histories correlate with class. An algorithm using apparently neutral variables can produce outputs that are almost as discriminatory as one explicitly using the prohibited variables, and it will be harder to detect.
Different definitions of fairness are mutually incompatible. As the COMPAS case illustrated, there are multiple reasonable definitions of what it means for an algorithm to be fair. Calibration (equal accuracy for each group at each score level). Equal false-positive rates. Equal false-negative rates. Equal outcomes regardless of group. When base rates of the underlying phenomenon differ between groups — as they do in most domains where predictive algorithms are used — these different definitions conflict. An algorithm that’s fair by one definition is often unfair by another. There’s no purely technical way to choose between the definitions; the choice requires ethical judgement.
Opacity obscures the problem. Many of the algorithms making consequential decisions about people are proprietary, complex, and opaque even to the people affected by them. A person denied a loan, or flagged as high-risk for parole, or screened out of a job application, often has no way to know why. They can’t challenge the decision; they can’t correct the input data; they can’t even identify which features of their situation produced the adverse outcome. This absence of explainability sits uncomfortably with basic ideas about procedural fairness.
The critical tradition
The most influential public articulation of these concerns came from the American mathematician and data scientist Cathy O’Neil, whose 2016 book Weapons of Math Destruction compiled many case studies of algorithmic harm. O’Neil’s argument, roughly: predictive algorithms become destructive when they meet three conditions — they operate at scale affecting many people, they produce feedback loops that reinforce their own predictions, and they’re opaque to those affected. A system hitting all three can produce compounding damage over years, with no individual decision looking obviously wrong but the aggregate pattern devastating specific populations.
The Australian-American scholar Kate Crawford, in her 2021 book Atlas of AI, has extended this critical analysis to the material and political foundations of AI systems — the labour conditions under which they’re trained, the environmental costs of computing infrastructure, the political interests of the companies building them. Her argument is that AI systems aren’t neutral tools; they’re artefacts of specific power relations, and understanding what they do requires understanding who built them and what interests they serve.
These critical traditions have shaped public and regulatory debate significantly. The European Union’s General Data Protection Regulation includes a provision giving individuals a qualified right to explanation of automated decisions affecting them. Multiple US cities have banned facial recognition use by law enforcement. The EU’s 2024 AI Act imposes specific restrictions on high-risk AI applications. The regulatory response, globally, has been patchy but growing.
The counter-thread worth hearing
Before endorsing the critical tradition wholesale, an important complication.
Some of the most widely-cited algorithmic-bias studies have been methodologically contested on careful re-examination. A 2018 Washington Post analysis found issues with some of the ProPublica COMPAS claims, and the subsequent academic literature has continued to debate how to interpret the data. The Gender Shades methodology has been extended and refined through subsequent research, and while the core finding of differential accuracy has held up, the specific magnitudes reported in early research haven’t always replicated in later work. None of this overturns the broader concern about algorithmic bias, but it does suggest that specific claims should be held with appropriate care.
A more fundamental counter-consideration: in some domains, algorithmic decision-making has actually reduced discrimination compared to human decision-making. Studies of blind auditions in orchestras, beginning with research by Claudia Goldin and Cecilia Rouse, have shown that removing identifying information from decisions substantially reduces demographic bias. Similar effects have been found in some algorithmic hiring systems that have been carefully designed to remove human judgement from early screening stages. Judges with discretion have been shown, in multiple studies, to exhibit their own patterns of bias — sometimes worse than well-designed algorithms. The relevant comparison for algorithmic fairness isn’t to a perfect decision-maker; it’s to the human decision-makers who would be doing the job without the algorithm.
What this suggests is that algorithmic decision-making can be better or worse than human decision-making, depending on the specific design and context. It isn’t uniformly worse, and it isn’t uniformly better. The serious question in any specific case is: would the human alternative be better or worse, given the biases and limitations that humans also bring to decisions? Sometimes the answer is that the human alternative is worse. Sometimes the answer is that the algorithmic version is worse. The blanket position — that algorithms should or shouldn’t be used — misses the real analysis.
What to actually think about
For a citizen living in a world increasingly full of algorithmic decisions, several working principles are worth holding.
When you’re affected by an algorithmic decision, ask for the reasoning. In many jurisdictions, including in some cases in Australia, you have legal rights to explanation of decisions significantly affecting you. These rights aren’t always easy to exercise, and the explanations you get may be partial, but the pressure of requests for explanation has been one of the meaningful forces improving how these systems are designed and deployed.
Recognise that opacity is often a choice, not a technical necessity. Many of the most opaque algorithmic systems could, in principle, be made more interpretable. The opacity usually serves the interests of the organisations building them rather than the people affected. Demands for transparency aren’t asking for the impossible; they’re asking for choices to be made differently.
Don’t romanticise human judgement as uniformly better. Human decision-makers have their own biases, often well-documented and persistent. The question in any domain is how the specific algorithmic system compares to the specific human alternative, not whether algorithms are good or bad in the abstract.
Support institutional responses, not just individual ones. The patterns documented by O’Neil, Crawford, Buolamwini and others can’t be solved by individual vigilance. They require regulatory frameworks, public oversight, professional ethics in data-science work, and legal remedies for harm. Being a citizen who supports these institutional responses matters more than any individual decision about your own data.
The question that remains
The deepest thing the research on algorithmic decision-making reveals is something about the transition humanity is in. Decisions that used to be made by humans, with all their biases and limitations, are increasingly being made by systems that aren’t obviously better or worse but are different in specific ways. These systems can encode discrimination at scales and speeds human decision-making couldn’t match. They can also remove forms of bias that human decision-makers reliably introduce. Whether the specific systems your society deploys are producing net harm or net benefit depends on choices that are being made, largely without public deliberation, by companies and governments whose interests don’t always align with the public’s.
This suggests a specific citizenly responsibility. Pay attention to where algorithmic decisions are being made about people. Ask whether the systems have been audited for bias. Support public oversight where it exists and advocate for it where it doesn’t. The alternative — leaving these decisions entirely to the technical and commercial logic of the institutions building the systems — has not, on the evidence available, produced the outcomes an informed public would have chosen.
The question worth carrying, the next time you encounter an algorithmic decision affecting you:
Who designed this system, with what data, for whose benefit — and what would it look like if you asked that question of every such decision that affects your life?
Key research referenced: ProPublica’s 2016 COMPAS investigation; Joy Buolamwini and Timnit Gebru, “Gender Shades” (2018); Ziad Obermeyer, Sendhil Mullainathan and colleagues’ 2019 Science paper on healthcare algorithmic bias; Cathy O’Neil, Weapons of Math Destruction (2016); Kate Crawford, Atlas of AI (2021); Claudia Goldin and Cecilia Rouse’s research on blind auditions.