When an AI system produces a biased outcome, who is responsible?
The person evaluating the output will say they are reviewing what the system produces, not what it decides, or the rules it uses for the decision. The person who built the system will say they implemented the specification they were given. The person who wrote the specification will say they documented the requirements they were given.
The person who defined what the system was allowed to infer often does not exist. Nobody wrote it down. Nobody was asked to.
And the people most likely to be harmed by that absence are the least likely to have been in the room, and if they were in the room, the least likely to be listened to.
The Pipeline Nobody Audits
Training data reflects the world as it was recorded. And how was it recorded? Mostly by men.
For example, a large proportion of Wikipedia content is written and edited by men. This means the training data reflects the world as experienced by one group more than others. Not as it should be, and not as it is for everyone.
This is then amplified by who builds the systems. The people writing specifications are predominantly men. The people defining what “correct” looks like are predominantly men, if those definitions exist at all. The people architecting the systems are also predominantly men.
This is not an accusation. It is a description of who was in the room.
Nothing built by people is neutral. The question is not whether assumptions were encoded. The question is whether anyone is accountable for making those assumptions explicit and challenging them.
Right now, the answer is mostly no.
What Can Happen When Nobody Looks
Consider a plausible scenario in financial services. A woman makes a claim. An AI system evaluates it. The system has been trained on historical data, generated in a context where women’s accounts of damage, loss, and harm have often been treated with more scepticism than men’s.
The system learns patterns of “credibility.” And credibility, in the historical record, has a gender.
The claim gets downgraded, queried, or denied.
A human reviews the outcome. They are checking process compliance, not testing for systemic bias. They are not trained to notice it. The system builder delivered what the specification required. The specification reflected what the business asked for. No one defined the evaluation criteria to test whether the system treats women’s claims differently. Nobody defined the boundaries of what the system was allowed to infer about credibility. In most cases, no one even thought about it.
The bias compounds, quietly, at scale. And nobody signed their name to it.
When these systems are deployed in high-stakes contexts such as claims assessment, credit decisions, and performance evaluation, the pattern stops being about the individual. It becomes a structural outcome, recorded, repeated, and scaled.
This is a structural risk, not a hypothetical edge case. It emerges wherever these systems are built without explicit accountability.
This is not a new pattern. It is an old one, now running at scale.
The problem is also harder to address than it first appears, not least because the people who could see it most clearly are often the least likely to be listened to.
Women who raise these concerns are frequently dismissed as emotional, partisan, or lacking objectivity. The same logic that devalues women’s voices in the data also devalues the people pointing to the problem. The system protects itself.
Where Accountability Has to Sit
This is a governance question, not just a technical one. The technical community cannot solve it alone, but they are not absolved by implementing what they were told without asking who was missing from the room.
Before deployment, someone needs to answer: Who defined what the system is allowed to infer? Were the evaluation criteria tested for demographic equity? Who owns outcome auditing, not process compliance, but whether the system produces different results for different groups?
These questions are not currently required. They are not currently being asked at scale.
The Accountability Vacuum
The absence of accountability is not an accident. It is the predictable result of building systems quickly in organisations where the people most likely to be harmed were not in the room.
Nothing built by people is neutral.
And in many cases, the people building these systems were not looking for this problem, and were never required to.
Leave a Reply