top of page

Who decides when the AI is wrong?

Updated: Apr 23


When an AI system produces outputs that worry the people closest to it, but no threshold has formally been breached, who decides whether that is a problem worth acting on?

This is where a quiet but consistent governance failure appears. Not the absence of escalation pathways, but the absence of defined intervention thresholds.



The Borderline Problem


AI systems do not typically fail in obvious ways; they drift. They produce outputs that are statistically defensible but operationally concerning. They perform within documented tolerances while generating results that experienced practitioners recognise as wrong.


In these conditions, escalation depends on someone making a judgment call.

Is this a threshold breach? Does this qualify for intervention? Who decides?

In most governance frameworks, those questions have procedural answers. Risk committees, model validation processes, performance monitoring dashboards. All of these exist.


But the procedural answer and the operational answer are often different.

In practice, when outputs are borderline, and the system is live, the decision about whether to act frequently falls to whoever is closest to the data, with whatever informal authority they happen to carry. The governance framework addresses the extreme case. The borderline case is governed by proximity and social permission, not by defined authority.


Research published in Harvard Business Review in March 2026 captures the C-suite version of this precisely. When AI systems cut across operational, risk, technology, financial and compliance functions, every leader has a legitimate claim to oversight. The result is not clarity but fragmentation. And fragmentation under uncertainty often defaults to inaction.



The authority vacuum


Organisations invest considerably in defining who owns AI systems. Who governs the deployment, who oversees the model, and which committee has accountability for outcomes?


Ownership is documented.

What is rarely documented is who can intervene, under what conditions, without delay, without committee approval.


Ownership is a governance question. Intervention authority is an operational one.

When a live system produces borderline outputs, the governance question does not help. What matters is whether a named individual has the authority, the mandate, and the defined trigger to call the system into question before harm scales.


Gartner's June 2025 research projects that over 40% of agentic AI projects will be cancelled by the end of 2027, citing inadequate risk controls as the primary cause. Inadequate risk controls is another way of describing what happens when authority vacuums meet borderline signals.


The system continues because no one has been given the unambiguous right to stop it.



What the UnitedHealth Case shows


Patients were being denied Medicare coverage. A lot of them. When those denials were reviewed by humans, the error rate was reportedly close to 90%.

UnitedHealth's class action case has been running since 2023. The details remain in dispute.


But the structural question it raises is straightforward. A system was making consequential decisions continuously and at scale. The signal that something was wrong was visible in the data, so the gap was not detection. It was the absence of a named person with a defined obligation to call the system into question before that scale became irreversible.



What defined thresholds actually require


So what does fixing this actually look like?

Three things, not complicated in principle, but consistently avoided in practice because they require naming people rather than assigning responsibility to functions.


First, a named person, rather than the committee that owns the AI system, or the function that oversees it. A specific individual who can pause it, question its outputs, or call it in without needing to book a meeting to get permission.


Second, a defined trigger with a defined condition. Something concrete enough that when it appears, there is no debate about whether it qualifies, such as a complaint volume, a measurable deviation from expected outputs, data pattern. 

Third, a pathway that has actually been used. Documentation of what should happen is not the same as knowing what will happen. If the mechanism has never been tested, it has not been proven.


Singapore's IMDA Model AI Governance Framework for Agentic AI,, launched at the World Economic Forum in January 2026, puts this plainly: the ability to intervene must be designed in, not assumed. That standard applies not only to agentic AI but to any AI system making consequential decisions at scale.


McKinsey's 2025 State of AI survey found that 51% of organisations experienced at least one negative AI consequence in the past year. Borderline outputs are not edge cases. They are the operating conditions.


Governance frameworks are good at defining who is responsible when something goes wrong.


They are much less good at defining who is empowered to act before it does.

Ownership assigns accountability after the fact. Intervention authority prevents the fact from occurring.


In your organisation, does the person responsible for a live AI process know specifically what would trigger them to intervene, or is that threshold still a judgment call waiting to be made under pressure?


Part of a 12-week series on AI governance and organisational readiness.


Dr Joanna Michalska is the founder of  Ethica Group Ltd, which advises boards and C-suite leaders on decision authority and governance architecture under automation.


Sources: 





 
 
 

Comments

bottom of page