The irony of human oversight of AI agents

We rely on human oversight to keep AI agents safe. As long as there is a human in the loop somewhere — watching, assessing and able to intervene — we assume we ultimately remain in control. But that is exactly where an uncomfortable paradox emerges: using the very AI systems that human is supposed to supervise can slowly weaken the skills required for good supervision.

The human-in-the-loop paradox

That is the central warning in the recent scientific position paper AI Agents Push Humans Out of the Loop by Margaret Mitchell, Avijit Ghosh and Samir Passi. The authors argue that the problem is not only that increasingly autonomous AI agents are harder to control. Their sharper point is that prolonged use of these systems can also erode the cognitive capacities of human supervisors. As an agent gathers more information, runs analyses, makes choices and executes actions autonomously, the human needs to think, weigh and decide less often. As a result, precisely the expertise we count on when the system fails or ends up in an exceptional situation can fade.

Not new: Ironies of Automation

Strikingly, this problem is not new at all. In 1983, psychologist and researcher Lisanne Bainbridge described the same mechanism in her now-classic paper Ironies of Automation. Her observation was as simple as it was powerful: through automation we try to remove human error and limitations from a system, but in the end we leave the human responsible for exactly the situations that could not be automated. Those are usually the rare, complex and unexpected situations. At the same time, because of automation, that same human has had fewer and fewer opportunities to practise the skills needed to act well at precisely such a moment.

That is the irony of automation: the better automation takes over normal work, the less prepared the human may become for the moment when human intervention is needed most.

A new dimension with agentic AI

With agentic AI, that old irony gains a new dimension. Traditional automation mainly took over actions and predictable process steps. AI agents can potentially also perform cognitive work: searching for information, interpreting it, making plans, weighing alternatives, communicating with other systems and executing follow-up actions autonomously. The human thereby shifts ever further from executor to supervisor.

That sounds efficient, but supervising work you barely perform yourself is complicated. To recognise an error you need to understand what a good outcome looks like. To correct a decision you need enough knowledge yourself to formulate an alternative. And to intervene at a critical moment, you need to have been mentally engaged enough with what preceded it.

Human in the loop is no reassurance

This makes human in the loop an overly simplistic reassurance. The question is not whether a human formally sits somewhere in the process. The relevant question is whether that human is still genuinely capable of meaningful oversight. Does that person have sufficient visibility into what the agent is doing? Do they understand why certain choices are made? Is their own critical thinking still regularly engaged? And do their knowledge and skills remain developed enough to push back against a system when needed?

Mitchell, Ghosh and Passi therefore argue that supporting human cognitive capacities should be as serious a part of AI agent design as improving the capacities of the agent itself.

Implications for productivity and organisations

This also has consequences for how organisations think about productivity. The temptation is great to ask at every step: can AI take this over too? But perhaps we should ask a second question alongside it: what happens to our organisation when people no longer do this themselves?

Not every human action that can technically be automated should automatically disappear. Sometimes it can be wise to deliberately preserve moments of human involvement: making your own analysis before looking at the AI output, explicitly leaving certain decisions to people, regularly letting employees practise without AI, or designing systems so that not only the answer but also important intermediate steps remain visible. Not because AI could not do the work, but because the human must remain able to understand and assess it.

The fundamental design question

The fundamental design question for agentic AI is therefore not how many people we can take out of the loop. It is which role we want humans to retain in a world where ever more of that loop can be automated. Because if human oversight is ultimately our last line of defence, we cannot afford to slowly degrade that human into someone who only has to click a green checkmark. The greatest irony would be that we try to make AI safe with human oversight, while simultaneously building systems that hollow out the very capacity for that human oversight ever further.

About the author: Job van den Berg is an AI keynote speaker, tech entrepreneur and author of five books on AI. He deploys agents in production weekly and delivers 150+ keynotes per year on AI agents and agentic commerce.