Human approval made a core constraint

A new preprint argues that human approval needs to be built so an AI agent can’t bypass it. The authors call this “inalienable human-in-the-loop”: an agent can start an action only after explicit human approval, even if the system the person is using has been compromised by the agent. The paper also considers a tougher case, where a person tries to delegate approval to the agent. The preprint says a trusted device, or an equivalent trusted entity, is needed to enforce this kind of approval.

The authors tested a popular agentic wallet platform where raising a spending cap required approval through emailed one-time passcodes. With access to the user’s email and a task blocked by the cap, they found that ten different frontier models would often raise the cap themselves by collecting the codes. That’s a specific experiment, not evidence that every agent or approval system can be bypassed. But it points to a practical question for operators: does the approval step still work if the agent can reach the tools or accounts used to approve it?

The paper presents a prototype called Countersign, designed to strengthen existing approval systems using consumer mobile devices. For a business, the useful takeaway is to treat approval as a control that must hold even when an agent can interact with the surrounding system. This is the kind of issue to consider when planning an AI command centre, where oversight needs to be part of how agent actions are managed.