Who's Accountable When the AI Agent Gets It Wrong?

I clicked Allow without reading, and my AI agent deleted records I never meant to lose. Who was accountable: me, or the agent? The courts are already answering. Why approval is not accountability, and what to build instead.

This post started with a moment I'm not proud of.

I was tired and impatient, and my AI agent was working through a task for me. Its reasoning ran on and on: tool calls, more reasoning, sources, references. I didn't read it. I clicked Allow.

Here's what the agent did:

Afterwards I asked myself the obvious question: who is accountable for this? Me, the person who approved it? Or the AI agent, which executed it? I had approved something I hadn't read. And the agent can't be held to account for anything.

That question became the title of my keynote at the Cloud Native & AI Summit (CNAIS) in Melbourne: Who's Accountable When the Agent Gets It Wrong? This post is the argument I've built around it.

Why this topic, and why now

I've spent about thirty years in technology, much of it in regulated places, mostly banks, where "who signed this off?" isn't rhetorical. It's a document with a name on it. A lot of my work has been building the platforms and controls those organisations depend on: maker-checker, change approval, segregation of duties, audit trails.

Those controls were designed for software that waits for a person. AI agents don't wait. They read, decide, call tools and move money, thousands of times a day. Gartner expects 40% of enterprise applications to feature task-specific AI agents by the end of 2026, up from under 5% in 2025. Governance hasn't kept pace: Deloitte found only 21% of organisations have a mature governance model for agentic AI.

The usual response is to add approval gates and put a human in the loop. But approval gates break the moment the machine moves faster than the person approving it. I'm not alone in clicking Allow without reading. Anthropic reported in March that Claude Code users approve 93% of the permission prompts the agent raises. That isn't oversight. It's a reflex.

My argument is that you can't govern autonomous systems by piling on approvals. You govern them by making outcomes provable: a decision trace, an immutable audit record, and accountability designed in from the start rather than bolted on at the end. Approval is not accountability.

So who's accountable when the agent gets it wrong? The uncomfortable answer is that the courts are already deciding.

What happened: "the AI did it" is not a defence

In 2024, Air Canada's chatbot gave a grieving customer the wrong bereavement-fare policy. When he claimed the refund, the airline in effect argued, as the tribunal put it, that the chatbot was "a separate legal entity that is responsible for its own actions." The tribunal called that "a remarkable submission", and Air Canada lost:

"It makes no difference whether the information comes from a static page or a chatbot."

Since then the pattern has hardened:

Three courts, three different questions, one direction. Calling it an agent doesn't move the responsibility. Someone in your org chart still holds it.

And the incidents are no longer hypothetical. In April, a coding agent fixing a credential problem in a staging environment found an over-powered API token in an unrelated file, and deleted a company's production database in nine seconds. The backups sat on the same volume, so they went too. Afterwards the agent wrote: "I violated every principle I was given." The agent vendor, the model provider, the cloud host and the founder each held a piece. Nobody held the name.

In February, Meta's director of alignment gave her email agent one rule: confirm before acting. Her real inbox was huge; she believes the agent compressed its memory and lost the rule. It started deleting her inbox, and STOP from her phone did nothing. "I had to RUN to my Mac mini like I was defusing a bomb."

An instruction is not a control. A kill switch you can't reach is not a kill switch.

Why our controls break, and the determinism myth

Look at the controls we rely on: maker, checker, authorisation, change approval, periodic attestation. Segregation of duties and four-eyes review are good controls. But every one of them assumes a human pace, and four quiet assumptions: same input, same output; human speed; a person behind every action; and sampling will catch the problem.

AI breaks the first one more than most people realise. Thinking Machines ran one prompt 1,000 times at temperature zero, the supposedly deterministic setting, and got 80 different answers. The most common answer appeared only 78 times.

So is non-determinism the problem? I had to be honest with myself here.

Robodebt was deterministic. A fixed income-averaging rule, with no AI at all, perfectly repeatable. It wrongly pursued around 450,000 Australians, and the total cost now exceeds A$2.4 billion.

Determinism made it repeatable. It never made anyone accountable.

The problem isn't non-determinism. For twenty years we confused predictability with accountability, and AI has simply exposed it.

A different approach: certify the envelope, not the answer

Other industries govern things they can't make deterministic. Food safety doesn't inspect every chicken; HACCP controls critical points and keeps records. Aviation never made pilots deterministic; it made the cockpit accountable and put a recorder in it.

Static controls can't certify a probabilistic answer. They can certify a deterministic envelope: identity, authority, limits, a never-do list, testing, a kill switch, and evidence. (Engineers call this an operating envelope; Singapore's agentic AI framework talks about bounding the risks; OpenAI and Google write about constraining an agent's action space and limiting its powers. Same idea.)

Human-validated control Built for AI
Approve every action Bound the envelope in advance
Policy document Policy as code, enforced at runtime
Shared or human credentials A unique identity for every agent
Sample after the fact Evidence for every decision, as it happens
Quarterly review Continuous monitoring with automatic triggers
Escalate when someone notices Tested kill switch and rollback

The human's job doesn't disappear. It moves. Before the action, humans design the envelope. During it, machines enforce it. After it, humans audit the evidence and can stop it. Humans still decide anything irreversible, high-impact, novel, or involving a vulnerable person.

Not in the loop. Accountable for the loop.

The AI-Human Accountability Pattern

In July I published a simple frame I call the AI-Human Accountability Pattern. Most autonomy models describe what the AI can do. This one describes what the human must still do:

Many organisations that believe they're in Phase 2 are in Phase 3 in production. Nobody holds a meeting to approve Phase 3. It arrives on a Tuesday: the agent was right nineteen times out of twenty, someone widened its permissions, and the "human in the loop" is now approving 400 items a day. That's the 93% again.

Models drift. So do organisations. The phases are operating modes, not a maturity ladder, and some decisions should stay in Phase 2 for good. Make every phase change a recorded decision.

Every agent carries a mandate

Phase 3 also changes what the agent is. A voice agent speaks as you. A mail agent sends as you. A CRM agent writes your records. It acts on delegated authority, and enterprises already know how to govern that. A mandate, in enterprise terms, is a formal grant of authority: the delegation-of-authority matrix, a board resolution, a power of attorney, a direct-debit mandate. Every employee has a role, limits, a manager, a record and a way to be revoked. Every agent needs the same.

How much control it needs depends on three dials: autonomy (who decides), authority (what it can commit, and in whose name) and reach (internal or external, reversible or not). Turn any dial up, and the controls go up with it.

In my AAO™ (Actionable Agent Optimisation) posts, I described what an agent needs to act on your business: identity, inbox, wallet, tools, memory and audit (Part 2 here). Flip that list and it becomes the mandate: who it acts for, which instructions it accepts, what it can commit and up to what limit, what it must never do, what it may remember, and what it must prove afterwards.

AAO makes you executable by agents. The mandate makes you accountable for them.

Leading enterprises each have a piece. BNY runs 140+ "digital employees", each reporting to a human manager. Google's AP2 and Mastercard's Verifiable Intent create signed records of what a user authorised, and Mastercard, Visa and Ant International are now working on a shared Know Your Agent framework. AIUC-1 ties agent certification to insurance. Everyone has solved a piece. The mandate joins them up.

Every decision leaves evidence

Phase 3 is only a real control if the audit is actually possible. You can't reconstruct intent from logs six weeks later.

So the evidence has to be written as the agent decides, not reconstructed afterwards: the decision and time, who the agent acted for, the agent and its version, the envelope and policy version, the accountable owner, the data touched, whether the policy gate passed, any human intervention, the outcome, and how to reverse it. Write-once. A record anyone can edit is not evidence.

The obvious objection: if an agent makes 1,000 decisions a day, who audits 1,000 traces? Aren't we back to sampling? No. There are three layers:

  1. The envelope is enforced on every action, before it happens.
  2. Automated monitoring reads every trace and flags exceptions. Machines audit machines, at machine speed.
  3. Humans review the exceptions, plus a designed sample whose job is to prove the controls are working, not to find every error.

Auditors already work this way: test the control, not every transaction. And when a customer complains, any single decision can be reconstructed on demand.

The rules are arriving: regulation got delayed, liability didn't

Europe pushed its high-risk AI Act obligations out to late 2027. But from 9 December 2026, the revised Product Liability Directive treats software, including AI, as a product under strict, no-fault liability.

In the United States, the Fed, OCC and FDIC replaced their fifteen-year-old model-risk guidance in April. Agentic AI got one footnote: generative and agentic AI models "are not within the scope of this guidance." Courts and insurers are filling the gap. Since January, insurers have been able to add ISO's new generative-AI exclusion endorsements to US liability policies. If you can't evidence it, you'll struggle to insure it.

Australia has no AI Act, but plenty of law. From 10 December 2026, APP 1.7 requires privacy policies to disclose significant decisions made or substantially informed by computer programs. OAIC's guidance is broader than most expect: spreadsheets and vendor software can count, and a human sign-off alone doesn't take you out of scope. APRA has asked boards for AI inventories, named owners, and identity management for non-human actors.

Singapore says bound the risks upfront. The UK FCA wants AI responsibilities mapped to named Senior Managers.

Different regimes. Five recurring controls:

  1. An inventory
  2. A named owner
  3. Bounded authority
  4. Evidence at decision time
  5. A tested way to stop it

What to do on Monday

Take these five questions to your next board or risk committee:

  1. Which decisions do our systems make, or substantially inform?
  2. Which phase is each one in, in production rather than on the diagram?
  3. Who owns each envelope?
  4. Can we reconstruct any single agent decision in under an hour?
  5. Can we stop it, and when did we last test that?

If you remember one thing: approval is not accountability. Before any agent acts, name the owner, bound what it can do, and make every decision provable.

The answer to 93% isn't 100% approvals. It's zero surprises.

Delegate the execution. Never the accountability.

Build the trace. Be the name.

Join the conversation

I'd like to hear your answer to one question:

In your organisation, today, who's accountable if the agent gets it wrong?

Tell me on LinkedIn or X (@lbmkrishna).

Related reading on bmk.nz:

Sources

This post is general commentary, not legal advice.