Claude Duet, the EU AI Act, and the decision that happens before you’re asked

I read Anthropic’s Claude Duet announcement this week and stopped at one line. The agent, now in research preview for US Pro and Max users, works in the background. It reviews your connected accounts, flags what needs you, and asks for approval before sensitive actions.

That last phrase is doing a lot of work. It’s the phrase that makes a background agent feel safe. And it’s the phrase that most people will read as “a human stays in control.” I want to slow down on it. Not because Duet is badly designed. I haven’t used it, and asking before acting is a reasonable choice. But the phrase hides a structure that every agentic product shares, and that structure is where the real questions sit.

Two layers, one of them visible

An approval prompt has two layers.

The first is the checkpoint: the moment you see “Approve this action?” and choose yes or no. This layer is visible, interactive, and gives you something that feels like authority. You decide. The system waits.

The second layer is the filter. It’s the logic that decided this action needed your approval and the previous forty didn’t. You never see this layer. You don’t interact with it. In most products you can’t inspect it, and you can’t change it in any meaningful way. Yet it shapes your oversight more than anything else, because it decides what reaches you at all.

This is the interface-and-system gap in a very clean form. The interface shows you a decision point. The system has already made the bigger decision: where to put the decision points. When we talk about human-in-the-loop design, we usually evaluate the first layer (is the prompt clear, can the user decline, does the agent wait). We almost never evaluate the second layer, which is where the authority actually lives.

I call this the authority illusion. You’re given a real choice at the checkpoint, so the checkpoint feels like the place where your authority sits. But the power to decide what you’ll ever be asked about lives one level up, in a judgment you didn’t make and can’t see.

There’s a second asymmetry on top of this. When you hand an agent a background task, you delegate the broad work (monitoring, sorting, deciding what matters) and keep only a narrow slice (ratifying what it surfaces). The agent’s scope grows over time. Yours shrinks to the yes/no box. That’s delegation asymmetry, and the approval prompt is where it shows up.

What the law means by oversight

It helps to compare the everyday meaning of “human oversight” with the legal one, because they’re not the same.

Article 14 of the EU AI Act defines oversight for high-risk systems quite precisely. The people overseeing a system should be able to:

• understand what the system can and can’t do, and spot anomalies or unexpected behavior
• stay aware of automation bias, the tendency to over-trust outputs, especially when the system is recommending a decision
• correctly interpret what the system shows them
• decide not to use the system, or override or reverse its output
• intervene or stop it, bringing it to a safe state

Look at that list next to an Approve / Deny prompt. The prompt covers part of one item (override, for this one action). It doesn’t help you understand the system’s limits. It doesn’t protect you from automation bias. In fact a stream of reasonable-looking prompts trains automation bias: after the twentieth sensible request, the twenty-first gets approved on reflex. And it says nothing about the actions that never produced a prompt.

So by the Act’s own definition, an approval gate isn’t oversight. It’s one control inside what oversight is supposed to be.

The scope gap

There’s a catch, and it matters for how we talk about these products.

Article 14 applies to high-risk systems. The Act defines high-risk mostly through Annex III, which lists specific use areas: biometrics, critical infrastructure, education, employment, access to essential services, law enforcement, migration, and the administration of justice. A system is high-risk under that route only if it falls in one of those areas and performs the function described. A general consumer assistant that manages your email and calendar probably doesn’t qualify. (That’s my reading of how the test works, not an official ruling on Duet. A lawyer would give you a firmer answer.)

The timing matters too. This summer EU lawmakers agreed to delay the high-risk obligations. Article 14 now applies from December 2027 for Annex III systems and August 2028 for Annex I systems.

Put those together. The legal definition of meaningful oversight exists on paper, but it doesn’t reach most consumer agents, and where it does reach, it isn’t binding yet. That leaves a gap. Products ship now using the language of oversight (“asks for approval,” “you stay in control”) without being held to what oversight means. In that gap, “it asks for approval” is a product decision, not a compliance claim. That’s not a scandal. But it does mean the quality of the oversight depends entirely on design choices, and those choices deserve the scrutiny we’d give a regulation.

Regulators are already moving past the approve button

The more interesting development this year is in industries where oversight is enforced. There, regulators and firms are quietly admitting that reviewing actions one at a time doesn’t work for agentic systems.

In February, a Bank of England summary of industry roundtables reported firms saying traditional model validation wouldn’t be sustainable, and the human-in-the-loop concept was openly challenged. In May, the Federal Reserve, OCC, and FDIC amended their model-risk guidance to exclude generative and agentic AI, which effectively admits the old framework doesn’t fit. IOSCO’s capital-markets toolkit, also from May, warned about agentic AI and noted regulators experimenting with “AI as a judge.” And in April the World Economic Forum told boards to govern the reward function rather than audit execution, and to build in “legible friction”: deliberate pause points for high-stakes actions, placed by design.

The common thread is a move from overseeing actions to overseeing objectives, guardrails, and outcomes. Put differently, governance is shifting from the checkpoint layer to the filter layer. The question is no longer “did a human approve this step.” It’s “who set the boundaries, and can anyone see and question them.”

That’s the same question the Duet announcement raises, coming from the other side. Regulated finance is concluding that per-action approval can’t carry the weight of oversight. Meanwhile consumer agents are leading with per-action approval as their main safety story. Both can be true at once, but they shouldn’t be confused.

The filter is a design surface

This is where UX comes in, and why I think this belongs to designers as much as to legal teams.

If the filter is where authority lives, then the filter is a design surface, and it can be made visible, explainable, and adjustable like any other part of the product. When I work with teams building agentic features, these are the patterns I push for:

1. Show the negative space. Give people a readable log of what the agent did without asking. Not a raw audit trail, a plain summary (“handled 38 routine items this week; here’s the list”). Oversight starts with knowing what happened when you weren’t asked.
2. Explain the flag. Every approval prompt should say why it crossed the line (“this sends money,” “this emails someone new,” “this deletes data”). The reason teaches the user the filter’s logic, which is the first item on Article 14’s list.
3. Let people move the boundary. “Sensitive” shouldn’t be a fixed vendor setting. Let users tighten or loosen it by category (always ask before payments, never ask about archiving newsletters). That turns the filter from something done to you into something you configure.
4. Make stopping easy. A clear pause-everything control, reachable in one step, that leaves things in a known state. That’s Article 14’s “stop” requirement in interface form.
5. Measure the gate, not just the agent. If users approve 99% of prompts in under two seconds, the gate is ceremony. Approval rate, time-to-decision, and how often people reverse an approved action afterwards are signals of whether oversight is real or just a habit.
6. Place friction on purpose. Following the WEF’s phrase, friction should be legible: placed where consequences are high and hard to undo, explained when it appears, and absent where it adds nothing. Too many prompts cause fatigue. Too few hide the filter. Both are failures of placement.

None of this requires regulation to arrive first. It requires treating the boundary-drawing as a product decision with consequences, owned by someone who can explain it.

The boundary is the policy

Claude Duet is a useful case because it does something reasonable. It asks. But a reasonable checkpoint can still sit on top of an unexamined filter. The honest way to evaluate any agent isn’t “does it ask before acting.” It’s three questions:

• Who decided what it would ask about?
• Can I see what it chose not to ask?
• Can I change where that line sits?

Until oversight rules catch up with agentic products (and on current timelines that’s years away for most of them), the answers to those questions aren’t set by law. They’re set by design. Right now, the boundary is the policy, and designers are the ones drawing it.

More on interface and system design for AI at uxandai.com.

Sources:

• Article 14, Human Oversight, EU AI Act
• EU AI Act Annex III high-risk use cases
• EU Lawmakers Reach Provisional Agreement to Delay Key EU AI Act Obligations (Sidley)
• After Human-in-the-Loop: Regulators Concede Oversight Cannot Scale to Agentic AI
• Claude Duet announcement, Anthropic