"Just add a human review step."
That's what most compliance advice says about Article 14. It's technically correct and practically useless. I've seen three implementations of human oversight for AI systems in the last six months. All three would fail an audit.
The problem isn't that teams are ignoring human oversight. They're implementing it wrong—in ways that satisfy neither the letter nor the spirit of the regulation.
Here are the three mistakes I see most often, and what to do instead.
What Article 14 actually requires
Before we get to the mistakes, let's establish what Article 14 actually says. The regulation requires that high-risk AI systems be designed to allow "effective oversight by natural persons."
Article 14(4) specifies that human oversight must achieve one or more of these functions:
- Fully understand the AI system's capacities and limitations and be able to monitor its operation
- Remain aware of automation bias and be equipped to address it
- Correctly interpret the AI system's output taking into account the characteristics of the system
- Decide not to use the AI system or disregard, override, or reverse its output
- Intervene or interrupt through a "stop" button or similar procedure
Notice what this requires: not just the ability to review, but the ability to understand, interpret, override, and stop. That's a much higher bar than "click to approve."
Mistake 1: The rubber-stamp button
The most common implementation I see: AI makes a decision, human clicks "Approve" or "Reject," decision is executed.
This fails Article 14 in multiple ways.
The problem
A button doesn't ensure understanding. If the human can't see why the AI made its recommendation, they can't "correctly interpret the AI system's output" (requirement 3). They're just rubber-stamping.
I watched one team demonstrate their "human oversight" system. An AI screened job applications and flagged candidates as "proceed" or "reject." A human reviewer saw the candidate name, the AI's recommendation, and two buttons. Average review time: 4 seconds.
That's not oversight. That's a liability shield that won't hold up.
Why teams do this
Speed. Adding meaningful human review slows things down. If your AI processes 1,000 applications per day, and meaningful review takes 2 minutes each, you need 33 hours of human time daily. That's expensive.
So teams optimize for minimal friction. The result is oversight that looks compliant but isn't.
The fix
Provide context for each decision. At minimum, the human reviewer needs to see:
- The inputs the AI considered
- The factors that influenced the decision (even if approximate)
- Confidence level or uncertainty indicators
- Similar past decisions for comparison
If you can't provide this information, you either need to build explainability into your AI, or you need to redesign your oversight process.
Mistake 2: Oversight without authority
The second mistake: putting oversight in the hands of people who can't actually stop or override the AI.
The problem
Article 14(4)(d) requires that humans be able to "decide not to use the AI system or to disregard, override or reverse the output of the AI system."
In practice, many oversight systems give reviewers a "flag for review" option but no actual authority to override. The flag goes into a queue. The AI decision proceeds. Eventually someone might look at the queue.
I've seen this in credit decisioning systems. A loan officer can flag a decision for review, but the automated decision executes immediately. The review is post-hoc. By the time someone looks at it, the loan is already approved or rejected.
That's not oversight. That's logging.
Why teams do this
Because blocking decisions on human approval creates bottlenecks. If your SLA is "credit decision in 60 seconds" and human review takes 10 minutes, you have a problem.
Teams solve this by making human review async and non-blocking. Fast, but non-compliant.
The fix
There are legitimate ways to make human oversight efficient without eliminating authority:
- Risk-based routing: Only high-risk or edge-case decisions require blocking human review. Low-confidence or high-impact decisions get queued for approval before execution.
- Sampling with consequences: Review a random sample in real-time. If override rates exceed a threshold, halt automated decisions until investigation.
- Tiered approval: Low-stakes decisions proceed automatically. High-stakes decisions require human sign-off.
The key: for at least some decisions, humans must be able to stop execution before it happens.
Mistake 3: Ignoring automation bias
Article 14(4)(b) requires that humans be able to "remain aware of the possible tendency of automatically relying on or over-relying on the output" of the AI system.
This is automation bias: the tendency to defer to automated systems even when they're wrong.
The problem
Most human oversight implementations ignore this entirely. They assume that putting a human in the loop solves the problem. It doesn't.
Research consistently shows that humans defer to AI recommendations, especially when:
- The AI is usually right
- Overriding takes effort
- There's time pressure
- The human lacks domain expertise
In one study, radiologists missed 30% of cancer findings that an AI system also missed—even though they would have caught them without the AI. The AI's confidence was contagious.
If your oversight system makes it easy to agree with the AI and hard to disagree, you're building automation bias into your process.
Why teams do this
Because the AI is usually right. If your AI has 95% accuracy, overriding it feels like betting against the house. Reviewers learn that agreeing is usually correct and requires less justification.
The fix
Design your oversight system to counter automation bias:
- Require justification for agreement, not just disagreement. If reviewers must explain why they agree with the AI (even briefly), they engage more critically.
- Insert deliberate friction. Delay showing the AI's recommendation until the human has formed an initial view. This is "blinding" and reduces anchoring effects.
- Train reviewers on automation bias. Make it an explicit part of onboarding. Show examples where the AI was wrong and humans missed it.
- Monitor and flag low-override rates. If a reviewer agrees with the AI 99% of the time, that's a signal—either the AI is perfect for their cases, or they're not really reviewing.
- Rotate difficult cases. Some cases are harder than others. If the same reviewer handles all edge cases, they may become fatigued. Rotate to keep engagement high.
What actually works
After seeing the mistakes, here's what a compliant human oversight implementation looks like:
1. Context-rich review interface
Don't show just the recommendation. Show the inputs, the reasoning (to the extent explainable), the confidence, and relevant comparisons. Give reviewers what they need to actually evaluate the decision.
2. Real authority with appropriate routing
For high-stakes or uncertain decisions, make human approval blocking. The decision doesn't execute until a human signs off. For lower-stakes decisions, use sampling and monitoring with clear escalation triggers.
3. Bias-aware design
Build in friction that counters automation bias. Consider showing the AI recommendation after the human forms an initial view. Require explanations for both agreement and disagreement. Monitor override patterns.
4. Trained, accountable reviewers
Human oversight only works if the humans are equipped for it. That means:
- Training on the specific AI system's capabilities and limitations
- Training on automation bias and how to counter it
- Clear accountability for review quality
- Adequate time to actually review
5. Documentation and metrics
You need evidence that oversight is working. Track:
- Review time per decision
- Override rates
- Outcome accuracy for human-reviewed vs. non-reviewed decisions
- Reviewer calibration (do their judgments match eventual outcomes?)
Key Takeaways
- Rubber-stamp buttons don't count: Reviewers need context to actually review. A button alone fails Article 14.
- Authority matters: Oversight without the power to stop or override isn't oversight. At least some decisions must require blocking human approval.
- Automation bias is real: Humans defer to AI recommendations. Your system must counter this tendency, not enable it.
- Design for engagement: Make it easy to understand, possible to override, and hard to rubber-stamp. Build friction that ensures genuine review.
What to Do Next
- Audit your current oversight process. Time a few reviews. Can reviewers explain why they agreed with the AI? If not, you have a rubber-stamp problem.
- Check for blocking authority. Can any human reviewer stop a decision from executing? If not, you have an authority problem.
- Measure override rates. If they're near zero, investigate whether reviewers are genuinely reviewing or just approving.
Human oversight is one of the hardest requirements to implement well. It's also one of the most likely to be scrutinized in an audit. Get it right.
Stay compliant out there.
— The Compliantist