The AI Learning Hub Journal

Where a Human Must Stay in the Decision

Where a human must stay in the decision — and what makes that realexplanation and appeal duties attach to the decision, not to the technology that produced it DECISIONS WHERE A DUTY SURVIVES AUTOMATION — NAME THE REGIME BEFORE ASSUMING ITa credit decline in the US —ECOA and Regulation B requirespecific principal reasons onan adverse actiona solely automated decisionwith significant effects in theEU — rights to humanintervention and to contesta retail recommendation —MiFID II suitability in the EU;Regulation Best Interest and thefiduciary standard in the USaccount freezes and exits,claim declines, suspicious-activity filings — each landson an identifiable person“the model said no” satisfies none of them — and elsewhere the duty may be differently drawn, or absent WHAT MEANINGFUL REVIEW NEEDS — FOUR THINGS THE DESIGN USUALLY OMITSthe inputs and the reasons,not only a score — otherwisethey cannot disagreetime proportionate to thedecision being reviewedauthority to overturn, withouta personal cost for doing itmonitored override rates —the metric that shows whetherreview happens at all the review step is doing work near-zero overrides: deference to a confident systemnear-total: the output is being ignoredautomation bias is a design problem — telling people to be sceptical does not fix an interface WHO IS ACCOUNTABLE WHEN THE MODEL IS WRONG an owner for the model named, and currently in post an owner for the process the model sits inside a line to a senior individual answerable for the outcomea named role with nobody currently in it is an unowned model — check the name, not the org chartthe test: when a supervisor asks who decided, is there a name — and can that person describe the basis? A RATE NEAR ZERO AND A RATE NEAR TOTAL BOTH MEAN THE REVIEW IS NOT FUNCTIONING accountability rests with a named person inside the institution — never with a model, a vendor or a committee
A reviewer with only a score, no time, or no authority to overturn is documentation, not control.

Decisions That Carry a Duty to Explain

Some financial decisions carry obligations attached to the decision itself, and those survive automation entirely. Where the duty arises varies, so name the regime before assuming it. In US consumer credit, ECOA and Regulation B require specific principal reasons on an adverse action; in the EU, data protection law gives rights to human intervention and to contest a solely automated decision with significant effects. "The model said no" satisfies neither, and elsewhere the duty may be differently drawn or absent. Recommendations to retail clients engage the local advice regime: MiFID II suitability in the EU, Regulation Best Interest and the adviser fiduciary standard in the US. Account freezes, exits, claim declines and suspicious-activity filings land on an identifiable person.

  • Explanation and appeal duties attach to the decision, not to the technology that produced it
  • Name the regime: ECOA and Regulation B in US consumer credit, GDPR rights in the EU, neither by default elsewhere
  • Retail recommendations engage MiFID II suitability in the EU and Regulation Best Interest in the US
  • Regulatory tiering follows the same logic — module 2 sets out the EU AI Act's high-risk category and its carve-outs

Review That Is Real Rather Than a Rubber Stamp

A human in the loop who approves almost everything that arrives is documentation, not control. Meaningful review needs four things the design usually omits. The reviewer must have what they need in order to disagree — the inputs and the reasons, not only a score. They must have time proportionate to the decision. They must have authority to overturn without a personal cost for doing it. And override rates must be monitored, because a rate near zero and a rate near total are both evidence that review is not functioning. Automation bias — deference to a confident system — is well documented, and it is a design problem rather than something training fixes. Securing AI Systems goes further into where approval gates should sit so that they are read rather than clicked through.

  • A reviewer given only a score cannot disagree in principle, whatever the process says
  • Time proportionate to the decision, and authority to overturn without personal consequence
  • Monitor override rates: near-zero and near-total both mean the review step is not working
  • Automation bias is a design problem — telling people to be sceptical does not fix an interface

Who Is Accountable When the Model Is Wrong

The answer that holds up under examination is that accountability sits with a named person inside the institution — not with the model, the vendor, or the committee that approved it. Supervisory expectations across jurisdictions push the same way: an identified owner for the model, an identified owner for the process it sits inside, and a clear line to a senior individual answerable for the outcome. Module 2 sets out why buying rather than building does not move that line. What is worth adding here is that "named" has to mean a person rather than a role that is presently vacant, and that the person has to be reachable on the day the complaint arrives. The practical test is simple. When a supervisor asks who decided, is there a name, and could that person describe the basis of the decision?

  • Accountability rests with a named person inside the institution, never with a model or a vendor
  • Expect an owner for the model, an owner for the process, and a line to a senior individual
  • A named role with nobody currently in it is an unowned model — check the name, not the org chart
  • The test: when a supervisor asks who decided, is there a name and can they explain the basis

Prefer slides, quizzes, and saved progress? Read this lesson in the library — free, no sign-up.