The AI Learning Hub Journal

Auditing an Estimate You Cannot Re-Derive

The estimate you cannot re-derivean old discipline meets a new opacity — the three-part scrutiny survives the technology, and only its texture changesTHE SAME THREE-PART SCRUTINY, A NEW TEXTURE FOR EACH PARTDATA → PIPELINEwhat fed the model, over whatperiod, with what known gaps —not just which spreadsheetASSUMPTIONS → TRAINING CHOICESwhat the model was optimisedfor, and what it wasallowed to ignoreMETHOD → BEHAVIOURhow it acts at the edges,and how anyonewould knowtogether with an honest eye on management’s tendency to lean the estimate the convenient wayMANAGEMENT’S OWN GOVERNANCE IS EVIDENCE ABOUT RELIABILITY — NOT A COURTESYvalidated before anyonerelied on itoutput monitored againstactual outcomeschanges controlled,recorded and datedreviewed by someoneindependent of the buildersa model surrounded by working governance and the same model running unwatched are different audit objectsWHEN THE TEAM’S GRIP RUNS OUTescalation, not surrender: an auditor’s specialist,engaged to interrogate the method on theteam’s behalf — whose work the team must stillunderstand well enough to stand behindWHAT IS NEVER AVAILABLE — THE SHRUG“the model is too complex to audit” is not aconclusion — it is the absence of one, and anestimate does not become exempt fromevidence by being sophisticatedGROUNDS ARE THINGS A FILE CAN HOLDthe data lineageexaminedthe validationreviewedthe monitoringevidence obtainedsensitivity to the choicesmanagement madepast outputs against whatactually happenedAUDITABLE GROUNDS FOR BELIEVING THE PROCESS — NOT A RE-DERIVATION OF THE NUMBERwhen the grip runs out, the answer is a specialist — never a shrug
Scrutiny of data, assumptions, method and governance builds grounds for belief the file can hold.

An Old Discipline Meets a New Opacity

An estimate produced by a model the client cannot fully explain feels like a new problem, and mostly is not. The estimates standards have always asked, in substance, for the same three-part scrutiny: the data the estimate rests on, the assumptions embedded in it, and the method that connects them — together with an honest eye on management's tendency to lean the estimate the convenient way. A learned model changes the texture of each part without changing the list. Data questions become pipeline questions: what fed the model, over what period, with what known gaps. Assumption questions move from named inputs on a spreadsheet to choices made in training — what the model was optimised for, and what it was allowed to ignore. Method questions become behaviour questions: how it acts at the edges, and how anyone would know. Harder, certainly. Different in kind, no.

  • The estimates discipline was already data, assumptions and method — the list survives the technology
  • Data scrutiny becomes pipeline scrutiny: what fed the model, over what period, with what gaps
  • Assumptions now live in training choices — what was optimised for, what was allowed to be ignored
  • Method questions become behaviour questions: what happens at the edges, and how anyone would know

Management's Governance Is Evidence

When the method cannot be inspected line by line, the process around it starts doing evidential work. Whether management validated the model before relying on it, whether anyone monitors its output against actual outcomes, whether changes are controlled and recorded, whether someone independent of the builders ever reviewed it — these are not courtesies. They are facts that bear directly on how much the number can be believed, which is why the risk work of the previous lesson feeds straight into the estimate work of this one. A model surrounded by working governance is a different audit object from the same model running unwatched. The vocabulary for all of this — model risk management, independent validation, explainability duties — belongs to this site's AI for Finance course, and this lesson borrows its conclusions rather than re-teaching them. The auditor's need is narrower: enough grip on the client's governance to weigh it as evidence, and the nerve to say when it is too thin to weigh.

  • Validation, monitoring, change control and independent review are facts that bear on believability
  • A governed model and the same model running unwatched are different audit objects
  • The inventory and red-flag work from the previous lesson feeds directly into the estimate response
  • The auditor needs enough grip to weigh governance as evidence — and to say when it is too thin

Grounds, Not Re-Derivation

The honest line of this lesson: you do not need to re-derive the number; you need auditable grounds for believing the process that produced it. Grounds are things a file can hold — the data lineage examined, the validation reviewed, the monitoring evidence obtained, the sensitivity of the estimate to the choices management made, the record of past model outputs against what actually happened. When the team's own grip runs out, the answer is escalation rather than surrender: an auditor's specialist, engaged in the territory of the standards on using the work of experts, who can interrogate the method on the team's behalf — and whose work the team must still understand well enough to stand behind. What is never available is the shrug. 'The model is too complex to audit' is not a conclusion; it is the absence of one, and an estimate does not become exempt from evidence by being sophisticated.

  • Not re-derivation — auditable grounds for believing the process that produced the number
  • Grounds live in the file: lineage, validation, monitoring, sensitivity, and outcomes against past outputs
  • When the team's grip runs out, bring the auditor's specialist — and still understand the work relied on
  • 'Too complex to audit' is not a conclusion; sophistication does not exempt an estimate from evidence

Prefer slides, quizzes, and saved progress? Read this lesson in the library — free, no sign-up.