Inventory, Documentation and Change Control
If It Is Not in the Inventory, It Is Not Managed
The model inventory is the spine of the whole regime, and its usual weakness is completeness rather than format. A workable inventory records every model in use, its owner, its intended purpose and limitations, its materiality tier, its validation status and date, its dependencies including data sources and any upstream models, and its monitoring arrangements. Three things routinely go missing: models embedded in purchased systems, consequential spreadsheets and end-user computing tools, and AI features adopted directly by a business team without going through technology procurement. That last category is growing fastest. Tiering by materiality is what keeps the regime proportionate; without it, everything is either over-governed or quietly ignored.
- Record owner, intended use and limits, tier, validation status, dependencies and monitoring
- Models inside purchased systems and consequential spreadsheets are the classic omissions
- Business-adopted AI features bypass technology procurement and never reach the inventory
- Materiality tiering keeps oversight proportionate — without it the regime collapses at both ends
Documentation That Must Exist Before Deployment
Documentation written after go-live is a different artefact from documentation written before, and supervisors can tell. Before deployment there should be a statement of intended use and limitations, data sources and lineage, a development record with alternatives rejected, segment-level rather than average performance, the validation report with findings and approval conditions, a monitoring plan with named thresholds, the review design, and a decommissioning plan. In the EU, the AI Act's Annex III places creditworthiness assessment and credit scoring of natural persons in the high-risk tier, carving out systems used purely to detect financial fraud, and also covers risk assessment and pricing in life and health insurance. That classification carries documentation obligations of its own.
- Intended use and stated limitations first — most misuse traces back to a vague purpose statement
- Segment-level performance, not a headline average that hides the failures that matter
- Validation findings, approval conditions, monitoring thresholds, controls, fallback and decommissioning
- In the EU, Annex III covers creditworthiness and life and health insurance pricing as high-risk, but carves out pure fraud detection
A Changed Model Is a New Model
Change control is where mature regimes are separated from decorative ones. A retrain on newer data, a reweighting, a new or altered input source, a moved cut-off, an edited prompt, a swapped retrieval corpus and a provider-side version upgrade are all changes to the thing that was approved, and each raises the question of whether the approval still stands. Materiality should drive how much re-validation follows, but the question must be asked and the answer recorded. The genuinely hard case is the change you did not initiate. Contract for advance notice of provider model changes, keep a held-out evaluation set you re-run on a schedule, and monitor for behaviour shifts, because a silent upgrade will not announce itself.
- Retrains, reweightings, new inputs, moved thresholds, edited prompts and changed corpora are all changes
- Materiality sets the depth of re-validation, but the question is always asked and the answer recorded
- The hard case is the provider-side change you did not initiate and might not notice
- Contract for change notice, keep a held-out evaluation set, and monitor for behaviour shifts
Prefer slides, quizzes, and saved progress? Read this lesson in the library — free, no sign-up.