Where the Pilots Die

Data Quality and Lineage Beat Model Quality

The most common cause of death is not the model. It is that the data is worse than anyone admitted. Customer records are duplicated and inconsistent across core banking, CRM and product systems. Entity resolution across legal entities and jurisdictions is unsolved. Reference data has multiple contested golden sources. Critical facts exist only inside scanned documents. On top of quality sits lineage: validation and audit both require you to show where a number came from, through which transformations, under whose ownership. A pilot on a hand-cleaned extract proves the model works on data your institution cannot actually produce at run time, which is a different and much less useful claim.

• Duplicate, inconsistent and unresolved entity data across core systems is the usual blocker
• Multiple contested golden sources for the same reference data defeat reconciliation before modelling starts
• Validation and audit both require lineage — origin, transformations and ownership, evidenced
• A pilot on a hand-cleaned extract proves nothing about production data you cannot reproduce

Legacy Systems and the Sign-Off Nobody Scoped

Two structural killers arrive late. Integration with core platforms means batch windows, change freezes, brittle interfaces and release cycles measured in quarters; getting an output in front of a user inside a system they already use is usually harder than building the capability. Then there is the governance path nobody put in the plan: model risk classification, independent validation with a real queue, second-line review, data protection assessment, records-retention treatment, procurement and third-party risk, and an approval committee that meets monthly. Teams that discover this after the demo find they have built something that cannot be deployed for reasons that were entirely foreseeable. Map the approval path before you build, not after.

• Core-platform integration, batch windows and change freezes dominate the timeline, not the modelling
• Getting output in front of a user inside a system they already use is usually the harder half
• Model classification, independent validation, data protection review and procurement all queue
• Map the full approval path before building — the sequence is knowable in advance and rarely mapped

Ownership and Vendor Concentration

Pilots are staffed by enthusiasts with protected time; production is staffed by whoever is on shift. Ask the unglamorous questions early. Who is the accountable owner in the first line? Who monitors performance, against what thresholds, and on whose desk does the alert land? Who takes the call when a user reports that the output looks wrong on an ordinary working day, and who is authorised to switch it off? Who re-validates after the provider updates the underlying model, and how would you even know they had? Behind all of those sits a dependency question: a small number of model providers and cloud hosts now sit underneath much of the industry, so an outage or a behaviour change is a correlated event rather than a firm-specific one. Module 3 takes that further.

• Name the first-line owner, the monitoring thresholds, the escalation path and who may switch it off
• A provider-side update is a change to your model; module 2 sets out what change control then requires
• Pilots that depend on one champion expire quietly when the champion changes role
• Concentration turns a supplier problem into an industry-wide correlated one — module 3 returns to it

Try It Yourself

A pre-mortem costs an hour and routinely saves a quarter. Take one AI use case your institution is actually considering and kill it on paper before anyone builds it.

Use case, described generically:
Data: which source, whose golden copy, can lineage be evidenced end to end?
Integration: which core system, whose release cycle, where does the output appear?
Governance: model tier, who validates, which committee approves, how long is the queue?
Ownership: first-line owner, monitoring thresholds, who switches it off?
Vendor: who provides the model, what happens on their update, what is the exit?
Earliest detectable signal for each failure: