G5 - Without a Locked Baseline, Your AI Grades Its Own Homework.
Ask a team how their AI is performing and most will compare this month to last month. That isn't a standard; it's a slow-motion surrender. The only defensible reference is a baseline captured and locked before deployment.
Where this gets hard
- Comparing to a rolling average lets gradual degradation hide inside its own trend.
- Without a locked ‘before’, every value claim and every safety claim is an estimate that dissolves under challenge.
- Model updates and threshold changes arrive unannounced, so legitimate adaptation looks identical to drift — and vice versa.
- Re-baselining happens casually, in exactly the moments when someone wants the numbers to look better.
- The first time anyone wants a baseline is during an incident — precisely when it can no longer be created.
Where to start
- Compute the baseline before deployment: performance metrics, input and output distributions, and their normal variance.
- Version it, lock it, and reference every dashboard against it — never against the window being plotted.
- Make it a gate condition: no locked baseline, no deployment. No exceptions for pilots that will ‘definitely be redone properly later’.
- Annotate every approved change as an adaptation event, so the charts can distinguish deliberate movement from decay.
- Allow re-baselining only through change control, approved and logged — recalibration is itself an audited change.
The companion consulting document on our website includes the baseline protocol, the re-baseline triggers and an adaptation-versus-drift decision guide.
Part of RMAT's 12-part series on AI governance — Governing AI with Evidence. The companion consulting document — detailed checklists, a risk table, a maturity self-assessment and a 90-day action roadmap — is available on our website.
#CEO #CIO #CTO #Governance # Risk #AI
For your most important AI system: what was its accuracy the week before launch — and could you produce that number, or would you be reconstructing it?