G6 - A Sampled Log Is a Partial Alibi.
Two quiet compromises hollow out most AI monitoring: sampling the logs to save money, and running the monitor inside the environment it monitors. Each feels pragmatic. Each fails at exactly the moment monitoring exists for.
Where this gets hard
- Record-keeping obligations are per-event; a ten percent sample cannot prove what happened in the other ninety.
- Cost pressure always arrives, and capture completeness is the line item that looks safest to cut.
- Monitoring that shares credentials and infrastructure with production can be starved, edited or disabled by the thing it watches.
- Logs that could have been altered are treated, by every sceptic who matters, as logs that were.
- The pipeline itself fails silently: a green dashboard fed by a broken collector manufactures false assurance.
Where to start
- Make one hundred percent capture of in-scope events an invariant; let cost optimisation touch analysis frequency and storage tiers instead.
- Separate the evidence plane: its own boundary, one-way data flow, no production credentials.
- Fail open toward production — monitoring outages must never take the business down, or the business will take monitoring down.
- Make evidence tamper-evident: append-only, hash-chained, externally timestamped, and verified on a schedule.
- Monitor the monitor: capture completeness, ingestion lag and verification status on every dashboard, with failures forcing red.
The companion consulting document on our website includes the evidence-plane design principles and a monitoring-health metric set.
Part of RMAT's 12-part series on AI governance — Governing AI with Evidence. The companion consulting document — detailed checklists, a risk table, a maturity self-assessment and a 90-day action roadmap — is available on our website. #CEO #CIO #CTO #AI #Risk #Governance
If your AI logged only ninety percent of its decisions, which ten percent would you be comfortable explaining away to a regulator?