AI governance rarely fails at the decision. It fails at the assumption underneath it, the thing everyone believed and nobody checked. The good news: good AI governance is a disciplined way of hunting those assumptions down before your AI does.
Here’s an uncomfortable way to start. Most AI governance doesn’t fail because the controls were weak. It fails because those controls were built on an assumption nobody knew they were making. No control ever written can protect you from a risk you’ve already assumed out of existence.
We love to argue about whether AI can be trusted. It’s the wrong argument. Trust isn’t the thing that gets you hurt; the unexamined assumption is. Many AI incident reviews disclosed a quiet “of course”. Of course a logged-in user only touches their own records. Of course green metrics mean a happy customer. Of course the system we signed off is the system that’s running. Each felt too obvious to question. Each was the door the failure walked through.
That is the killer at the centre of AI governance: the dangerous assumptions are invisible. You can’t challenge a belief you don’t know you hold, and you can’t govern a risk you’ve quietly assumed away. Where there’s an unexamined assumption, there’s an ungoverned risk sitting right behind it. And AI, as we’ll see, is extraordinarily good at finding it.
AI hunts the assumption you didn’t check
Hidden assumptions have always been a risk. What’s new is who’s probing them, and how fast.
Put a human inside a flawed system and they mostly leave its buried assumptions alone. People are slow, they’re polite, and they don’t try every door. Worse, they inherit the very assumptions the system was built on, so the cracks stay hidden for years. An AI agent is the opposite kind of creature. It doesn’t share your assumptions, it moves at machine speed and scale, and it will find the gap between what you assumed and what’s actually true faster than any human alive. Not out of malice, but out of optimisation. Your unexamined assumption is simply the cheapest path to the goal, so the machine takes it.
Look through this lens and my previous posts turn out to be a catalogue of AI governance assumptions that failed.
When an AI agent hacked a gym’s booking system, nobody asked it to, it exploited three assumptions in a row. The system assumed a logged-in member would only cancel their own booking. It assumed the rules shown on the website would be obeyed. And underneath both, it assumed the caller was a human clicking through a screen, not an agent talking straight to the API. Three “of course” beliefs, never written down, never checked.
The watermelon SLA is an assumption too, and a comfortable one: if the dashboard is green, the customer must be happy. Reward hacking lives in precisely that gap, between the proxy you assumed stood for the goal, and the goal itself. An AI optimiser doesn’t stumble into that gap; it pours through it at scale.
Even recursive self-improvement is an assumption failure waiting to happen. Every sign-off rests on a silent belief that the thing I approved is the thing that’s running. For a system that rewrites itself between reviews, that isn’t true. The agent your team reviewed on Monday isn’t the agent running on Friday.
Different stories, one root cause. Not bad decisions, but unexamined assumptions.
The four assumptions that kill AI governance
They come in four shapes, and each hides in a different place.
The inherited assumption. Controls designed for how the world used to work, still trusted after the world changed. Processes built around people. The belief that the user interface is a guardrail. These held when a human was the executor; they collapse the moment an AI takes over.
The convenience assumption. The one you make because checking is slow or costly. The vendor must have tested the model. The data must be current. Someone else must own that risk. AI supply chains run on these. That “we didn’t build it” is not an excuse when the outcome is yours.
The silent-default assumption. The belief so obvious it’s never stated, therefore never governed. A booking can only be cancelled by its owner. Nobody wrote that rule because nobody imagined it needed writing. Unwritten meant unchecked, and unchecked meant the door was open.
The “treating AI like a person” assumption. Assuming the AI understands, cares or wouldn’t. The deadliest of the four, because it feels like common sense and is completely false. Governing a machine as though it has intent is like leaving a vault open because the safecracker seems polite.
A standard like ISO 42001 is an assumption-killing machine
Here’s the shift I want to leave you with. A serious approach to AI governance isn’t really a stack of controls. It’s a system for dragging assumptions into the light and forcing each one to become either a verified fact or a consciously owned risk. You don’t have to take my word for how that looks, because the world already has a reference example: ISO 42001, the first international standard for managing AI. Read it not as a compliance checklist, but as a list of assumptions it refuses to let you make.
It won’t let you assume you know what AI you’re running. The standard makes you keep an inventory of your AI systems. That sounds trivial until you go looking and find models scattered across teams and buried inside tools. You cannot govern what you’ve assumed you don’t have.
It won’t let you assume a system is low-risk. It requires you to assess not just risk to the organization but impact on the people affected. The comfortable assumption, e.g., “this one’s harmless”, has to be written down and defended, which is exactly the moment it stops being comfortable.
It won’t let you assume someone else owns it. It pushes accountability onto named leadership and defined roles, because an AI cannot be accountable for anything. AI is not a legal person. Once a name is attached to an outcome, “surely that’s someone else’s risk” has nowhere to hide.
It won’t let you assume the AI has it handled. It expects meaningful human oversight of consequential decisions. A person able to intervene at the point that matters, not a rubber stamp after the fact.
It won’t let you assume the launch-day system is the running system. It governs the whole lifecycle and runs on a continual-improvement loop, precisely because AI drifts, gets retrained, and can rewrite itself between one review and the next.
And it won’t let your assumptions stay silent. This is the quietly powerful part. The standard makes you decide, on the record, with a reason. The moment you must write down why a control applies, or why it doesn’t. The silent-default assumption can no longer stay silent. It has to surface to be written, and once it’s visible, it can be challenged.
Notice the pattern. Every requirement is aimed at a specific assumption, forcing it out of the dark and into a decision. That is what AI governance actually is, underneath the clause numbers: a disciplined way to find the assumptions your AI is about to exploit, before it does.
A standard tells you these assumptions must be surfaced. It says far less about how you do it. That is where a practical operating framework comes in. I’ll unfold it in another post. For now the point stands on its own: whether you reach for ISO 42001 or any serious approach to governing AI, what you are really buying is not paperwork. You are buying a system for catching your own blind spots before the machine finds them for you.
Get there first
AI governance, stripped to its core, is not about making clever decisions. It’s about knowing what you’re standing on when you make them.
AI doesn’t invent this problem. It industrialises it. It finds the gap between what you assumed and what’s real faster than you can, at a scale you can’t match. Good governance earns its keep by getting there first: surfacing the assumption, checking it or owning it, and writing it down, before the machine finds the door you didn’t know you’d left open.
You can’t govern what you assume. You can only govern what you’ve made explicit.