Why Trustworthy AI Starts With Data Governance

Everyone talks about trustworthy AI. Fewer people can tell you what would actually make a model trustworthy to the person leaning on it to make a call. You can have a model that scores well on every metric and still hasn't helped anyone, because a good number on a page doesn't tell a human how much to trust it, or why. That gap is what this conversation is really about.

The model is rarely the problem. We are.

Ask where trustworthy AI falls over and most people point at the model. Wrong place to look, mostly. The models O'Sullivan works with are built on well-mapped data, checked by experts, and they hold up - you can tune them, test them, and stand over the result. The trouble starts on the way back to a human. Someone uses a bit of jargon they didn't mean to. Someone mishears the brief. A result gets written up for a slightly different question than the one that was asked.

"I think a lot of folks who are technically inclined like myself don't love when I say: guys, I think we might be the problem. The models will stand up to scrutiny. The messy bit is when we talk to stakeholders." - William O'Sullivan

So the trust doesn't really live in the code. It lives in the conversation around it. A model can be technically perfect and still worse than useless if it was built against a misread brief, and from the outside, that just looks like the AI getting it wrong.

Compliant isn't the same as trustworthy

The two get treated as one thing. They aren't. For O'Sullivan, compliance is the work you do before you're even allowed to start - a proper data management plan, a clear read on what you're obliged to do. Necessary, but it's the floor, not the ceiling. Trust gets built on top: how clearly the work is explained, how well the answer fits the decision someone actually has to make. Compliance gets you into the room. It doesn't make anyone believe the model.

That's also why he talks about governance before he talks about metrics. Nothing gets built until everyone agrees on what the problem is, how you'll tackle it, and what the answer is for. And when a scientist and a data scientist are in the same meeting, they often don't share a vocabulary, governance is what gives them one.

Shadow AI is an old problem moving faster

People using tools nobody signed off on isn't new. CTOs have been chasing unsanctioned software around their own companies for years. What's different now is the speed, and the fact that someone casually pasting into a chatbot can trigger a compliance obligation without having the faintest idea they've done it. Tracking that is hard, and it takes a fairly open, curious culture to even try, which not every company has. The ones most exposed, O'Sullivan reckons, are often the ones that most need to be compliant and don't know it.

His first move isn't clever. It's a register. Find out what your teams are actually using, all of them, right down to marketing drafting copy in ChatGPT.

"You have to ask people the uncomfortable question: what AI are you using? Until you take a measurement of your system, you're only ever guessing." - William O'Sullivan

And it doesn't need to be pretty. A register that works beats one that looks good, it just has to be something you can show and something you can keep up to date. The proof you're compliant is the trail that boring work leaves behind: the record, the versions, the timestamps. You don't add evidence at the end. You generate it by doing the thing properly in the first place.

If a leader changes one thing

He was asked what the single highest-leverage move would be this year. No tool, no framework. Communication. Sort out how you talk about results, stop leading with model metrics and start with the problem you're solving. In practice it's a short chain: how does this tool help a stakeholder do their job, and how does their job serve the wider strategy? Get those two links right and everyone - the technical team, the stakeholder, whoever's paying for it - ends up pointed the same way, and the work ends up being useful.

Which was the whole point. The risk was never the model. It's whether the people around it share enough language to trust what it's telling them, and whether anyone wrote down what's being used in the first place.