This is a post about AI accountability in business. Specifically, about what happens when you stop supervising the tools you’ve handed the keys to.
David Morrison said something that stopped me in my tracks. We’d started the call the way many business owners do. Both of us apologising for being a few minutes late, both of us trying to squeeze one more thing into a day that was already too full.
David had a decent excuse.
One of COUNT’s customers had recently migrated onto the platform and shared their experience publicly. The post took off. Suddenly there were demos being booked, conversations happening everywhere, and all the excitement that comes when something you’ve been building starts finding its audience. Everything kicking off at once.
I’d been following what David and the team at COUNT were doing for a while, so I thought it would be good to jump on a call and have a yarn. What followed wasn’t really a conversation about accounting software. It wasn’t even a conversation about AI. It became a conversation about responsibility.
And somewhere in the middle of it, David delivered the line that stayed with me long after we’d hung up.
“You can delegate the task, but you can’t delegate the accountability.”
The more I thought about it, the more I realised that might be one of the most important things business owners need to understand about AI. Because while the technology is moving incredibly fast, responsibility hasn’t moved at all.
A Bloke Who Saw This Coming
David isn’t your typical accounting software guy. His path into this world has been anything but straightforward.
Before COUNT, he spent years working across technology, software and data. He led innovation and data science teams at InternetNZ, served as CEO of Thank You Payroll, worked in a range of roles including at Provoke, Intergen and Silverstripe, and spent time in the Creative HQ accelerator ecosystem. He also runs a couple of social enterprise side operations – dogoodjobs.co.nz and impactdirectory.nz
In other words, he’d been watching this wave build long before most people knew there was a wave coming. And that completely changes how he sees the current AI boom.
For many people, it feels as though artificial intelligence appeared overnight. One minute nobody was talking about it. The next, every conference, every LinkedIn post and every software company was.
David sees something different. From his perspective, this isn’t a sudden revolution. It’s the next step in a journey that’s been happening for years.
First came rules-based automation, as businesses looked for ways to eliminate repetitive processes and simplify workflows. Then came the rise of data science, with organisations processing larger volumes of information and building increasingly sophisticated models to understand what was happening inside their businesses. Then applications started getting built on top of those capabilities. And eventually we arrived where we are today, with generative AI sitting on everyone’s desktop.
The technology has changed dramatically. The principle hasn’t.
As David put it, throughout that entire spectrum there has always been a requirement for human in the loop intervention. In fact, he believes the further we move into AI, the more important that human layer becomes. Not less important. More important.
At first that sounds backwards. Shouldn’t smarter systems require less human involvement?
Maybe. But they also create bigger consequences when things go wrong.
And that’s where the real conversation starts.
The Most Dangerous Button: AI Accountability in Business
This was the point where the discussion became very real for me.
David started explaining how systems like COUNT actually work. Behind the scenes, AI isn’t magically making decisions on its own. It has access to a collection of tools that allow it to interact with software, context and information. In COUNT’s world, that includes what is known as an MCP server which packages COUNT’s APIs in a way AI tools can easily consume.
Think of it as a toolbox. Some tools allow the AI to read information. Some allow it to write information. Some allow it to make changes inside core systems.
The important thing is that businesses can decide how much freedom those tools have. In many cases, a human can be required to approve actions before anything actually happens.
On paper, that sounds incredibly safe. And it is, at least initially.
Then human nature enters the picture.
A notification appears. Approval required. You review it. Looks fine. You click approve.
A second notification appears. Also looks fine. Approve.
Then another. And another. And another.
Eventually a thought starts creeping in. Why am I reviewing every one of these? Maybe I should just click “approve all”.
David laughed when we talked about this, because everyone knows exactly how it happens. You verified it once. It looked right. You start trusting the process.
The danger is that trust quietly turns into complacency. That’s where people stop supervising the tool and start deferring to it. The computer said it was right, therefore it must be right.
Except that’s not necessarily true.
David’s view on this is refreshingly blunt. If an AI makes a change inside your business systems, that change still happened on your watch. You don’t get to outsource accountability simply because the action was taken by software. The software doesn’t own the consequence. You do.
That’s why COUNT is actively exploring ways to maintain clear records of how actions occur. Whether a change is made directly by a person or initiated through AI, there needs to be a trail that leads back to a human decision-maker.
Because that’s reality. When something goes wrong, nobody asks the software to explain itself. Nobody sues the AI. Nobody holds the algorithm accountable. The accountability always lands back on a person.
Which brings us back to David’s line.
AI won’t go to jail for you.
We have done a lot of work in this space – if AI for your business interests you flick me a message. I would love to chat!
