AI can build a confident, well-sourced-looking case for almost anything, and most organisations have no habit of checking the reasoning underneath before it lands in a decision. I've been talking to a product director at a mid-sized software company about exactly this, how it's actually showing up for people running teams day-to-day. I've heard the same thing, almost word for word, from CEOs and COOs at similar companies (and so have others), and this piece is the sum of all those conversations.
Their team is a fairly standard product org: a handful of product managers split across consumer and business customers. They used to run an analyst embedded directly in the team; now analytics works more like a shared service that PMs go to with questions, so a steady stream of ad hoc questions lands on one central team instead. Count's Slack agent has absorbed a lot of that traffic, letting people ask a question and get a trustworthy answer without waiting on someone to run it for them.
Confidence at all costs
This was the bit that stuck with me most, because it's a failure mode we care a lot about at Count, and it came up unprompted, in the middle of a conversation about something else entirely.
AI, the director said, can build a case for almost any story you push a dataset toward, and it rarely flags its own flaws unless you explicitly tell it to look for them.
What worries them isn't fabricated numbers, which people already know to double-check. It's a growing volume of rigorous-sounding claims that are genuinely hard to substantiate.
Trusting a polished dashboard means taking it on faith that whoever built it asked the right questions, because none of that reasoning is visible, only the confident output at the end of it. When every AI-assisted conversation sounds equally fantastic, that fluency is itself a warning sign, not a reassurance.
We end up with a lot of slideware and a lot of very good-sounding conversations. But for me, the fact that every conversation sounds fantastic is a signal that something's going wrong.
We've seen the same pattern, again and again, from the CEOs and COOs I've spoken with. Several pointed to AI's confident tone as the real obstacle, more than its accuracy: getting an answer was easy, getting one worth acting on without re-verifying it from scratch was the expensive part. One founder even described an AI assistant quietly dropping a constraint or misreading context, then citing something that didn't actually support its own conclusion, meaning they had to redo, by hand, the exact analysis it was supposed to save time on.
How this looks in the boardroom
The same pattern shows up a level up, in the decks built on top of those conversations. The director described using AI to produce full slide decks, references and all, and how that veneer of sourcing still creates far too much unearned confidence, because almost nobody goes back and reviews every detail.
Their explanation: in an environment where anyone can move fast and build almost anything, there's real hidden pressure against being the person who raises doubts in the room. Teams settle for having enough to decide rather than having verified it, and end up making bad calls by accident.
What needs to change
What pushed the director toward wanting traceability, rather than just better dashboards, was a roadmap exercise: producing commercial value estimates for a set of initiatives, the kind of case that justifies shipping something next quarter. They built traceable logic pathways themselves, pulled supporting data from Count, and whilst they did it to a less rigorous standard than the analytics team would have applied, the goal was just enough to get buy-in.
This director, like most of us, wants to present rigorous work rather than getting lost in a general sea of AI slop, and wants someone else to push back on their claims with real reasons and real data, not just accept them.
Everyone just feels a little bit too confident at the moment. They've got a confidence bubble.
Their current AI setup is fragmented: several different assistants for different teams, plus a data tool available to only a subset of the analytics team. Most people chat with an assistant in isolation, then separately pull in real numbers from elsewhere, manually combining these sources themselves. That fragmentation isn't unique to one company: across the leaders I spoke with, a good workflow tended to stay locked inside whoever discovered it, rather than becoming something the wider team could reuse.
How we're thinking about this at Count: shared infrastructure sitting underneath every AI conversation, so the reasoning behind a claim is something other people can interrogate and challenge, without it reading as an attack on whoever's presenting. Other leaders I talk to regularly come up with exactly that, unprompted: a way to push a full piece of AI-assisted work into a shared space where someone else could inspect it, rerun it, and build on it, rather than starting from nothing.
Trust follows the paper trail
The director has already watched this play out informally: people trust a claim substantiated in Count noticeably more than the same claim generated entirely inside a chat, and are openly sceptical of anything presented with no references at all.
Most CEOs and COOs I talk to rated trust and auditing AI work a real, ongoing pain point, and AI-generated slop came up even more consistently, ahead of almost every other complaint. One co-founder summed up the issue as one of result integrity: needing real confidence in the numbers, the sources, and the accuracy behind them.
What they'd do with the reasoning behind a claim
They had three uses for seeing the reasoning behind any claim that they would do today:
- strengthening their own argument by surfacing weak premises before anyone else could;
- presenting with more confidence, because the audience could verify a claim themselves instead of taking it on faith; and
- running the same check on other people's presentations, not out of suspicion, but because they wanted to see the reasoning for themselves before accepting someone else's conclusion.
They're already testing something like this, a shared AI workspace meant to be the starting point whenever the team builds a presentation, rather than everyone starting fresh in their own private chat. Even so, they're not convinced it solves the real problem yet: it still feels like a black box that will tell you whatever you want to hear. Other leaders described the same frustration, that a genuinely useful AI conversation is hard to hand off to anyone else without flattening it into a summary that loses the reasoning.
Where this leaves us
What struck me most is that this product director arrived at the phrase "confidence bubble" entirely on their own, describing something we think about constantly at Count: fluent, well-referenced, individually-true-sounding output that nobody has a habit of checking before it becomes a decision. Several other leaders who never heard that phrase described the same problem independently.
None of the people I've spoken to want less AI in their workflow. But what comes up again and again was wanting a way for someone, including themselves, to attack their own conclusions with reason and data before those conclusions left the room. That's the thing we keep building toward at Count.
