Episode transcript
Note: This transcript has been edited for clarity and readability while staying true to the spirit of the conversation.
Hello and welcome to Judgement Calls: Leading in the AI era.
I’m Carl Raath, and this is the weekly podcast for leaders who want to elevate their own impact and the impact of the people they lead.
Here’s this week’s idea.
Something about these tools took me longer than it should have to properly accept: they sound just as confident when they’re wrong as when they’re right.
Written down, that seems obvious. But think about what it does to you as a leader, because every instinct you have for spotting shaky work assumes the opposite.
When someone on your team isn’t sure, you can usually tell. They hedge. They slow down. The document goes vague in spots they haven’t thought through.
Twenty years of reading people has trained you to hear uncertainty.
These tools give you none of that. The answer reads the same whether it’s genuinely good or confidently made up. The radar you’ve spent a career building simply doesn’t fire here.
A few weeks back, I talked about pointing AI at your judgement rather than your busy work—getting it to interrogate your thinking instead of writing your emails. I stand by all of that.
But there’s a skill sitting underneath that practice, and I want to spend this week on it because I think it’s quietly becoming the leadership skill that matters most: knowing where to trust the tool and where not to, on the specific problem in front of you today.
The Harvard–BCG research I mentioned before has a shape to it that I glossed over last time.
When the consultants in that study worked on problems the tool is genuinely good at, they completed about 12% more tasks, worked roughly a quarter faster and produced noticeably better-quality work.
Those are real gains.
But on problems sitting just outside that capability, the people using AI were about 19 percentage points more likely to get the answer wrong.
And they didn’t catch it, because nothing in the output changed.
The researchers called this boundary a “jagged frontier”. Jagged is the important word because the line wanders. It cuts through your week in ways you can’t see from the outside, and you don’t learn it once and move on.
So, who decides which side of the frontier a problem sits on?
Honestly, nobody can decide that for you. Not the vendor, not your CIO and certainly not the tool—because the tool answers either way.
Knowing whether the answer is worth anything requires knowing the problem.
And that’s your job.
Here’s a rough rule I’ve landed on.
If a problem has been solved thousands of times in the world’s written record, you’re probably inside the frontier: summarising, structuring an argument, conducting standard analysis or drafting against a known pattern. The tool has seen it all before.
If the answer depends on things nobody ever wrote down, you’re probably outside.
What your people will actually tolerate. The history behind a relationship. The politics of your board. What your competitor is actually likely to do, as opposed to what a textbook says they should.
The tool will still answer those questions. It will just be guessing fluently in your accent.
Take something like a market-entry decision. Ask AI whether you should proceed and you’ll get back a strategy memo.
It will be well structured and plausible. It will cover all the obvious bases.
That plausibility is precisely the danger, because the things that will actually decide the outcome are the unwritten ones—and the memo can’t know them.
Take it at face value and you’ve handed the hardest call in the business to something working outside its competence. It never told you it was.
Now run it the other way.
Lay out your own reasoning for the same decision—where you’ve landed and why—and then ask the tool to attack it.
What am I treating as fixed that might not be?
What’s the strongest case that my reading of this market is wrong?
What would have to be true for this to fail within a year?
That works because you’ve moved back inside the frontier.
Generating challenges, offering alternative frames and playing the sceptic are all things it does genuinely well. The judgement about which challenge actually lands stays exactly where it belongs: with you.
One distinction within that practice is worth calling out.
Most people, when they ask AI to critique something, ask it to critique the content.
Is this analysis sound? Is this document clear?
That’s fine, but it only polishes the answer to the question you already asked.
The leaders getting real value push it at the framing instead.
Have I even defined this problem correctly?
Whose problem is this really?
What decision am I avoiding by focusing on this one?
Most of the expensive mistakes in strategy are framing mistakes. The analysis was fine. The questions were wrong.
A machine that has absorbed an enormous amount of written knowledge can be surprisingly good at shaking your framing loose because it carries none of your attachments to it.
A decision that has survived a proper attack is also one that you own differently, because you’ve heard the objections and answered them.
When those objections surface later in a board meeting or town hall, they don’t rattle you because you’ve been there already.
A decision you’ve handed to a machine doesn’t build that muscle.
So, here’s what I’d do this week.
Take one live decision you’re carrying. Before you open any tool, write down one sentence:
Does this problem sit inside the frontier or outside it—and why do I think so?
Simply forcing yourself to make that call changes how you treat whatever comes back.
If you judge it to be inside, let the tool accelerate you and enjoy the speed.
If you judge it to be outside, use it purely as an adversary. Have it argue against you, question your framing and stress your assumptions.
Just don’t let it write the answer, because whoever writes the first draft frames the decision.
Keep one test in your pocket for everything else.
Before you accept anything AI gives you on a decision that matters, ask yourself honestly:
Would I recognise it if it were wrong?
If the answer is no, you’ve found the edge of the frontier.
That edge is worth knowing, because it’s exactly where your judgement starts earning its keep.
That’s all for this week. Thank you for listening or watching, and for being the kind of authentic leader people choose to follow.
I’ll see you next week.
