AI implementation has accelerated across Australian and New Zealand organisations.
Fifty-three per cent of respondents to our latest AI research say AI has been implemented in workflows or embedded into organisational strategy. Eighty-five per cent report efficiency gains.
However, only 8% measure AI’s impact through clear metrics or KPIs.
This gap was a central theme in our parent company, Talent’s, recent webinar, From use to impact: How is AI really changing the way we work? featuring Avec’s General Manager – Data, AI & Innovation, Jack Jorgensen, joined Emily Zhang, Founder and Director of The HumAIn Impact, and Talent’s JP Browne to examine what organisations should focus on after adoption.
Here are six practical lessons for technology, data and transformation leaders.
1. Start with the business outcome
A new tool isn’t a use case.
Before introducing AI, organisations should identify a specific problem, constraint or source of friction. That could be slow customer response times, duplicated processing, inconsistent classification or a decision bottleneck.
The next question is whether AI is the right intervention.
Some tasks may be better addressed through conventional automation, process simplification or changes to system design. AI is most useful where its capabilities suit the nature of the work, not where a team is searching for somewhere to deploy it.
A strong use case should define:
- The problem being addressed
- The business outcome required
- Why AI is suitable
- Who owns the result
- What would justify continuing, scaling or stopping
2. Establish the baseline before implementation
It is difficult to demonstrate improvement without understanding the current state.
Before implementing AI, establish a baseline for the workflow. Depending on the use case, this could include processing time, operational cost, throughput, quality, error rates, rework, customer outcomes or risk exposure.
These measures should reflect how the organisation already defines good performance.
As Jack explained during the webinar, organisations do not necessarily need an entirely new category of “AI metrics”. They need to assess whether AI improves the business and operational measures that already matter.
Usage volumes, token consumption and adoption rates may support technical monitoring or cost management. On their own, they do not demonstrate value.
3. Measure the full workflow, not one task
AI can make an individual task faster while creating additional work elsewhere.
An increase in output may generate more errors, quality assurance requirements or downstream corrections. A team may save time while another team absorbs the cost.
As Jack noted:
“Saving time doesn’t necessarily mean you’re creating additional value for your team or the organisation.”
Measurement therefore needs to extend beyond the immediate user or task, and organisations should account for:
- Output quality
- Error and exception rates
- Human review
- Rework
- Downstream workload
- Technology and token costs
- Security and operational risk
- The value created by released capacity
Time saved is an input. The business value depends on what that capacity then enables.
4. Set clear ownership and accountability
Delegating work to AI doesn’t delegate accountability. The person or team accountable for the process should remain accountable for its outcome, and that ownership needs to be established before the use case moves into production.
Clear ownership determines who approves the design, monitors performance, reviews incidents and decides whether the use case should continue.
Human review points should also reflect the consequence of failure. Customer communications, financial decisions, sensitive data, regulatory obligations and high-impact operational actions require stronger controls than low-risk internal tasks.
A useful rule raised during the webinar was simple: if failure could place the organisation in the news, human oversight should remain essential.
5. Create practical guardrails and safe test environments
Seventy-nine per cent of respondents say their organisation’s AI rules are clear. Yet only 16% are very confident people understand what data can and cannot be entered into AI tools.
High-level policy is necessary, but implementation requires more specific controls.
These may include:
- Approved tools and models
- Data classification and input restrictions
- Personally identifiable information masking
- Prompt and output filtering
- Logging and monitoring
- Role-based access and permissions
- Escalation and incident-response pathways
- Defined human review points
AI pilots should also be separated from production wherever possible.
Teams need an environment in which they can test, explore and fail safely without placing live systems or production data at unnecessary risk. When a use case begins interacting with production, it should be introduced gradually, monitored closely and supported by clear rollback and shutdown controls.
6. Test models continuously
AI models, vendor policies and performance can change over time, and something new does not automatically mean it’s better for a particular workflow.
Organisations should develop repeatable test suites that compare model performance against representative inputs and agreed success criteria. This allows teams to assess changes in accuracy, quality, cost, latency and risk before updating a production use case.
The same discipline should continue after deployment.
Monitoring should identify performance degradation, unexpected behaviour, rising costs and new downstream impacts. Evidence—not implementation momentum—should determine whether an organisation scales, modifies, pauses or retires a use case.
Move beyond perceived efficiency
The objective should not be to maximise AI adoption but to improve business performance by applying AI where it genuinely supports the required outcome.
That requires disciplined use-case selection, reliable baselines, clear accountability and controls that extend from experimentation into production.
Organisations that apply this discipline will be better positioned to distinguish between AI activity that feels productive and implementation that creates measurable value.
Explore the full discussion on governance, measurement and meaningful AI impact on YouTube.
Turn AI opportunities into governed, measurable delivery with Avec.
