Lessons from our workshop: Retail Hive October 2026

“It works” isn't the same as “we understand it”

Last week, Darryl Kennedy and I hosted a group of retail technology leaders at a Retail Hive workshop. The brief was simple: take AI as a given, and look at what it means for testing, governance and integrating with the systems we already run. 

We ran it as three short exercises rather than a talk, under Chatham House rules. View our findings in our Retail Hive digest.

The conversation was open, practical and, at times, a little uncomfortable in a useful way. One line came up more than once: “it works” is not the same as “we understand it”.

Here's what we heard.

AI is already in your stack, whether you built it or not

Much of the AI in retail today arrives through suppliers and platforms rather than through your own builds. That changes the question from “should we use AI?” to “do we know where it is, and what it touches?”

Across the room, AI is most established in what customers see: web, mobile, search, recommendations, product content and chat. It is growing in merchandising, pricing and fulfilment, where it is automating manual processes and helping with freight and warehouse planning. It is earliest in store and in finance, and in store it is mostly helping colleagues rather than customers.

Being in use is not the same as being trusted. Confidence varied widely, and it was lowest where the AI comes built into third-party platforms: AI that teams rely on but do not control.

The risk lives where unpredictable meets predictable

Traditional testing is deterministic. The same input gives the same output, and a test passes or fails. That still matters for the business processes most retailers run on.

AI brings non-deterministic results. Search is moving from “a pair of trousers” to “an outfit for a wedding”, where the right answer is a matter of judgement. The same is true of analytics and reporting.

The biggest risk raised in the room was where the two meet: non-deterministic outputs feeding processes that expect the same answer every time, often without anyone noticing. Finance is under particular scrutiny. Auditors are starting to ask whether AI has touched the financial data, and few teams can yet answer with confidence.

The process is not broken. The volume is.

When we mapped AI against a typical path to live, a clear pattern emerged. AI is most established in build and least established in deploy and run. Testing is lagging behind build.

In other words, AI is producing change faster than teams can test and release it. When anyone can build something in minutes, demand rises, and you cannot look at everything. The skill is deciding where people focus.

The room was clear on the answer: triage by type of change. Fast-track low-risk cosmetic changes. Slow down anything with financial or regulatory impact, and keep a human in the loop for now. Exit criteria for AI-assisted change remain an open question for most.

There is a delivery model question too. Heavily sliced delivery models sit at odds with an AI-enabled lifecycle, and that sets up a cost conversation with the CFO. And as more work passes between agents, how do you know quality is holding?

Test in proportion to how much it matters

Not everything deserves the same rigour, but the choice should be deliberate. That is the idea behind our Graded Test Approach, which matches the level of testing to what is being built, who built it and its impact.

• G0, personal utility: if it breaks, “I shrug”.
• G1, shared team tool: if it breaks, “someone pings me”.
• G2, internal dependency: if it breaks, “work stops”.
• G3, customer-facing: if it breaks, “customers notice”.
• G4, mission critical: if it breaks, “we lose money or trust”.

Where will we be in five years?

We closed by asking the room to look ahead. Four themes came through:

• Ops focus on outcomes. Business users should not need to understand APIs or platform internals to get the outcome they want.
• Translation layers. Agents and AI integration tooling will map plain-language requirements to the systems underneath.
• An AI-enabled lifecycle. Requirements, acceptance criteria and tests generated end to end, with people reviewing. Waterfall may even be worth another look for well-scoped work.
• Deciding what risk to accept. With automation covering functional checks, UAT focuses on acceptance, and the question becomes what risk we accept at deployment.

One group summed it up as an owned, AI-enabled delivery factory, with people leading change management. When coding gets cheaper, the cost does not disappear. It moves to validation, release and change management.

Keep the basics, then add

None of this replaces the fundamentals. Clear requirements held in one place, test cases and automation, a good defect process, and a process that is clear before you automate it.

What AI adds is a need for more exploratory testing, UAT that handles the edge cases, fuzzy matching that judges “good enough”, closer links between product owners, testers and developers, test environments that can recreate the model, and business rules from production fed back into test.

AI and automation make a process faster. They do not make an unclear process clear.

Questions to take back to your team

1. If your auditor asked tomorrow which of your processes AI has touched, how confident would you be in the answer?
2. Where is a non-deterministic output feeding a process that assumes the same answer every time?
3. Is your testing keeping pace with the AI in your build?
4. Which of your changes sit at G0 or G1, and which at G3 or G4? Is your test effort split the same way?
5. What level of risk are you prepared to accept at deployment, and who signs that off?

Continue the conversation

Thank you to everyone who joined us and shared their experience so openly. Relevant links: