Your AI Test Looks Better. Is the Work Actually Better?
Last Wednesday, we started with one annoying problem, one owner and a small test. Since then, our editions on workplace monitoring and data-centre trade-offs have returned to the same awkward question: what does the number actually prove?
A green dashboard can hide poor work. Efficient cooling is not the same as useful heat recovery. And faster drafting is not automatically a better customer response.
So this week, let’s come back to the test. You have changed something. You have a result. Now comes the part that rarely makes the product demo: deciding whether it deserves to continue.
1. Put the original question back on the table
Before opening the results, find the sentence you started with. Which recurring problem were you trying to improve? What counted as a useful outcome? Who was responsible for the decision?
If the test began with unanswered enquiries, a count of AI-generated replies is not your answer. You need to know whether the customer received a useful, accurate response and whether the next action had an owner.
It is remarkably easy to finish a trial measuring whatever the software happens to display. Convenient numbers have a habit of becoming important numbers when nobody challenges them.
Write the original question at the top of the review. Keep it there.
2. Count the checking, corrections and awkward bits
Here is a fictional example, not a customer result. A team used to spend 20 minutes preparing and checking a straightforward quote. With an AI draft, preparation takes five minutes. Excellent.
Except checking now takes 12 minutes and correcting missing assumptions takes another eight. Total: 25 minutes.
The drafting screen improved. The whole task got slower.
That does not mean the idea can never work. It means you have found the next problem to investigate. Perhaps the source information is incomplete. Perhaps the draft needs a narrower scope. Perhaps a checklist would do more than another prompt.
Count preparation, checking, corrections and handoffs consistently in both versions. Record mistakes and exceptions alongside time. A faster wrong quote is not a saving.
3. Compare work that belongs in the same comparison
Five simple enquiries on a quiet morning are not a fair comparison with five complicated ones during a staff shortage.
Use the same definition of the task, the same start and finish points and a comparable mix of work. Keep the sample size visible. Note missing records rather than quietly leaving them out.
If you only have a small sample, describe it as a small sample. “This looked promising in six straightforward cases” is a useful statement. “AI transformed our operations” requires rather more evidence.
Also ask what changed besides the tool. A named owner, a cleaner inbox or better instructions might explain part of the improvement. You are trying to improve the business, so it matters which change did the work.
4. Ask who inherited the problem
A task can leave your desk without leaving the business.
Did the sales team save time by sending operations a less complete brief? Did customer service close the ticket while the customer was still confused? Did a manager get a cleaner report because someone else spent an hour fixing the inputs?
Ask the person receiving the output what they had to do next. A short conversation can reveal something the completion counter cannot.
This follows the point in our monitoring edition: activity is a signal to interpret, not a verdict about a person. Review the workflow and its conditions. Do not turn a small experiment into an unexplained staff ranking.
5. Make a decision the evidence can support
You have three perfectly respectable choices.
Stop: the test created unacceptable errors, exceeded its boundaries or failed to produce enough value to justify continuing.
Refine: there is a plausible improvement, but one specific weakness needs another bounded test. Name the weakness and change one thing.
Repeat: the result is promising enough to check on another comparable sample before expanding.
“Roll it out everywhere” is not the compulsory fourth option.
Keep the person approving the next step named. Keep the stop condition. If the next test needs different data, permissions or customer commitments, review those before proceeding. A useful trial does not grant itself a larger job.
Your next action: a ten-minute review
Make six short notes:
- The original problem and measure.
- What happened before and during the test, including sample size.
- The checking, corrections and exceptions.
- What else changed.
- Who received the output and what they had to fix.
- Stop, refine or repeat, with an owner and review date.
If you cannot fill in a line, mark it unknown. That is a next action, not a reason to manufacture an answer.
The useful result may be that AI helped. It may be that clearer ownership helped more. It may be that the problem needs a different approach entirely.
All three beat paying monthly for an impressive demonstration you never return to.
Put the review somewhere you can return to
The AI is OK Working Playbook is A$10/month self-service access to the original Systems That Work PDF, an editable workbook and the interactive Find the Problem exercise. Define one problem, build a small test and review the result at your own pace. Download your project file and reopen it to continue; progress is stored in your file, not in your member account.
Selected resources are added when useful and ready. Consulting and individual implementation are separately scoped. Recurring monthly access; cancel any time.
Explore the A$10/month Working Playbook
The weekly newsletter stays free. Subscribe by email or read past issues, with no LinkedIn account required.
Continue the conversation
Previous editions: Your Next AI Win Starts With One Annoying Problem; Your AI Can Count Clicks. It Still Can’t Manage People.; Finding a Middle Ground on Data Centres and Waste Heat.
Start small. Keep people accountable. Check what actually changed.
Until next time, stay curious, stay scrappy, and remember: AI is OK.
Steve AGI / C51