Misleading AI data and the prompts that catch it.
AI can calculate a number with total accuracy and still lead you to a bad decision. In a recent training exercise, an e-commerce sales file “totaled” $174,500. But that figure included every order status: shipped, canceled, pending, and delivered. Only $43,300 was tied to orders that shipped and were paid for.
The math was right. The business question was wrong.
Key Takeaways
- AI can do the math correctly while still answering a misleading version of your question.
- A single “total” number often hides different order or transaction statuses that change what the number means.
- Asking AI to separate categories, like completed versus pending versus canceled, surfaces the real picture.
- The habit of asking “what could make this number misleading?” before accepting output applies to firm data and client data alike.
- Firms should build a habit of prompting AI to flag assumptions, not just produce a result.
Can accounting firms trust AI with financial numbers?
AI tools are very good at arithmetic. The problem isn’t the math. It’s the question behind the math.
In one of my recent AI training sessions for small business owners, I gave participants a simple e-commerce sales file to analyze. The first prompt was broad: “Analyze this file and tell me the five most important things a small business owner should know.” The AI returned a confident summary, including a total order value of $174,500.
That number sounds like revenue. It isn’t. The file contained 85 shipped orders, 76 canceled orders, 73 pending orders, and 66 delivered orders, all lumped into one total. Once the class asked the AI to separate order status and explain what the numbers represented, the real fulfilled and delivered order value came out to $43,300.
The AI hadn’t made an error. It had just answered the question it was asked, not the question that mattered.
Why does this happen with AI tools?
AI tools respond to the prompt they’re given, not the intent behind it. If you ask for “the total,” you’ll get a total, even if that total blends categories that shouldn’t be combined. This isn’t unique to sales data.
The same pattern shows up in:
- Client billing reports that mix invoiced, paid, and written-off amounts.
- Marketing metrics that combine clicks, impressions, and conversions into one vague “engagement” figure.
- Time-tracking summaries that don’t separate billable from non-billable hours.
A firm owner can review AI output quickly, without asking what’s inside the number, and walk away with a confident but wrong impression of how the business is performing.
What should a firm owner ask AI before trusting a number?
A short set of follow-up prompts catches most of these issues:
- “Before giving me recommendations, examine the data for anything that could make the numbers misleading.”
- “Separate this by status or category and explain what each group represents.”
- “Audit your previous answer. Label each conclusion as fact, interpretation, or recommendation.”
- “What additional information would you want before making a decision based on this?”
These prompts cost almost nothing in time and dramatically reduce the risk of acting on a number that sounds impressive but doesn’t reflect reality.
What does this mean for client-facing work?
If your firm is experimenting with AI for client reporting, advisory work, or internal dashboards, the same discipline applies.
A client won’t know to ask whether a number includes canceled transactions. That’s the firm’s job, whether the analysis came from a staff accountant or an AI tool.
AI can speed up the first pass of analysis, but a knowledgeable person still needs to check what’s inside the total before it goes in front of a client or a partner meeting.
FAQs
Can AI tools make math mistakes?
Basic arithmetic errors are rare with current AI tools. The more common risk is that AI answers the literal question asked, even when that question combines categories that shouldn’t be combined, producing a technically correct but misleading result.
How can I tell if an AI-generated number is misleading?
Ask the AI directly to separate the data by category or status and explain what each group represents, then ask it to flag anything that could make the total misleading if read at face value.
Should firms let AI handle client financial data without review?
No. Financial analysis, client communications, and any output tied to real dollar figures should get a careful human review before it’s used for a decision or shared with a client.

