When teams connect AI to company data, the promise sounds simple: sales can check conversion, support can spot rising issues, and product managers can review retention without waiting for the data team to write every query.
The delay is real, but query speed is only part of the problem. An AI system can produce valid SQL and still return the wrong business number because it selected an old table, used another team’s metric definition, or omitted an important exclusion. The answer may sound confident even when its source is not the company’s official one.
Anthropic’s internal analytics case illustrates the difference. The company says Claude now automatically handles about 95% of internal business analytics queries with roughly 95% accuracy. In an internal evaluation, the same Claude did not exceed 21% accuracy without a prepared analytics process and background knowledge. Accuracy rose above 95% after metric definitions, data sources, and analysis steps were organized into instructions—called skills—that Claude reads before answering.
You do not need to copy Anthropic’s data platform. The practical question is narrower: before AI answers company numbers, has your team fixed where it should look, how it should interpret the metric, and who must confirm the result?
That check belongs beside other workflow controls. If the project is justified as a cost saver, use When an AI Tool Says It Saves Budget, First Ask Which Cost It Replaces to identify the work and expense it actually replaces. If model usage may expand with retries or broader access, pair the rollout with The AI Model Bill Usually Runs Away Through Scope and Retries before opening the tool to more teammates.
Why natural-language access can return the wrong number
Self-service analytics lets people outside the data team check operating numbers without opening a ticket for every question. Before AI, that usually meant dashboards and reporting tools. With AI, a teammate can instead ask, “Which channel brought new customers last week?”
Company data, however, is rarely a clean dictionary. Different teams may calculate “active users” differently. Old and new tables can coexist. A customer may have different fields in the CRM—the customer relationship management system—and the billing system. A data warehouse may centrally store all of those tables without indicating which one should answer an official business question.
Three failures follow from that ambiguity. AI may choose an obsolete or test table. It may combine trial, paid, and cancelled accounts under one undefined label such as “customers.” It may also return a number without showing its source, time range, exclusions, or query path, leaving no practical way to review the answer.
Asking again does not remove these failures. The team first needs an answer map that connects common questions to approved sources and definitions.
The release check for AI analytics
Before allowing AI to answer revenue, customer, usage, or product questions, test the workflow against these four conditions. This is the primary release aid: if a row cannot be completed, keep the affected questions in a limited trial or require human confirmation.
| Release condition | Evidence to verify | Stop condition |
|---|---|---|
| Official source | Each common question points to one approved table, report, system, or semantic layer—the centrally managed layer for company metric definitions. | AI can choose among several similar tables with no rule identifying the official one. |
| Metric definition | Terms such as “active user,” “churn,” and “conversion” include their calculation, time range, and exclusions. | Managers use different formulas for the same term, or old and new definitions still coexist. |
| Query instructions | AI is told where to look first, what to do when data is missing, and when to request more context. | The instruction merely says “answer the question,” allowing AI to force an answer from similar data. |
| Review responsibility | A named owner samples high-risk answers, and required outputs include source, query conditions, and uncertainty. | No one can trace how a number was produced before it enters a meeting, deck, or decision. |
Questions affecting budgets, performance reviews, layoffs, pricing, or customer promises should still receive human confirmation. The same applies when personal data, sensitive customers, contract terms, or permission limits are involved. Vague requests such as “How did we do last week?” also need clarification about the product, region, segment, and time range before a number is produced.
In these cases, AI can still help identify possible metrics, expose missing conditions, or draft a better query. The final figure should be confirmed by the data owner, metric owner, or responsible business owner.
A small team’s first step
A small team does not need a new data platform to begin. Start with the ten operating numbers people request most often. For each question, record one official source, one metric definition, and one person responsible for resolving ambiguity.
Then require every AI-generated answer to state where it looked, the time range it used, and what it excluded. If the system cannot provide those details, treat the result as a draft for investigation rather than an input to a decision.
This small registry will reveal whether the team is ready for broader self-service access. It also gives later work—automatic queries, a semantic layer, or a dedicated analytics workflow—a stable foundation. Faster access becomes useful only when everyone can trace why a particular number is the approved answer.
AI handoff card
Use this card to test one real analytics question
Give the AI one question your team wants to make self-service, along with the proposed official source, metric definition, responsible owner, and required review evidence. Mark anything you do not yet know as “to be confirmed.”
I want to assess whether AI may answer this company analytics question:
[question]
Available inputs:
- proposed official source: [source or to be confirmed]
- metric definition, time range, and exclusions: [details or to be confirmed]
- responsible data or business owner: [owner or to be confirmed]
- evidence the answer must show: [source, query conditions, uncertainty, or other requirements]
Ask follow-up questions about any missing or ambiguous input. Then tell me whether this question is ready for self-service, suitable only for a limited trial, or should remain under human confirmation. Explain which missing evidence determines that result and give me one smallest next step. Do not calculate or invent the business number itself.
Use the response to expose missing ownership or definitions. Do not use it as a substitute for confirming the underlying data.
Everyday four-panel comic
- A teammate wants to ask AI directly for an operating number, while too many similar data cards surround the question.
- AI finds several seemingly related tables, so the team pauses before it answers.
- The team fixes official sources, metric definitions, and review checkpoints on one workflow board.
- AI follows the fixed path to prepare an answer, and a human reviewer confirms the source and conditions.
References
- Anthropic: How Anthropic enables self-service data analytics with Claude (the internal 21%/95% case figures) — https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude (2026-06-03)
- InfoQ: Anthropic Reports Claude Now Handles 95% of Internal Analytics Queries — https://www.infoq.com/news/2026/06/anthropic-claude-analytics/ (2026-06-21)
- Kenta Yamamoto (note): From 21% to Over 95% Accuracy with the Same Claude: How Anthropic’s Internal Self-Service Analytics Revealed the Importance of Data Infrastructure — https://note.com/pn_yamaken/n/nd3ae714ce54a?hl=en (2026-06-15)
- dbt Labs: Semantic Layer: What it is and when to adopt it (centralized metric definitions and business logic) — https://www.getdbt.com/blog/semantic-layer-introduction (updated 2026-05-04)
- Microsoft Learn: Lineage in Fabric (tracing data sources and upstream/downstream relationships) — https://learn.microsoft.com/en-us/fabric/governance/lineage (updated 2026-06-04)
