AI can help teams explore data, write queries, check calculations and explain findings. Using it well requires a defined business question, appropriate access to the data and a calculation someone can inspect. The useful result is an answer that survives scrutiny and helps a team decide what to do.
Consider a software company whose activation rate has fallen. Product suspects a recent onboarding change. Sales says the company is attracting different customers. Engineering wants to check whether an event stopped firing. Each explanation suggests a different response. Asking an assistant to summarize the dashboard will not resolve that disagreement on its own.
The investigation needs a common definition of activation, a comparable population and a way to distinguish an observed change from a proposed explanation. This is where AI for data analysis becomes a shared company practice.
Turn the concern into a measurable question
“Why is activation down?” contains several decisions that have not yet been made. Does activation refer to a person or an account? Does it mean opening the product, connecting a data source or completing useful work? How long does a new customer have to do it?
For this investigation, suppose the team defines activation as a new customer account connecting a data source and completing its first successful report within seven days of signup. Internal test accounts are excluded. An account counts once, even if several people use it.
The question becomes: among eligible accounts with a complete seven day observation window, how did activation change between the two signup periods, and which differences deserve investigation?
That wording gives an assistant something concrete to help with. It can locate the relevant tables, propose checks and draft a query. A product owner still needs to confirm that a successful report represents the intended customer outcome. A data owner needs to confirm which records establish it.
Microsoft's guidance on Copilot with semantic models stresses preparation of the data, its semantic model and the people using it. A semantic model describes the business meaning and relationships behind the data. Clear definitions help prevent a plausible interpretation of the wrong measure.
Write the agreed definition beside the analysis. If someone later changes the event that indicates success, the team should be able to tell whether the metric changed or customer behavior changed.
Before interpreting the chart, inspect its ingredients
A query can execute successfully while measuring the wrong thing. Start with a small data receipt: the source tables or files, their refresh times, the included periods, exclusions and the unit represented by a row.
An accounts table might contain one row per customer. An events table might contain thousands of rows per customer. Joining them and counting rows can turn an activity count into an apparent customer count. The pandas documentation on merging illustrates how repeated keys on both sides of a join can multiply the matching rows. Inspect the expected relationship before treating a combined table as a measurement.
For the activation investigation, ask the assistant to report the eligible account count before and after each major transformation. Compare the final total with a trusted existing report. Inspect several accounts that passed, failed or disappeared from the calculation. An unexplained difference is a reason to examine the query, not a reason to choose the more attractive result.
Timing requires similar care. An account created yesterday has not had seven days to activate. Comparing it with an older account treats an unfinished observation as a failure. Keep incomplete signup cohorts outside the comparison, or report them separately with their observation window clearly stated.
Missing events are also ambiguous. They may indicate that a customer did not finish, that tracking failed or that the data has not arrived yet. AI can help identify patterns in those gaps. It should not silently convert an unknown into a confirmed product failure.
The aggregate can tell a different story from the groups
Here are illustrative counts for two fully observed signup periods. Accounts are separated by whether they began through self service or an assisted sales process. Each account belongs to exactly one group.
| Signup group | Earlier period activated / eligible | Earlier rate | Later period activated / eligible | Later rate |
|---|---|---|---|---|
| Self service | 640 / 800 | 80% | 170 / 200 | 85% |
| Assisted | 80 / 200 | 40% | 360 / 800 | 45% |
| All accounts | 720 / 1,000 | 72% | 530 / 1,000 | 53% |
The overall rate falls by 19 percentage points. Yet the rate within each group improves by five percentage points. The later period contains many more assisted accounts, whose activation rate is lower in both periods.
There is no arithmetic contradiction. The combined rate depends on both the rate within each group and how many accounts belong to it. An assistant that reports only the combined decline could encourage a team to undo a useful product change.
To isolate the effect of the changing group mix in this illustration, apply the earlier population weights to the later group rates. With 80% self service and 20% assisted accounts, the later rate would be 77%: 0.8 multiplied by 85%, plus 0.2 multiplied by 45%. That is five percentage points above the earlier 72%.
This standardized comparison answers a narrower question: what would the combined rate look like under the same group weights? It does not prove that the onboarding change caused an improvement. Customers within either group could still differ in size, intent or experience. The calculation identifies where the original explanation was incomplete.
The original decline remains operationally important. More assisted customers may need different onboarding support, even if the product is performing better for comparable groups. A good investigation can preserve both findings without forcing them into one verdict.
Ask AI for competing explanations and specific tests
Once the calculations are clear, use AI to develop questions that could distinguish the remaining explanations. “Give me more insights” is less useful than asking what evidence would weaken a particular account of the change.
For this example, there are at least three useful lines of investigation.
Did the customers change?
Compare the group definitions, acquisition sources and account characteristics available at signup. Check whether the assisted category itself has changed meaning. A routing rule that reclassifies accounts can alter the apparent mix without any change in customer demand.
Keep subgroup analysis purposeful. Repeatedly slicing a small sample can produce an eye catching pattern by chance. Report the counts behind a percentage, especially when one large account or a handful of observations can change the conclusion. For decisions that depend on statistical inference, involve someone who can choose and interpret an appropriate method.
Did the measurement change?
Inspect event definitions, tracking releases, time zones and ingestion delays. If the first successful report event changed during the period, compare it with an independent operational record where possible. A list of report completions may help distinguish missing tracking from missing work.
Ask the assistant to state what it checked and what remains unresolved. It should be possible to trace a claim about missing events to the records or query that support it.
Did the experience change?
Review the product release alongside the timeline. Look at onboarding steps, errors and support findings for affected groups. An improvement after a release is a reason to investigate a causal explanation, not enough evidence to settle it.
If the team needs to estimate the effect of the onboarding change itself, consider a suitable controlled experiment or a carefully designed observational analysis. Preserve the difference between describing what happened, predicting an outcome and estimating what caused it.
Give each tool a job it can do well
A language model is useful for translating a business question into candidate queries, explaining an unfamiliar schema, suggesting checks and drafting a narrative from verified results. A query engine or analytical program should perform the actual calculation. A reviewer should inspect whether that calculation answers the agreed question.
Ask for the query or transformation with the answer, including filters and joins. Run it in an appropriate analytical environment with limited permissions. A request to explore activation should not grant permission to update production accounts, export all customer records or run an unlimited workload.
For a small approved spreadsheet, a bounded file analysis may be enough. For a recurring company metric, use maintained data models and access controls. For an investigation spanning product events, support cases and sales notes, different participants may need to contribute summaries within their permitted scope.
Keep customer identifiers and sensitive text out of shared outputs unless they are necessary and authorized. Aggregated results can still reveal information about a very small group. Decide who needs the underlying records and who only needs the reviewed finding.
Assess the assistant on known questions before relying on it for unfamiliar ones. Include a question with a duplicate join risk, a missing observation window and an ambiguous metric name. The guide to LLM evaluation explains how representative cases can expose failures that a fluent answer conceals.
Make the finding useful to the next person
The team now has something more useful than “activation fell.” It has a qualified result: the aggregate declined as the signup mix changed, both named groups improved in this illustration, and the cause of those improvements still needs investigation.
An analysis brief should preserve the question, the agreed metric, the source snapshot or refresh time, the query, the result and the unresolved explanations. Add the decision the finding supports and the conditions that would cause the team to revisit it. A chart without that surrounding record is difficult to reuse responsibly.
Swarm Spaces provide a shared place for that ongoing work. A team can keep the analysis brief, supporting Artifacts, discussion and agent Runs together. An authorized agent can continue from the saved context, while a product owner or analyst contributes the definitions and judgments the investigation needs.
For example, keep the reviewed activation calculation in the Space and create a Task to investigate assisted onboarding failures. A connected agent can use its approved analytical tools to examine the relevant records, then return an Artifact with the query, findings and source references. The next person can inspect that contribution without rebuilding the entire conversation. Swarm's connection guide explains how to connect an agent to permitted Spaces.
The outcome should also have an owner. If the team decides to improve assisted onboarding, record who will act, what result they expect and when the same measurement will be reviewed again. Avoid treating a generated recommendation as a decision that someone has accepted.
When the next report arrives
The next period should not restart the debate over what activation means. Reuse the reviewed definition, then check whether the data, customer mix or product behavior has changed enough to require a new comparison.
Measure the analysis process as well as the business outcome. Can another analyst reproduce the reported totals? Can a reviewer find the query behind a claim? Are corrections reaching the people who used the earlier answer? Is the team spending less time reconstructing definitions and more time investigating the remaining uncertainty?
These checks make AI useful beyond a single impressive answer. The company gains a body of analysis that can be questioned, corrected and continued as the business changes.