Implementation worksheet · 6 min read
How to Review an AI-Generated Analytics Dashboard for Misleading Numbers
Check eight things before anyone acts on a generated dashboard: the date boundary (inclusive or exclusive, and in whose timezone), whether the current period is partial and labelled as such, the denominator of every rate, whether test and internal users are excluded, how nulls are treated in averages, whether the axis starts at zero, whether a percentage change is relative or absolute, and whether the numbers reconcile with one number you already trust. A generated dashboard is confident and complete-looking on first render, which removes the friction that normally makes people ask these questions. The friction was doing useful work.
Describing a dashboard and receiving one is genuinely faster than building it, and the output usually renders correct-looking charts from real data on the first attempt. What it cannot do is know which of several defensible choices you meant — whether last-7-days includes today, whether a conversion rate divides by signups or by visitors, whether a churned account should still appear. It picks one, plausibly, and does not flag that it picked.
Put it into practice
1. Check the date boundary and the timezone
Does 'last 7 days' include today, and is a day measured in UTC or the viewer's timezone? These two choices routinely move a headline number by several percent and are almost never stated on the chart. Fix the convention once, label it on the dashboard, and the weekly 'why is this different from yesterday' conversation stops.
2. Label the partial period
The current week or month is incomplete and will always look like a decline. Either exclude it or mark it visibly. This is the single most common cause of a false alarm in a weekly review — someone sees a cliff on the right edge of a chart that is only there because the week has two days in it.
3. Name the denominator of every rate
A conversion rate divided by signups and one divided by visitors are different metrics with the same name. Read each rate's denominator explicitly. If the dashboard does not state it, that is the finding — put it in the title rather than in someone's memory.
4. Confirm test and internal users are excluded
Your own team is the heaviest user of your product and behaves nothing like a customer. Generated dashboards do not know your internal accounts exist. Check for an explicit exclusion; a low filter count where you expected one is the tell.
5. Ask how nulls are handled in averages
An average that skips nulls and one that treats them as zero differ substantially, and both are defensible. The dashboard will do one silently. Decide which you want, then check which you got — this is where 'average time to first value' quietly becomes a different metric.
6. Check axes and change framing
A truncated y-axis makes a 2% move look like a collapse. A 'change' can be relative (from 4% to 5% is +25%) or absolute (+1 point) and both get written as a number with a percent sign. Both are legitimate; only one is what the reader will assume.
7. Reconcile against one number you already trust
Pick a figure from an existing report or the billing system and check the dashboard agrees. If it does not, find out why before trusting anything else on the page. One reconciled number does more for credibility than eight plausible ones.
The review checklist
Copy this structure into your review document and record your observed result for each row.
| Check | What to look for | Finding | Passed |
|---|---|---|---|
| Date boundary | inclusive/exclusive, stated | ||
| Timezone | UTC or local, stated | ||
| Partial period | excluded or clearly labelled | ||
| Denominator | named in the title of every rate | ||
| Test/internal users | explicit exclusion exists | ||
| Null handling in averages | skipped or zero, chosen deliberately | ||
| Axis origin | zero unless there is a reason | ||
| Relative vs absolute change | stated which | ||
| Reconciliation | matches one trusted number | ||
| Metric definitions | written down and linked |
A failure worth checking
Acting on the partial-period cliff. The current week appears on the chart with two days of data, the line drops off a shelf, and someone raises it in a meeting as a decline. An hour goes into investigating, then somebody notices the date. It is harmless once. What is not harmless is the second-order effect: after a few of these the team learns to discount the dashboard, and then a real decline is dismissed the same way.
Common questions
Is a generated dashboard less trustworthy than a hand-built one?
Not inherently — the same defects appear in hand-built dashboards, which is why this checklist is not specific to generated ones. The difference is friction. Building by hand forces someone to make each choice consciously; generating it means the choices were made for you and not reported.
Who should run this review?
Someone who did not specify the dashboard, ideally the person who will act on it. The specifier knows what they meant and reads the chart through that intention, which is exactly the blind spot the checklist is for.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.