Skill.re
AI-Powered Business Intelligence for Tech Leaders
The Metrics Trap: When Numbers Don't Align
Three teams at a Series C fintech gave the board three different churn numbers at the same quarterly review. The CFO asked which one was real. Nobody knew.
Sales counted monthly churn as customers_lost / customers_start_of_month. Product counted it as customers_lost / average_customers. Finance counted it differently, excluding refunds. All three numbers were technically correct. None were wrong. All were measuring something different.
The board lost confidence. Not in the numbers, but in the company's ability to understand itself. And that's the real problem: without shared definitions and transparent calculations, data becomes a weapon in internal debates, not a tool for decision-making.
This is where business intelligence starts. Not with dashboards. Not with fancy visualizations. With alignment on what you're measuring, how you're measuring it, and why it matters.
AI accelerates this process dramatically. It can suggest metrics aligned with industry standards, generate the queries to calculate them, build the dashboards to display them, and even flag anomalies humans would miss. But AI can't tell you which metric matters for your business. That's judgment. That's leadership.
Building Your Metrics Framework
The best metrics drive decisions. They're measurable, aligned with business outcomes, and lead to action. Not every metric is useful. Some measure output (how many times did you deploy code). Others measure outcome (did customers notice the improvement)). You need both, but outcomes matter more.
Start by thinking about the decisions your team makes. For a SaaS company, the VP of Product decides: Which features should we build next? Which customer segment should we focus on? Should we raise prices? These questions need different metrics. Feature adoption tells you what users want. CAC and LTV tell you profitability of different segments. Churn trends tell you retention health.
McKinsey research on high-performing organizations found that companies with mature BI practices make 5-6x faster decisions and achieve 15-20% better financial outcomes than peers. The gap comes from having the right metrics ready, not from having more data.
A practical framework organizes metrics by decision-maker. Here's what each audience typically needs:
Executive Level (CEO, Board) needs business outcome metrics: Monthly Recurring Revenue (predictable revenue), Customer Acquisition Cost (capital efficiency), Customer Lifetime Value (long-term viability), churn rate (retention), and net retention (growth from existing customers). These metrics answer: Is the business viable? Is it growing?
Product Level (VP Product, Designers, PMs) needs engagement and behavior metrics: Daily/Weekly Active Users (engagement), feature adoption (what users actually use), session frequency (how often they return), feature completion rates (where users abandon), and user cohort trends (which user segments stay vs. leave). These answer: Are users engaged? What features matter? Why do users leave?
Engineering Level (VP Engineering, Tech Leads) needs velocity and reliability metrics: Deployment frequency (how often you ship), lead time for changes (code to production), change failure rate (how often deploys break things), Mean Time To Recovery (how fast you fix problems), test coverage (what percentage of code is tested), build time (CI/CD speed), error rate (request failures), and system availability (uptime). These answer: Are we shipping fast? Are we reliable? Are we slowing down?
The nuance: the same team might track different metrics for different purposes. Engineering needs deployment frequency for a velocity conversation, but also needs deployment confidence (change failure rate) for a quality conversation. These sometimes conflict. High frequency without confidence is just shipping bugs faster. This is where judgment matters. You choose what you're optimizing for.
Dashboards That Tell Stories
Metrics are invisible if nobody sees them. Dashboards solve that. A good dashboard shows the right metrics to the right audience at the right frequency, with enough context to act.
The problem most companies hit: too many metrics on one dashboard. Engineers see a dashboard with 30 metrics, half of which are irrelevant. They ignore it and go back to ad-hoc queries. The dashboard becomes a monument to ambition, not a tool for decisions.
Design dashboards around decision needs, not metric completeness. For an engineering leadership dashboard, you might show: (1) Deployment frequency as a bar chart comparing this week to last week, shows velocity and whether it's increasing or decreasing. (2) Incident count as a line chart over 8 weeks, shows stability and trends. (3) Average MTTR as a gauge with the SLA threshold marked, shows recovery speed and whether you're meeting commitments. (4) Test coverage as a gauge, shows code safety. (5) Build time as a trend, shows whether the system is getting faster or slower.
That's five metrics for one audience. Focused. Actionable. A VP Engineering can look at this dashboard once per week and know: Are we shipping fast? Are we stable? Are we getting better or worse?
AI helps with dashboard design by suggesting visualizations matched to your metrics. Line charts for trends (is this going up or down?). Bar charts for comparisons (this week vs. last week, this month vs. last month). Gauges for current status (are we at SLA?). Heatmaps for time-based patterns (are errors concentrated in certain hours?). The AI doesn't know your business, but it knows visual design patterns that make data readable.
Modern BI stacks use tools like Looker, Metabase, or Grafana for visualization, combined with dbt for metric definition, and Snowflake or BigQuery as the data warehouse. This stack is appealing because it separates concerns: dbt handles metric definitions (reusable, version-controlled), the warehouse handles scale, and the visualization tool focuses on making insights accessible. AI helps at every layer, suggesting metric definitions, writing dbt queries, designing dashboards, even answering questions in natural language.
The Right Tool Matters: Grafana is excellent for infrastructure and real-time metrics (monitoring). Looker/Metabase excel at self-service BI and complex business questions. Spreadsheets are fine for tiny teams, terrible for scale. Choose based on your data volume, query complexity, and audience self-service needs. Don't choose based on brand names. Choose based on what your team will actually use.
Metric Calculation: Where AI Shines
Writing correct SQL for metrics is non-trivial. Simple metrics (Daily Active Users = unique user count per day) are straightforward. Complex metrics (monthly churn, which requires cohort tracking over months, accounting for trial periods and usage history) are error-prone.
AI can generate SQL that handles complexity. You prompt: "Generate SQL for monthly churn: customers who had activity in month N-1 but not in month N. Exclude trial users." The AI generates a query with proper window functions, case statements, and edge case handling. Then you validate it by running it against sample data and checking the output makes sense.
This is a huge time saver for data engineers. A query that might take 2 hours to write and test takes 10 minutes with AI, especially for variations. "Generate the same metric but only for customers in the US." "Generate it for each customer segment." The AI generates variations instantly.
The critical step: humans validate. An engineer runs the AI-generated query, spot-checks the output (does 3% monthly churn for this cohort seem right?), and reviews the SQL logic (is the window function correct?). This takes 10 minutes instead of 2 hours. The validation is non-negotiable, AI has bugs, makes assumptions, and sometimes gets logic wrong. Your judgment catches those mistakes before they become decisions.
A practical workflow: (1) AI generates 10 metric definitions with suggested SQL. (2) Engineer validates 3 of them by hand (spot-check). (3) If those look good, trust the others with scheduled validation (run them weekly, alert if output looks anomalous). (4) Store metric definitions in version control, your dbt project or metrics layer. (5) Update incrementally. Start with 10 metrics, add 5 more next month once the first 10 are stable.
Trend Analysis: Spotting What Matters
A metric is static. A trend is dynamic. Deployment frequency of 5 per day tells you current speed. Deployment frequency increasing from 2 to 5 per day tells you you're accelerating. The second is more useful.
Humans are bad at spotting trends in raw numbers. Errors per minute over 30 days is 100 data points. Is it increasing or decreasing? Is last week's spike meaningful or noise? AI answers these questions by fitting trend lines, calculating moving averages, and flagging anomalies.
Spotify's DORA metrics program (Deployment Frequency, Lead Time, Change Failure Rate, Mean Time To Recovery) started with data but evolved to trend analysis. They don't just report current MTTR. They track MTTR by team by week, identify whose MTTR is degrading, and alert when a team drops below their baseline. This requires statistical analysis, AI's wheelhouse.
Airbnb's Minerva metrics platform does similar work at product scale. They define 50+ business metrics, track them in real-time, and use anomaly detection to alert product and engineering teams when something unexpected happens. "Feature adoption for this cohort dropped 15% overnight. What changed?" The anomaly alert gets the team investigating in minutes instead of discovering it in weekly reporting.
The pattern: automated detection, human verification. AI flags that something is unusual. Humans determine whether it's a problem. A spike in error rate is interesting. It's a problem if that spike caused customer impact (verify with monitoring). It's not a problem if it's a test deployment that rolled back in 2 minutes (context matters).
AI and Natural Language: The UX Revolution
Traditional BI requires SQL knowledge or BI tool expertise. A product manager asking "which customer segment has the highest LTV this quarter?" needs a data engineer to write the query. This slows decision-making.
AI-powered natural language querying changes this. You ask the BI system in plain English: "Which segments have LTV growing and churn declining?" The AI translates to SQL, runs the query, and returns: "Mid-market and enterprise segments both show healthy growth in LTV (up 12% and 8% respectively) with declining churn (down from 2.1% to 1.8% and 1.2% to 0.9%)."
This isn't science fiction. Looker has this. Metabase has this. Tabular AI systems exist. The technology works for well-defined metrics in clean databases. For messy data or unusual questions, it fails. But 80% of questions are routine variations. This month vs. last month, this segment vs. that segment, this week vs. last week. AI handles those instantly.
The killer feature: self-service. A PM or finance team member gets their own answer instead of waiting for engineering. This scales BI across the organization. Engineering can't be the bottleneck for every data question.
Common BI Pitfalls and How to Avoid Them
Pitfall 1: Measuring Output Instead of Outcome
You measure deployment frequency. The team optimizes for high numbers. Deploys go from 2 per week to 10 per week. But 60% of those deployments are hotfixes. You're shipping quickly and breaking things quickly. Velocity went up; reliability went down. The metric optimized the wrong thing.
The fix: pair metrics. Deployment frequency AND change failure rate. DORA does this explicitly. You can't look good on just one. This prevents gaming. A team can't optimize for frequency alone. They have to maintain reliability.
Pitfall 2: Stale Dashboards Nobody Uses
A company spends 8 weeks building an elaborate 50-metric dashboard. It launches. It's updated monthly. Teams continue using spreadsheets because the dashboard is always 3 weeks out of date. The $50k investment delivers zero value.
The fix: start small and live. Five metrics, updated daily. Make the dashboard so useful teams prefer it to spreadsheets. Add metrics incrementally. A dashboard 80% of teams use is more valuable than a dashboard 0% of teams use, no matter how comprehensive.
Pitfall 3: Metric Ambiguity and Gaming
Different teams define "customer churn" differently. Sales counts refund requests. Product counts account cancellations. Finance counts churn only after the customer doesn't renew. Three definitions, three different numbers. Leadership doesn't trust any of them.
The fix: lock down definitions before building anything. Write metric definitions in plain language: "Monthly churn = (customers at start of month - customers at end of month) / customers at start of month, including trial accounts, measured by subscription status change." Generate SQL from that definition and version control it. If definition changes, you have a record. When someone asks "why is this different from last month?" you have a clear answer.
Pitfall 4: Vanity Metrics
A B2B SaaS company optimized for "trial signups." They launched viral marketing. Trial signups tripled. Conversion to paid customers stayed flat. They spent $500k on users who never bought anything.
The fix: measure outcomes that drive revenue. Not signups (output). Paid conversions, revenue, LTV (outcomes). Not growth at any cost. Profitable growth.
Case Study: From Chaos to Insight, BI Transformation at a Series B SaaS Company
The Situation: A mid-market SaaS platform with $8M ARR faced a classic problem. Metrics were scattered across spreadsheets, custom dashboards, and tribal knowledge. The VP Engineering asked "what's our deployment frequency?" and got three different answers from three different teams. The VP Product asked "what's our feature adoption rate?" and discovered Sales, Product, and Analytics all calculated it differently. The board asked "what's our customer churn?" and nobody had a confident answer.
Different teams were making decisions based on different data. The CEO would approve conflicting product strategies. Engineering would optimize for metrics Product didn't believe. Finance would report numbers Engineering couldn't explain. This wasn't dishonesty. This was chaos.
The Challenge: The company had the data. Customer database with transaction history. Event logs with user behavior. Billing system with subscription status. But no unified metrics layer. Building a unified BI system normally takes months and costs six figures. The company needed something faster.
The Approach: The CTO allocated a 12-week project to build unified BI using AI to accelerate. The plan: define core metrics, use AI to generate SQL, build three focused dashboards, and deploy with automated trend analysis.
Phase 1 (Weeks 1-3): Metric Definition
The team used AI to brainstorm 50+ potential metrics across business, product, and engineering. They filtered to 15 core metrics based on: (1) Drives a key decision. (2) Feasible to calculate from existing data. (3) Updates daily or weekly. Final list:
Business: Monthly Recurring Revenue, monthly churn rate, customer acquisition cost, customer lifetime value, net retention rate.
Product: Daily active users, feature adoption (top 5 features), weekly session frequency, feature completion rate, weekly cohort retention.
Engineering: Deployment frequency, lead time for changes, change failure rate, Mean Time To Recovery, test coverage, error rate, system availability.
For each metric, they documented the definition in plain English, decided on calculation methodology (cohort vs. rolling window, trial account inclusion/exclusion), and assigned an owner. Then, they used AI to generate SQL for each metric and had a senior data engineer validate a sample (3 out of 15) by running queries and spot-checking output.
Effort: 1 engineer, 3 weeks. Cost: $15k. This approach was 50% faster than writing all SQL manually.
Phase 2 (Weeks 4-8): Dashboard Design and Implementation
They built three focused dashboards: Executive (Board-facing), Product (PM and design team), and Engineering (VP Engineering and tech leads).
Executive Dashboard: MRR, churn rate, CAC, LTV, net retention. Updated weekly. This is the "is the business healthy?" view. Simple. Clear. Stakeholder-friendly.
Product Dashboard: Daily active users (trend), feature adoption matrix, cohort retention (new users from each week, how many return), and user segment trends. Updated daily. Supports: Is adoption growing? Are different segments engaging? Where are users dropping?
Engineering Dashboard: Deployment frequency (trend), incident rate (trend), average MTTR (gauge + trend), test coverage (gauge), build time (trend), error rate (trend), system availability (gauge). Updated real-time for errors/availability, daily rollup for deployment metrics. Supports: Are we shipping fast and reliably? What's degrading?
For visualization, AI suggested: line charts for trends (deployment frequency over time), bar charts for comparisons (this week vs. last week), gauges for current status (are we at SLA?), and heatmaps for time-of-day patterns (when do errors spike?).
Implementation in Looker took 3 weeks with one engineer.
Effort: 1 engineer, 4 weeks. Cost: $20k (includes tool licensing and data pipeline setup).
Phase 3 (Weeks 9-12): Insights and Iteration
Once dashboards were live, they started analyzing trends and anomalies:
Discovery 1: Churn was 3.2% overall, but concentrated in small customers ($5k/month) had <1% churn. This shifted retention strategy: stop trying to save small customers; invest in mid-market expansion and land-and-expand motions.
Discovery 2: Feature adoption was bimodal. 40% of users used 6+ features (high engagement). 60% used 2-3 features (core only). This suggested two different users segments needing different experiences and upsell approaches. Product started building "power user" features separately from "casual user" flows.
Discovery 3: Deployment frequency was 3x per week (good), but change failure rate was 18% (too high). Every fifth deployment caused an incident. This triggered a quality initiative: implement staged rollouts, increase test coverage, improve on-call documentation. Within 8 weeks, failure rate dropped to 5%.
Discovery 4: Customer Acquisition Cost was $12k, Customer Lifetime Value was $80k (6.6x LTV/CAC, healthy). But LTV for new cohorts was declining quarter-over-quarter. Recent customers had shorter stay. The company investigated and found the product had weak retention for certain customer segments (SMB users who needed a simpler onboarding). This triggered a product focus on SMB experience.
Results:
Year 1 financial impact:
Consolidated 12 home-built dashboards into 3 unified dashboards. Maintenance cost: $50k/year saved.
Retention strategy shift: stopped burning retention budget on small customers, invested in high-value customer expansion. Reduced predicted churn loss by $200k annually.
Engineering quality: change failure rate dropped from 18% to 5%. This reduced incident load, on-call burden, and customer-facing outages. Estimated cost avoidance: $80k/year.
Product focus: bimodal adoption insight led to two distinct product tracks. New premium features increased mid-market customer LTV by 18% (estimated $160k incremental revenue). New simplified onboarding reduced SMB churn by 1.2 percentage points (estimated $120k retained revenue).
Total Year 1 ROI: ~$610k in value against $35k investment (17.4x return). Not including intangible benefits like faster decision-making and improved alignment across teams.
Lessons:
Start with 10-15 core metrics, not 50. Too many metrics = noise. Grow incrementally.
Dashboards should be simple enough that people use them weekly. Elaborate dashboards nobody looks at deliver zero value.
Metric definitions matter more than metric sophistication. A simple, well-defined metric beats a complex, ambiguous one.
AI generates SQL fast. Humans must validate. Always spot-check.
Focus on trends and anomalies, not absolute numbers. "Our error rate is 0.5%" is less useful than "our error rate doubled overnight."
When BI Goes Wrong: Failure Scenarios
Scenario 1: Metric Definition Gaps Lead to Decisions Nobody Trusts
A commerce company built BI dashboards showing $50M revenue monthly. The board looked at actual deposit records and saw $48M. The $2M gap was unexplained. Sales claimed it was refunds. Finance claimed it was unreconciled payments. Nobody had traced the definitions.
The dashboard was technically correct. It just measured something different from what stakeholders thought it measured. Lesson: lock down metric definitions, trace them to source data, and ensure stakeholders understand what they're looking at.
Scenario 2: Dashboards That Nobody Updates
A fintech company spent $60k building a 60-metric dashboard in Tableau. It took 12 weeks. When launched, it was updated monthly by a data team member. Teams wanted daily updates. They went back to spreadsheets updated by whoever had the latest query. The dashboard became a historical artifact, not a decision tool. Lesson: simpler, more frequently updated dashboards beat elaborate ones updated rarely. Start with 5 metrics updated daily. Add metrics slowly.
Scenario 3: Gaming the Metrics
An SaaS company measured "customer satisfaction" via post-purchase survey. Teams learned that customers who received personal calls (higher cost) gave higher ratings than those who didn't (lower cost). They optimized to call only high-value customers. Overall satisfaction scores improved. Actual customer satisfaction went down, low-value customers felt ignored.
The metric was gamed because it optimized for the wrong thing. The fix: measure satisfaction segmented by customer value. You can't optimize for one number without understanding what you're really measuring. Lesson: pair your metrics. Not just satisfaction score, but satisfaction score by customer segment.
Scenario 4: AI-Generated Insights That Don't Translate to Action
A company used AI to identify that error rate spiked on Friday afternoons. The AI flagged this as an anomaly. The team investigated: on Friday, their largest customer ran their weekly batch job, which generated 10x traffic. Of course error rate spiked. It was expected. The anomaly detection was noisy, flagging expected patterns as anomalies.
Lesson: AI generates hypotheses, humans add context. An anomaly alert is useful. Determining whether the anomaly matters requires human judgment.
What to Do Monday Morning
Start with these concrete actions:
List 5-10 metrics that matter for your team. Don't overthink this. For an engineering team: deployment frequency, incident count, MTTR, test coverage, error rate. For product: DAU, feature adoption, retention cohort, churn. Ask an AI: "What metrics should I track?" Then narrow to what actually drives your decisions.
Define each metric in writing. Not elaborate. A sentence: "Monthly churn = (customers_at_month_start - customers_at_month_end) / customers_at_month_start, updated on the 1st of each month, includes trial accounts." Version control this. When definitions change, track the change.
Generate SQL for each metric. Ask AI: "Generate SQL to calculate [metric definition]." Have someone validate by running the query and spot-checking output. Don't ship untested queries.
Build one dashboard. Start with 5 metrics. Show trends (line charts). Show current status (gauges). Update it weekly or daily depending on metric type.
Weekly, analyze your metrics. Ask: What's trending up (good or bad)? What's anomalous? What action should we take? Make one decision based on the data.
Share with stakeholders weekly. "Here's what's improving. Here's what's concerning. Here's what we're doing about it." Make data-driven decisions transparent.
Frequently Asked Questions
Q: How many metrics should we track?
A: Start with 5-10 per audience. Too many (>20) creates noise. Too few (
Q: How often should metrics update?
A: Depends on decision velocity. Critical metrics (incidents, errors, downtime) need real-time or hourly updates. Product metrics (DAU, feature adoption) need daily. Business metrics (MRR, churn, CAC) need weekly. Don't update more frequently than you can act. Daily updates on something you review weekly create noise.
Q: What are good target values for these metrics?
A: There are no universal targets. SaaS churn
Q: How do we explain metrics to non-technical stakeholders?
A: Use analogies and plain language. "Deployment frequency" = how often we ship code to production. "Change failure rate" = how often a deployment causes a problem. "Mean Time To Recovery" = how fast we fix an issue once we detect it. "Customer churn" = how many customers leave each month. Avoid jargon. A board member asking "what's our deployment frequency?" doesn't need you to say "continuous deployment cadence." Say "we ship code to production 3 times per week."
Q: Do seasonal variations mess up our metrics?
A: Yes, but you can work around it. Some metrics vary seasonally: traffic spikes on holidays, customer signups spike after product launches. Compare year-over-year (November this year vs. November last year) instead of month-to-month. Normalize for known variations. If you know Black Friday drives 3x traffic, don't alert on that spike; alert on unexpected deviations from the expected spike.
Q: If AI generates SQL and dashboards, do we still need data engineers?
A: Opposite. You need data engineers more than ever. AI generates SQL fast. But someone needs to validate it's correct, maintain data pipelines so metrics update reliably, ensure data quality, and design infrastructure so dashboards scale. Good data engineers used to spend 50% of time writing SQL from scratch. Now they spend 20% on that. They reinvest the other 30% in architecture, data quality, and scale. You get better engineers and better outcomes.
Q: What if AI's insight is wrong?
A: It often is. AI identifies patterns ("error rate spiked on Friday"). You add judgment ("did we deploy on Friday? did a customer change their load? is this noise?"). AI generates hypotheses. Humans validate them. Always question AI insights. Use data to inform decisions, not make them automatically.
Q: Isn't building BI a distraction from shipping product?
A: No. Shipping without data is the distraction. A team making a wrong product pivot based on intuition wastes months. A misallocated engineering project wastes quarters. A wrong go-to-market strategy wastes capital. Building BI properly (spending 12 weeks and $35k) prevents bad decisions (costing $200k+ in wasted effort). It's one of the highest-ROI investments you can make as a leader.
Natural Language Querying Will Reshape BI: In 5 years, most BI questions won't be answered by dashboards. They'll be answered conversationally. "Is retention better for annual vs. monthly plans?" →instant answer with confidence intervals and cohort breakdown. This requires clean metrics definitions and good data infrastructure. Build that now. You won't regret it.
Key Insight
Data-driven decisions start with shared definitions of what you're measuring. Build metrics aligned with business outcomes, not vanity metrics. Create dashboards simple enough your team uses them weekly. Use AI to generate SQL and dashboards, but use your judgment to validate and decide. The companies that win on data aren't the ones with the most data. They're the ones with the clearest metrics and fastest decision cycles.