Overview

Lesson 4.4: Exploratory Data Analysis with AI

This lesson teaches researchers how to use AI to conduct exploratory data analysis (EDA), the critical first step in data analysis where you understand your data's characteristics, distributions, and patterns before committing to a formal analysis strategy. You will learn to generate comprehensive EDA code, create publication-quality visualizations, identify unexpected patterns that should inform your analysis design, and document your EDA pipeline for reproducibility. The lesson covers univariate analysis of individual variables, bivariate and multivariate relationship analysis, distributional assumption checking, and the specific patterns in EDA output that signal problems or opportunities that warrant investigation before proceeding to confirmatory analysis.

Title

Lesson 4.4: Exploratory Data Analysis with AI

Purpose

This lesson teaches researchers how to use AI to conduct exploratory data analysis (EDA), the critical first step in data analysis where you understand your data's characteristics, distributions, and patterns. You will learn to generate EDA code, create publication-quality visualizations, identify unexpected patterns, and use EDA findings to inform subsequent analyses.

By the end of this lesson you will be able to: (1) generate a complete univariate EDA pipeline for any dataset using a structured AI prompt; (2) produce bivariate and multivariate visualizations to examine relationships between variables; (3) assess distributional assumptions relevant to your planned statistical analyses; (4) identify and investigate unexpected patterns including bimodality, non-linear relationships, and subgroup effects; and (5) produce a documented, reproducible EDA notebook that can be applied consistently to new data batches.

Core Concepts

EDA is not preliminary. It is foundational. Researchers who skip or conduct EDA superficially regularly make analytical errors that would have been visible had they looked at the data systematically first: applying parametric tests to severely non-normal distributions, missing influential outliers that drive apparent relationships, failing to detect subgroups with qualitatively different patterns, or building regression models with highly collinear predictors.

The EDA Question Framework

Systematic EDA answers a structured set of questions across three levels of analysis:

Level 1 - Univariate (each variable individually): What type is this variable (continuous, ordinal, nominal, binary)? What is its distribution shape? What are its central tendency and spread? How many missing values are there, and what proportion? Are there outliers? Does it meet the distributional assumptions required for planned analyses?

Level 2 - Bivariate (pairs of variables): Do continuous variables correlate? Do categorical variables associate? Do continuous variables differ across groups? Are relationships linear or non-linear? Are there variables that are collinear with each other?

Level 3 - Multivariate (variable sets): Are there variable clusters or factors? Are there subgroups of observations with systematically different profiles? Are there interaction patterns between predictors?

AI can generate the code to answer all three levels from a dataset description and variable list. The researcher's job is to examine the output and apply domain knowledge to interpret findings.

Distributional Analysis and Assumption Checking

Distribution shape guides analysis method choices in ways that directly affect validity. A normal distribution supports parametric methods (t-tests, ANOVA, linear regression); non-normal distributions may require transformation or non-parametric alternatives; bimodal distributions suggest the presence of distinct subgroups that should be modeled separately rather than combined.

AI can generate the standard set of distribution diagnostics: histograms with overlaid density curves, Q-Q plots comparing empirical quantiles to theoretical normal quantiles, boxplots showing spread and outliers, and formal normality tests such as Shapiro-Wilk (for small samples) or Kolmogorov-Smirnov. For non-normal variables, AI can also suggest and generate code to test common transformations (log, square root, Box-Cox), then visualize the distribution after transformation so you can assess whether it is improved.

The decision whether to transform is not a statistical one alone. It depends on the research context. Log-transforming a right-skewed income variable may be appropriate because the theoretical relationship is multiplicative. Transforming a Likert scale item to force normality is not appropriate because the transformed values are no longer interpretable. AI can generate the code and explain the statistical rationale; you supply the domain judgment.

Relationship Analysis

Beyond individual variables, EDA systematically examines pairwise and higher-order relationships. A correlation matrix with visualization (heatmap) identifies which continuous variables are strongly related, potentially collinear, or unexpectedly independent. Scatterplot matrices (pairplots) reveal whether relationships are linear or non-linear, whether outliers are influential, and whether there are subgroup patterns within the cloud of points. Mosaic plots and chi-square tests examine whether categorical variables associate. Group comparison plots (boxplots, violin plots by group) reveal whether the distribution of a continuous variable differs meaningfully across categories.

High correlations between predictors (multicollinearity) do not cause bias but inflate standard errors and make individual coefficient interpretation unstable. VIF (variance inflation factor) analysis, which AI can generate code for, quantifies multicollinearity levels. A VIF above 5-10 for any predictor warrants consideration of dimensionality reduction or predictor selection.

Pattern Recognition and Unexpected Findings

EDA frequently surfaces unexpected patterns that change the analysis plan or reveal data problems. Common examples: a bimodal distribution that suggests two distinct populations were inadvertently combined; a strong correlation between variables that should theoretically be independent, suggesting a data collection error or confound; a cluster of outliers at specific values that turn out to be a missing value code (999, -1) that was not recoded during cleaning; a non-linear relationship between predictor and outcome that a linear regression would mismodel.

AI accelerates pattern detection by generating comprehensive visualizations quickly, but pattern interpretation requires researcher judgment. When you identify an unexpected pattern, prompt AI to investigate: 'I see a bimodal distribution in the [variable] histogram. Please generate code to: (a) identify what value splits the two modes, (b) check whether the two modes correspond to any categorical variable in the dataset, (c) create separate distribution plots for each subgroup.'

Practical Applications

The following workflows illustrate how to apply AI-assisted EDA in realistic research contexts.

Workflow 1: Comprehensive EDA for a New Dataset

A researcher receives a merged dataset from three clinical sites, 320 participants, 28 variables. Before any formal analysis, she wants a comprehensive picture of the data.

Prompt: 'I have a dataset with 320 rows and 28 variables. Variable types are: [list continuous variables], [list categorical variables], [list binary variables]. Please generate a Python EDA script using pandas, matplotlib, and seaborn that produces: (1) summary statistics table (N, mean, SD, min, max, quartiles for continuous; N, frequency, proportion for categorical); (2) histograms with density overlay for all continuous variables; (3) bar charts for all categorical variables; (4) Q-Q plots for all continuous variables; (5) a heatmap correlation matrix for all continuous variables; (6) boxplots for each continuous variable by the group variable [name]; (7) a missing data heatmap showing missingness patterns across variables and rows.'

The resulting script runs in minutes and produces a multi-page PDF of visualizations. She examines each systematically, flagging any that warrant follow-up investigation.

Workflow 2: Distribution Assumption Checking Before Regression

Before running a multiple regression, a researcher needs to confirm that the outcome variable is approximately normally distributed and that the predictor-outcome relationships are linear.

Prompt: 'I plan to run a multiple linear regression predicting [outcome variable] from [list predictors]. Please generate R code that: (a) creates a histogram and Q-Q plot for the outcome variable; (b) runs a Shapiro-Wilk test on the outcome; (c) creates scatterplots of each continuous predictor against the outcome with a smoothed trend line (LOESS); (d) calculates VIF for all predictors to check multicollinearity; (e) creates residual diagnostic plots (residuals vs. fitted, Q-Q of residuals, scale-location, leverage plot) for the full regression model.'

If the Q-Q plot shows systematic deviation from the diagonal and Shapiro-Wilk rejects normality, she follows up: 'The outcome variable is right-skewed (Shapiro-Wilk W = 0.87, p < .001). Please generate code to apply and visualize log, square root, and Box-Cox transformations, and compare Q-Q plots of the original and transformed distributions side by side.'

Workflow 3: Investigating an Unexpected Correlation

EDA reveals a correlation of r = .72 between two variables that the researcher expected to be nearly independent based on theory.

Prompt: 'I found an unexpected correlation of r = .72 between [variable A] and [variable B]. Please generate code to: (a) create a scatterplot of A vs B with a linear trend line and confidence interval; (b) calculate the correlation for each study site separately using the site variable; (c) test whether the correlation is significantly different across sites using Fisher z-transformation; (d) identify any outliers (Cook's distance > 4/N) that may be driving the correlation; (e) run the correlation with and without those outliers and report the difference.'

This investigation reveals that the correlation exists primarily in one study site due to a data collection procedure difference, a finding that changes the analysis plan substantially.

Workflow 4: Subgroup Pattern Detection

A histogram of the primary outcome shows a suspicious bimodal distribution. The researcher wants to investigate whether this reflects real subgroups or a data problem.

Prompt: 'I see a bimodal distribution in the [outcome] variable histogram, with modes at approximately [value1] and [value2]. Please generate code to: (a) fit a two-component Gaussian mixture model to this variable; (b) assign each participant to their most likely component; (c) compare the two resulting groups on all other variables in the dataset using appropriate tests (t-test for continuous, chi-square for categorical); (d) create a table comparing group means and proportions.'

Saving and Reusing EDA Code

EDA code should be saved, documented, and reused rather than regenerated each time. Ask AI to structure your EDA as a function: 'Refactor this EDA code as a Python function called run_eda(df, outcome_var, group_var, output_dir) that takes a dataframe, an outcome variable name, a grouping variable name, and an output directory, and saves all plots as PNG files with descriptive names.' This reusable EDA function can be applied identically to each new data batch from a longitudinal study, ensuring consistent EDA that does not miss any variable.

Key Takeaways

EDA is foundational, not preliminary. Skipping or conducting EDA superficially leads to choosing inappropriate analyses, missing influential outliers, and failing to detect subgroups with qualitatively different patterns. The cost of thorough EDA is an hour; the cost of skipping it can be a retracted paper.

AI accelerates EDA by generating comprehensive visualizations and summary statistics in minutes rather than hours. The bottleneck shifts from code production to interpretation, which requires your domain knowledge, not AI's pattern matching.

Visualizations reveal patterns that summary statistics miss. A variable with mean 50 and SD 15 could be normally distributed, bimodal, or heavily skewed, only the histogram distinguishes these. Always visualize; never rely on summary statistics alone to understand a variable's distribution.

Unexpected findings identified in EDA often signal the most important analysis decisions: whether to transform a variable, whether to model subgroups separately, whether a correlation reflects a real relationship or a confound, whether missing data is concentrated in a way that threatens validity.

Distributional assumption checking should always precede formal analysis. Applying parametric tests to severely non-normal data or linear regression to non-linear relationships produces biased results. AI can generate the diagnostics; you interpret them against your analysis plan.

Documented and reproducible EDA code is a research asset. EDA pipelines structured as reusable functions can be applied consistently to new data batches, ensuring systematic analysis that does not miss variables as study data accumulates.

Choosing the Right Visualization for Your Data

A common EDA failure is applying a generic visualization to every variable rather than choosing the visualization that best reveals each variable's characteristics. The following guide covers the most important choices.

For a Single Continuous Variable
Use a histogram to see distribution shape, including skewness, kurtosis, bimodality, and the presence of outliers. Pair with a density curve overlay for a smoother representation of shape. Add a Q-Q plot to assess normality specifically, deviations from the diagonal indicate where the empirical distribution diverges from normal. A boxplot is better for comparing distributions across groups than for understanding a single distribution in detail.

For a Single Categorical Variable
Use a bar chart showing frequency or proportion. Order bars by frequency (highest to lowest) unless the categories have a natural order. Include a horizontal line at the expected frequency if testing for uniform distribution. Pie charts are appropriate only when there are 4 or fewer categories and you want to show part-to-whole composition; for more categories, bar charts are clearer.

For Two Continuous Variables
Use a scatterplot with a LOESS smoothing line to reveal the relationship's shape, linear, non-linear, or non-monotonic. Add a linear regression line separately to compare the linear fit to the actual relationship. Color-code points by a grouping variable to reveal whether the relationship differs between groups. If the scatterplot is overplotted (too many points to see), use a hexbin or 2D density plot instead.

For a Continuous Variable Across Groups
Use a violin plot, which shows the full distribution shape for each group. A boxplot shows the median and spread but misses bimodality or skewness within groups. Pair with individual data points (jittered) for small samples (N < 100 per group) to show the actual data rather than just the summary.

For Multiple Continuous Variables
Use a correlation heatmap for a quick overview of all pairwise correlations, with hierarchical clustering to group similar variables. Use a scatterplot matrix (pairplot in seaborn, pairs in R) for the most important 5-8 variables when you want to see both the correlation strength and the distributional shape of each relationship. Ask AI to generate both: 'Generate a seaborn pairplot for these variables [list] with kde on the diagonal, colored by [group variable], and a separate correlation heatmap with hierarchical clustering for the same variables.'