AI excels at routine, well-defined analytical tasks.

Key points:

• Code generation: Describing an analysis in words, and AI generates code
• Exploratory data analysis (EDA): AI can generate standard EDA visualizations and summary statistics
• Standard analyses: AI can generate code for common statistical tests
• Visualization generation: AI can create publication-quality figures from data descriptions
• Debugging: AI can identify obvious coding errors and suggest fixes
• Documentation: AI can generate comments explaining code
• Reproducible scripts: AI can generate complete, runnable analysis scripts
• Multi-language code: AI can generate R, Python, MATLAB, etc.
• All of these work well because they're routine and well-defined in the training data

AI cannot make the statistical judgments that determine analysis validity.

Key points:

• Choosing appropriate tests: Knowing whether data meets assumptions for specific tests requires statistical expertise
• Checking assumptions: Validating that your data satisfies the assumptions of your chosen test is your responsibility
• Handling violations: When assumptions are violated, deciding how to proceed (transform data, use nonparametric tests, use robust methods) requires judgment
• Multiplicity correction: Understanding when you're making multiple comparisons and adjusting appropriately requires thinking
• Effect sizes and practical significance: Understanding whether a statistically significant effect is also practically meaningful is not automated
• Missing data handling: Deciding whether to listwise delete, impute, or use another approach requires understanding your data
• Confounding and mediation: These are conceptual, not computational; AI cannot reason about whether a variable is a confounder
• Interpretation: What a result means in context of your hypothesis and prior literature requires expertise
• These require you; they cannot be outsourced to AI

The type of analysis shapes what AI can appropriately do.

Key points:

• Exploratory analysis: Looking at data to understand patterns, generate hypotheses, identify problems
• Confirmatory analysis: Testing pre-specified hypotheses using pre-planned analyses
• AI is appropriate for exploratory analysis: "What patterns exist in this data?" can be answered by generated visualizations and summary statistics
• AI is riskier for confirmatory analysis: generating tests and choosing among them without pre-specification creates false positive inflation
• If you use AI to explore data and generate hypotheses, you must pre-specify confirmatory analysis before examining data
• Using AI to mine data for significant results (p-hacking or HARKing) is wrong whether AI does the mining or you do
• Good practice: use AI for exploratory data visualization, then pre-specify analyses, then run those specific analyses with verification

Code generated by AI must be verified for statistical appropriateness, not just syntax correctness.

Key points:

• Syntax verification: Does the code run without errors? (AI is usually good at this)
• Statistical verification: Does the code implement the analysis correctly? (requires your expertise)
• Assumption checking: Does the code test assumptions of the chosen test? (AI might not include this)
• Appropriateness for your data: Is this test appropriate given your specific data structure? (requires domain knowledge)
• Edge cases: Does the code handle your specific data issues (missing values, outliers, categorical coding)? (requires inspection)
• Verification checklist:
• Do I understand what this code does?
• Is this the right test for my data?
• Does my data meet assumptions for this test?
• Has the code checked assumptions?
• Are results sensible given my data?
• Have I verified results on a subset of data?

The analysis produces numbers; interpretation requires human judgment.

Key points:

• AI can generate code that produces p-values, confidence intervals, effect sizes
• AI cannot interpret what these numbers mean in context
• "p < 0.05 means reject the null hypothesis" is interpretation, not computation
• "This effect size of 0.4 is medium, suggesting practical significance" is interpretation
• "Surprisingly, anxiety decreased with social isolation in this sample, contradicting our hypothesis. This might reflect..." is interpretation
• AI-generated interpretation is especially risky: it can suggest plausible interpretations that don't fit your data or theoretical context
• Best practice: let AI generate analyses, interpret results yourself
• If AI suggests interpretations, evaluate them critically before accepting

—

Practical Research Use Cases

Scenario: You have a dataset of 500 patients with measures of depression, anxiety, and cortisol levels. You need publication-quality figures for these relationships.

Without AI:

• Write code to generate scatter plots (30 min)
• Debug code (15 min)
• Try different color schemes and sizes (30 min)
• Calculate correlations and add to figures (30 min)
• Get frustrated and ask a colleague (could be hours)
• Final figures ready: 3-4 hours minimum

With AI:

• Describe your data and ask: "Generate code for a figure showing relationships between depression, anxiety, and cortisol with correlation coefficients"
• Claude or GitHub Copilot generates code
• You review the code for correctness (5 min)
• You run it, get figures in seconds
• You make minor adjustments if needed
• Final figures ready: 15-30 minutes, verified by you

Verification:

• Does the code plot the right variables? Yes, review code
• Does it calculate correlations correctly? Check the formula and verify against manual calculation on a subset
• Are axes and labels appropriate? Yes, adjust if needed
• Are figures publication-ready? Yes, with minor aesthetic adjustments

Scenario: You have a complex dataset with 100+ variables and you're not sure what patterns exist or which variables relate to your outcome of interest.

Without AI: You spend days writing exploratory code, generating correlation matrices, scatter plots, distributions, looking for patterns.

With AI:

• Describe your dataset: "I have 200 observations of patients with 80 clinical variables and 1 outcome (remission yes/no). Generate exploratory analysis code showing distributions, correlations with outcome, and potential clustering."
• AI generates a comprehensive EDA script
• You review code for appropriateness
• You run it, getting dozens of visualizations and summary statistics in minutes
• You use these to understand your data and generate hypotheses
• You then pre-specify confirmatory analyses based on what you learned
• Time saved: 3-4 days of exploratory analysis becomes 2-3 hours

Critical point: This is appropriate exploratory use. Now you must pre-specify confirmatory analyses rather than continuing to mine the data.

Scenario: You have nested data (students within schools) and need mixed-effects models accounting for this structure.

Without AI: You look up mixed-effects modeling in R, learn the lme4 package, write code, test different random effect structures, validate assumptions, and interpret results. This is weeks of work for someone not expert in mixed models.

With AI:

• Describe: "I have students nested in schools. I want to test whether cognitive training improves math scores, accounting for school effects. Generate mixed-effects model code in R."
• Claude generates lme4 code with fixed effects for training and random intercepts for schools
• You review: Is this the right structure for my design? Do I need random slopes?
• You consult a statistician or handbook to verify this is appropriate
• You run code and check model assumptions
• You interpret results in context of your hypothesis

Verification essentials:

• Is the random effect structure appropriate? (You decide, possibly with expert help)
• Does the model converge? (Check AI's code output)
• Are assumptions met? (You check residuals)
• Is the interpretation correct? (You do the interpretation)

Scenario: You've run a complex analysis (e.g., mediation analysis) and have results (direct effect, indirect effect, confidence intervals, p-values).

Appropriate use of AI:

• Ask AI: "How should I interpret these mediation results? Direct effect p = 0.03, indirect effect 95% CI [0.01, 0.15]."
• AI suggests interpretations
• You evaluate whether they're accurate and appropriate for your context
• You incorporate useful interpretations into your paper

Inappropriate use of AI:

• Copy AI's interpretation into your paper without verification
• Risk: AI might suggest interpretations that don't fit your theory or data
• Result: your paper contains interpretations you don't understand or that are wrong

Verification:

• Do AI's interpretations align with your hypothesis?
• Do they make sense theoretically?
• Are they consistent with the numbers?
• Would you defend this interpretation to your advisor?

—

Hands-On Exercise

Objective: Learn to evaluate whether AI-generated analysis code is statistically appropriate for your data.

Steps:

1. Get a dataset: Use a public dataset from your field (or create a simple example dataset)

1. Describe an analysis: Write a plain-English description of an analysis you want to do (e.g., "Compare depression scores between treatment and control groups, adjusting for baseline depression")

1. Ask AI to generate code: Use Claude, ChatGPT, or similar to generate code implementing this analysis in R or Python

1. Verify the code systematically:

• Syntax check: Does it run without errors?
• Logic check: Does the code do what you asked?
• Statistical check: Is this the right analysis for your question?
• Assumption check: Does the code test assumptions?
• Manual verification: Run the analysis by hand on a subset and compare to AI output

1. Test your verification:

• If you requested the wrong analysis (e.g., t-test for non-normal data), did AI catch that?
• If you asked for assumption checks, did AI include them?
• What would have happened if you'd used the code without verification?

1. Reflect on limitations:

• What assumptions did AI make about your data?
• What would need different code for your real data?
• How much verification did you need to do?
• What statistical knowledge was required to verify the code?

Time required: 60-90 minutes

—

Common Mistakes and Misconceptions

Generated code can have subtle statistical errors that syntax checkers miss. A t-test that runs fine might be inappropriate for your data. Verification is always required, and statistical verification requires expertise.

AI can generate code that tests assumptions (normality tests, homogeneity of variance tests), but you must interpret the results and decide whether violations matter for your analysis. AI cannot decide whether to transform data, use nonparametric tests, or use robust methods; that's judgment.

If you use AI to mine data for significant results, you're still p-hacking even if AI does the computation. The problem is the research design (checking many comparisons without pre-specification), not the tool that does the computation.

AI can generate code for publication-quality visualizations, but you must review them for accuracy and appropriateness. Axes might be misleading. Color schemes might hide patterns. Labels might be unclear. And most importantly: does the visualization accurately represent your data?

Statistical thinking—choosing tests, understanding assumptions, interpreting results—cannot be delegated. This is where statistical expertise matters most. AI can speed execution; you must handle design and interpretation.

—

Key Takeaways

• AI excels at code generation, exploratory visualization, and routine analyses while code and analysis must be verified for statistical appropriateness
• Statistical thinking cannot be automated: choosing appropriate tests, validating assumptions, and ensuring designs don't create false positives are human responsibilities
• Verification requires examining code for statistical correctness, not just syntax; testing assumptions appropriate to your chosen analysis; and validating results make sense given your data
• Exploratory analysis can use AI heavily while confirmatory analysis must be pre-specified to avoid false positive inflation from data mining
• Interpretation cannot be outsourced: AI can generate interpretations but you must evaluate whether they're theoretically appropriate and consistent with your data
• Good practice separates exploration (AI-assisted) from confirmation (pre-specified, verified) while maintaining researcher responsibility for statistical reasoning

—

Reflection Questions

1. Your analytical workflow: What data analysis tasks consume most of your time? Which require statistical expertise versus routine execution? How could AI accelerate execution while you maintain analytical thinking?

1. Assumption checking: For analyses typical in your field, what assumptions must be checked? How would you verify that AI-generated code includes appropriate assumption checks?

1. Code verification: If you received AI-generated code for a complex analysis, what would you do to verify it's statistically appropriate for your specific research question and data?

1. Interpretation responsibility: For a complex analysis (e.g., interaction effects in regression), how would you approach interpretation? What role could AI play? What must remain your responsibility?