Overview

Lesson 2.3: Sample Size and Power Analysis with AI

This lesson teaches researchers how to use AI to navigate sample size and power analysis, from straightforward power calculations for simple designs to complex simulations for adaptive trials and multilevel studies. You'll learn to specify effect sizes based on literature or practical significance, understand power tradeoffs, and design studies with appropriate statistical power while managing resource constraints.

Title

Lesson 2.3: Sample Size and Power Analysis with AI

Purpose

This lesson teaches researchers how to use AI to navigate sample size and power analysis, from straightforward power calculations for simple designs to complex simulations for adaptive trials and multilevel studies. You'll learn to specify effect sizes based on literature or practical significance, understand power tradeoffs, and design studies with appropriate statistical power while managing resource constraints.

Why Statistical Power Is a Fundamental Research Design Issue

Statistical power is the probability that a study will detect a true effect of a given size, given its sample size and design. It is one of the most consequential, and most underappreciated, aspects of research design. Underpowered studies produce two kinds of scientific harm that are now well-documented in the literature on research methodology.

First, underpowered studies inflate false-negative rates. A study with 40% power will fail to detect true effects sixty percent of the time. When these null results are published, they contribute to an incorrect impression that the effect does not exist. When they are not published (which is more common), they disappear into file drawers while studies that happened to cross the significance threshold by chance get published, contributing to publication bias and an overestimated literature.

Second, when underpowered studies do produce significant results, those results are systematically biased toward overestimating effect sizes. This phenomenon, sometimes called the 'winner's curse', occurs because only the largest estimates from the sampling distribution are large enough to cross the significance threshold when power is low. The published effect size is therefore larger than the true effect size, causing replication failures when subsequent adequately-powered studies return to earth.

These two harms compound each other: the published literature overestimates effects (because only large estimates from underpowered studies are published), and replication attempts fail (because they are powered based on the inflated estimates). The result is a cascade of replication failures that has attracted substantial attention across psychology, medicine, and nutrition science.

AI is a powerful ally in power analysis because it can help researchers think through the inputs to power calculations, especially effect size specification, in ways that go beyond simply running the numbers. Getting the inputs right is the hard part; the calculation itself is mechanical.

The Fundamentals of Power Analysis

Power analysis involves four interrelated parameters, and knowing any three allows calculation of the fourth.

Effect size is the magnitude of the relationship or difference you expect to detect. It can be expressed in standardized form (Cohen's d for mean differences, r for correlations, f² for regression, η² or ω² for ANOVA) or in raw, unstandardized form. Effect size specification is the most consequential and most difficult part of power analysis.

Alpha (α) is the significance threshold, the probability of a false positive that the researcher is willing to accept. Conventionally set at .05, though some fields use .01 or .005 for more stringent criteria. There is increasing discussion in the literature about lowering the default threshold to .005 to reduce false-positive rates.

Power (1-β) is the probability of detecting a true effect of the specified size. The conventional minimum is .80, meaning an 80% chance of detection, though .90 and .95 are increasingly advocated as more appropriate given the costs of false negatives.

Sample size is the number of participants or observations. For a given effect size and alpha, increasing sample size increases power. The sample size calculation determines how many participants are needed to achieve the desired power level.

The relationship between these parameters means that specifying a larger effect size will reduce the required sample size (because larger effects are easier to detect), while specifying a more stringent alpha will increase the required sample size (because the bar for significance is higher). A researcher who needs a smaller sample can achieve this by accepting lower power, but this tradeoff should be explicit and justified, not hidden.

AI can help researchers understand these relationships intuitively, not just mechanically. When a researcher says 'I only have access to 50 participants,' AI can calculate the power that 50 participants would provide for the expected effect size, explain what that power level means for the likelihood of detecting the effect, and help the researcher decide whether to proceed with the study, seek additional participants, or modify the design to increase power within the constraint.

Specifying Effect Sizes: The Hardest Part

The effect size assumption is the input to power analysis that determines the sample size most directly, and it is also the input that is most frequently chosen poorly. Three main approaches to effect size specification exist, with very different implications for the quality of the power calculation.

Convention-based specification uses Cohen's (1988) benchmarks: small (d = 0.2), medium (d = 0.5), large (d = 0.8) for mean differences. These benchmarks were intended as rough guidelines for use when better information was unavailable, but they have been widely misused as primary inputs to power calculations without any empirical basis. When a researcher says 'we powered for a medium effect size,' they are almost always using this convention, and it frequently produces either dramatically underpowered or massively overpowered studies because medium effects in one domain may be very large or very small relative to the typical effects in another.

Literature-based specification uses effect sizes from published studies in the relevant domain. This is better than convention-based specification but has an important limitation: published effect sizes are biased upward by publication bias. If only studies with significant results are published, the distribution of published effect sizes overestimates the true distribution of effect sizes in the domain. AI can help researchers identify this bias and apply corrections, for example, calculating a lower bound for the expected effect based on the range of published effects, or using effect sizes from well-powered independent replication studies rather than original studies.

Anchored substantive specification uses the minimum effect size of practical or theoretical significance to the field. This approach asks: 'What is the smallest effect that would matter?' rather than 'What effect size do I expect to find?' If the minimum clinically important difference for the outcome of interest is d = 0.3, then the study should be powered to detect d = 0.3, not because that is the expected effect, but because anything smaller would not warrant clinical action even if true. This approach is often the most intellectually defensible and is increasingly recommended by statisticians and methodologists.

AI is particularly useful for helping researchers move from convention-based to substantive specification. By asking about the domain, the literature, and the practical implications of effects of different sizes, AI can guide researchers toward a more grounded effect size choice and help them articulate the reasoning in the methods section or grant application.

From Simple Designs to Complex Power Considerations

Power analysis for simple designs, independent samples t-test, correlation, simple ANOVA, is well-understood and can be carried out with widely available tools like G*Power. AI can help researchers use these tools effectively and understand the output.

But many research designs are more complex, and power analysis for complex designs is where AI provides its most distinctive value.

Mediation analysis requires substantially larger samples than direct effect tests. The indirect effect (the product a*b of the two path coefficients) has a non-normal sampling distribution, particularly when sample sizes are modest. Power calculations for mediation using bootstrapping methods typically require sample sizes several times larger than a direct effect test of equivalent standardized magnitude. AI can help researchers understand why mediation analysis requires more power than direct effect tests and how to estimate the required sample size through simulation-based approaches.

Multilevel or clustered designs present a power challenge because the effective sample size is smaller than the actual number of participants when observations within clusters are correlated. The degree of within-cluster similarity is captured by the intraclass correlation coefficient (ICC). When the ICC is 0.05, design effects are modest; when the ICC is 0.15 or higher (common in organizational and educational research), design effects can more than double the required sample size. AI can help researchers understand how to calculate design effects, estimate ICCs from the literature for their specific context, and incorporate these corrections into their sample size planning.

Measurement reliability affects power in ways that are frequently overlooked. When outcome measures are unreliable, the measured effect size is attenuated relative to the true effect size, a phenomenon known as attenuation due to measurement error. A study designed for a true effect of d = 0.5 may only detect an effect of d = 0.35 if the outcome measure has reliability of r = 0.7. AI can help researchers understand the attenuation formula, identify the likely reliability of their proposed measures from published validation studies, and adjust their power calculation accordingly.

Moderation analysis (testing whether the effect of X on Y depends on the level of Z) is typically powered for much smaller effects than main effects, because interaction terms are nearly always smaller in magnitude than main effects in behavioral and social science. Simmons and Simonsohn's simulations suggest that interaction effects are on average half the size of main effects, meaning studies designed to detect main effects of d = 0.5 will have very low power for interactions. AI can help researchers understand this asymmetry and plan moderation studies with appropriate sample sizes.

Sensitivity Analysis and Power Curves

A powerful complement to conventional power analysis is sensitivity analysis, asking not 'how many participants do I need to achieve 80% power?' but 'what is the minimum detectable effect size given my feasible sample size?' This reframes the question in a way that makes the tradeoff transparent.

If a researcher can realistically recruit 100 participants, a sensitivity analysis will calculate the minimum effect size detectable with 80% power at the planned sample size. If that minimum detectable effect is d = 0.4, the researcher can ask: is an effect of d = 0.4 plausible given the literature? Is an effect smaller than d = 0.4 theoretically meaningful? If the expected effect based on prior evidence is d = 0.2, the study will be massively underpowered and the researcher should either abandon the study or seek a much larger sample.

Power curves, graphs showing power as a function of sample size for a range of effect sizes, are even more informative. Instead of a single power calculation for a single assumed effect size, a power curve shows the full landscape: how power increases as sample size grows, and how that increase depends on effect size. AI can help researchers generate and interpret power curves, understand when they are approaching the flat part of the curve (where adding more participants yields diminishing power gains), and make informed decisions about sample size given resource constraints.

AI can also help researchers think through sensitivity analyses for design choices: how does power change if the ICC is 0.05 versus 0.15? How does power change if attrition is 20% versus 40%? How does power change if the correlation between the baseline covariate and the outcome is 0.3 versus 0.6? These sensitivity analyses transform power planning from a single-point calculation into an understanding of the power landscape under plausible parameter variations.

Simulation-Based Power Analysis for Complex Designs

For designs that cannot be handled by closed-form power formulas, complex multilevel designs, adaptive trials, mediation models with non-normal distributions, designs with ceiling or floor effects, simulation-based power analysis is the appropriate approach.

Simulation-based power analysis works by: generating synthetic data from a model that incorporates the assumed parameters (effect size, variance, ICC, measurement error, etc.); applying the planned analysis to the synthetic data; recording whether the result is statistically significant; and repeating this process thousands of times to estimate the proportion of replications in which the analysis reaches the correct conclusion. This proportion is the estimated power.

The major advantage of simulation is flexibility: any design can be simulated, regardless of analytical complexity. The limitation is that simulation requires the researcher to fully specify all parameters of the data-generating model, including ones that may be uncertain (variance components in multilevel models, the shape of outcome distributions).

AI can serve as a thinking partner for simulation-based power analysis by helping researchers: identify which parameters need to be specified; find estimates for those parameters from the published literature; think through the range of plausible values and conduct sensitivity analyses; and interpret the resulting power estimates. While AI may not run the simulations itself (in most contexts), it can help researchers understand what code or software (R packages like simr for multilevel power, or Monte Carlo simulation in R/Python) is appropriate and how to specify the key parameters.

Navigating Power Under Real-World Constraints

Few researchers have unlimited budgets and access. Most face hard constraints on sample size: rare populations, costly procedures, limited time, or fixed resources. The question is not 'how many participants do I ideally need?' but 'how can I maximize the scientific value of the study I can actually run?'

Several design strategies increase power without increasing sample size.

Within-subjects designs use participants as their own controls, dramatically reducing error variance and increasing statistical power relative to between-subjects designs for the same number of participants. A within-subjects study with 50 participants may have more power than a between-subjects study with 150 participants, depending on the correlation between conditions.

Using high-reliability measures reduces attenuation and increases effective power. If the measurement reliability of the current standard instrument in a domain is 0.6, but a newer instrument has reliability of 0.85, switching to the more reliable instrument will increase effective power, sometimes substantially.

Adding covariates that are strong predictors of the outcome reduces residual variance and increases power for the treatment effect. If baseline performance on the outcome explains 50% of outcome variance, including it as a covariate is equivalent to substantially increasing the sample size. Pre-post designs that include baseline as a covariate (ANCOVA) are substantially more powerful than post-only designs.

Increasing the manipulation strength or dose, using more extreme levels of the independent variable, increases the expected effect size and therefore requires fewer participants to detect. If a study compares high-intensity versus low-intensity exercise, a higher dose contrast will produce a larger effect size and require fewer participants than a comparison between high-intensity and moderate-intensity.

AI can help researchers systematically enumerate these power-increasing strategies, estimate how much each would contribute to power in the specific context, and make informed tradeoffs between design complexity and sample efficiency.

Reporting and Justifying Power Analyses

Power analyses must be reported and justified in grant applications, pre-registration documents, and methods sections. Reviewers are increasingly sophisticated about power, funders require power analyses for proposals, and methodological reviewers evaluate whether the power calculation is defensible, not just whether one was conducted.

A defensible power analysis report includes: the expected or minimum meaningful effect size with its basis (literature estimate, sensitivity analysis result, minimum clinically important difference); the significance level used (alpha) and any rationale for departing from the conventional .05; the desired power level and its justification; the specific design elements accounted for (ICC corrections, attrition adjustments, reliability adjustments); and the resulting required sample size.

AI can help researchers draft power analysis reports that are complete and transparent, pressing them to articulate the basis for each input rather than simply plugging in conventional values. AI can also help researchers identify common reviewer concerns, such as 'the effect size is taken from a single underpowered original study' or 'no correction was made for attrition', and address them proactively.

For grant applications in particular, AI can help researchers explain power calculations to non-specialist reviewers in plain language, translate statistical concepts into practical implications ('this sample size gives us an 85% chance of detecting an effect that would be clinically meaningful'), and frame the power analysis as part of a coherent research plan rather than a bureaucratic checkbox.

Key Takeaways

Statistical power is a fundamental research design issue with real consequences for scientific quality: underpowered studies inflate false-negative rates, bias effect size estimates upward in published literature, and contribute to replication failures.

The most important and most difficult part of power analysis is effect size specification. Convention-based specification is the weakest approach; anchored substantive specification (minimum detectable effect that is practically or theoretically meaningful) is the most defensible. AI helps researchers move from convention-based to grounded effect size choices and articulate the reasoning explicitly.

Complexity requires additional power considerations: mediation analysis needs larger samples than direct effect tests; multilevel designs require ICC corrections; unreliable measures attenuate effects; and moderation effects are typically smaller than main effects. AI helps researchers navigate these complexities and conduct sensitivity analyses that illuminate the power landscape.

Design strategies can increase power without increasing sample size: within-subjects designs, high-reliability measures, strong predictor covariates, and stronger manipulation doses. AI helps researchers systematically identify which strategies are applicable and estimate their impact.

Power analyses must be reported transparently, with explicit justification for all inputs. AI helps researchers produce defensible power analysis reports that address anticipated reviewer concerns and communicate the reasoning clearly.