Overview

Reproducibility is one of science's foundational values: the requirement that research methods be described transparently enough for others to understand, evaluate, and replicate the work. AI introduces specific reproducibility challenges that researchers using traditional methods do not face: AI model outputs can vary across API calls, model versions are updated without researchers' knowledge, prompt wording affects outputs in non-obvious ways, and the reasoning processes of AI systems are often opaque even when the outputs are visible. These characteristics do not make AI-assisted research inherently less reproducible, but they require researchers to adopt new documentation practices that account for AI's specific nature and address the particular ways that AI-assisted methods can fail to meet reproducibility standards.

Title

Lesson 5.2: AI and Research Reproducibility

Purpose

This lesson teaches researchers how to ensure that AI-assisted research remains reproducible, that findings can be verified, analyses can be rerun, and methods can be understood and replicated by others. You'll learn to document AI decisions thoroughly, maintain version control of models and prompts, and create reproducible workflows despite AI's complexity.

Understanding AI-Specific Reproducibility Challenges

Traditional research reproducibility challenges, replication of experimental conditions, availability of data and materials, transparency of analysis code, are well-understood in the scientific community and addressed by established practices: data sharing mandates, preregistration, open materials repositories, and code-sharing norms. AI-assisted research inherits all of these challenges and adds new ones specific to the nature of large language models and other AI systems.

The first AI-specific challenge is output variability. AI model outputs are probabilistic, the same prompt, run against the same model at different times, may produce different outputs. This is because AI systems sample from probability distributions rather than computing deterministic functions. For qualitative AI-assisted tasks (thematic coding, content analysis, summarization), output variability means that a researcher who attempts to replicate their own analysis a month later may get different results if the model or temperature settings have changed. Documentation of this variability, and mitigation through techniques like setting temperature to zero for deterministic outputs when reproducibility is essential, is a methodological responsibility.

The second challenge is model version uncertainty. AI providers update models frequently, and updated versions may produce systematically different outputs for the same prompts. A study that used 'GPT-4' in March may not be replicable using 'GPT-4' in September if the model has been updated. Researchers must document the specific model version identifier (not just the model family) and, where possible, use pinned model versions through API calls that allow version locking. Many AI providers make older model versions available through their APIs for exactly this purpose.

The third challenge is prompt sensitivity. Small changes in prompt wording can produce meaningfully different AI outputs. A prompt that asks for 'a brief summary' versus 'a 100-word summary' versus 'a summary in three bullet points' may produce outputs that code to different categories in a content analysis or generate different argument structures in a writing workflow. Documenting prompts with exact wording is not optional for reproducible AI-assisted research. It is as essential as documenting the exact wording of survey instruments in social science research.

The fourth challenge is opacity. Traditional computational methods produce outputs through processes that are, at least in principle, fully inspectable. You can trace every calculation. AI reasoning is not fully inspectable: even when the output is visible, the internal process that generated it is not. This does not make AI-assisted research invalid, but it means that 'the AI produced this result' is not a sufficient methodological description, the researcher must document what the AI was asked to do, under what conditions, with what level of output variability, and how the outputs were verified and processed.

Documentation Practices for Reproducible AI Research

Reproducible documentation of AI-assisted research requires treating AI as a methodological tool with specific parameters that must be reported, just as researchers report the specific parameters of their measurement instruments, statistical software, and analysis procedures.

The minimum documentation requirements for AI-assisted research tasks are: (1) the AI provider and specific model version identifier (e.g., 'OpenAI GPT-4o, version 2024-11-20, accessed via API' rather than simply 'GPT-4'); (2) the exact text of all prompts used, including system prompts and any chain-of-thought instructions; (3) the parameter settings used, particularly temperature (which controls output randomness), maximum tokens, and any other generation parameters; (4) the date of access; and (5) the procedure for handling AI outputs, how they were reviewed, verified, and processed before being incorporated into the research.

For studies where AI outputs directly contribute to findings, AI-assisted content analysis, AI-assisted screening in systematic reviews, AI-generated summaries that were coded, an additional requirement is documentation of the AI's error rate or inter-rater reliability. If AI coding was validated against a human-coded reference sample, the agreement statistics should be reported. If AI screening was calibrated and spot-checked, the spot-check sample size and error rate should be reported. These metrics allow readers to evaluate how much weight to place on the AI-assisted analysis.

Prompt documentation is a specific documentation challenge because prompts can be long, iterative, and context-dependent. A practical system is to maintain a prompt library, a structured document or repository that logs every prompt used in the research, the task it was applied to, the model and date it was used with, and any notable patterns in the outputs it produced. This library serves both as methodological documentation and as an institutional resource for future research using similar methods.

For AI-assisted workflows that involve multiple sequential steps, where the output of one AI task is fed as input to the next, documentation must describe the entire chain: what was input at each step, what the AI produced, how the output was reviewed or processed before becoming the next step's input. Workflow diagrams are useful for communicating complex AI-assisted pipelines visually, and AI can help generate these diagrams from a written description of the workflow.

Version Control for Prompts and AI Workflows

Version control, the practice of tracking changes to files over time, typically using systems like Git, is standard practice for research code and increasingly expected for research data and analysis scripts. The same version control logic applies to prompts and AI workflows used in research: changes to prompts are changes to your methodology, and those changes should be tracked, documented, and restorable.

A practical approach to prompt version control is to store all research prompts in a Git repository alongside the research code and data, with meaningful commit messages that describe what changed and why. A commit message like 'Updated screening prompt to include preprint exclusion criterion following pilot calibration, previous version missed 12% of preprint records in calibration sample' provides the methodological context that makes version history interpretable.

For AI workflow automation, where prompts are embedded in scripts that call AI APIs, version control of the scripts automatically tracks prompt changes. This is the most reliable version control approach because it ties prompt versions to the code execution environment, the model version specified in the API call, and the date of execution. Researchers who build automated AI workflows should ensure that model version pinning is included in the API call parameters, not left to the API default, which may change as providers update their models.

Model version pinning, specifying the exact model version in API calls rather than using generic model names that resolve to the current version, is one of the most important reproducibility practices for computational AI-assisted research. When a study's analysis was conducted using a specific model version, pinning that version in the code allows future researchers to run the same code against the same model behavior, even if the provider has since released updated versions.

For AI tools that do not offer API access with version pinning, many consumer-facing AI interfaces do not, researchers should document the model version as specifically as the tool allows, accept that some degree of reproducibility limitation exists for these tools, and where possible conduct research through API access rather than through consumer interfaces when reproducibility is methodologically critical.

Verification and Replication in AI-Assisted Workflows

Reproducibility requires not just that methods are documented but that they actually produce consistent outputs when re-applied. For AI-assisted research, verification of consistency is especially important given the output variability characteristics described above.

For qualitative AI-assisted analysis tasks, a practical verification approach is the repeated-run test: run the same prompts on the same data three times and measure the agreement between runs. If the three runs produce substantially different outputs (high inter-run disagreement), the prompt and parameter settings need adjustment before the analysis is used in research. Setting temperature to zero produces more deterministic outputs; adding explicit output formatting requirements reduces variability in how the AI structures its response.

For AI-assisted systematic review screening, calibration against a human-coded reference sample, as described in earlier chapters, is both a validation practice and a reproducibility check. If the AI screening approach was validated against a reference sample before being applied at scale, this validation provides a basis for estimating how well the method would reproduce across runs.

A distinct but related concept is transparency rather than exact reproducibility. In some AI-assisted workflows, exact reproducibility may not be achievable, creative AI outputs cannot be precisely reproduced, but transparency can still be maintained. Transparency means that the methods used are described precisely enough that a reader can understand what was done, evaluate its appropriateness, and assess the likely robustness of the results. This standard, methodological transparency, is often the appropriate goal for qualitative or creative AI-assisted work where exact output replication is neither expected nor necessary.

For research that involves AI-assisted data analysis, where AI tools are used to generate code for statistical analyses or to assist with data cleaning, reproducibility requires saving not only the final analysis code but also the AI interaction logs that produced it. A data cleaning decision made based on AI suggestion should be documented as such, with the prompt and output recorded, so that future researchers can understand why specific data points were handled in specific ways.

Reporting AI Methods in Research Publications

Reproducibility requires not just internal documentation but external reporting: describing AI methods in published research with enough detail for the scientific community to evaluate the methods and, where necessary, attempt replication. The standards for this reporting are evolving rapidly as journals and professional associations develop guidelines, but the fundamental principle is clear, AI methods must be described with the same specificity as any other methodological component.

The Methods section of an AI-assisted research paper should include a dedicated AI methods subsection that describes: the AI tools used (provider, model, version, access date); the specific tasks AI was used for; the prompts used or reference to a supplementary appendix where prompts are provided; the parameter settings; and the procedures for reviewing and processing AI outputs. 'We used an AI tool to assist with analysis' is not sufficient description; 'We used OpenAI GPT-4o (version 2024-11-20) via API at temperature=0 to code interview transcripts for five thematic categories [prompts provided in Supplementary Appendix A]; coding was validated against human coding of a random 20% subsample (Cohen's kappa = 0.82)' is.

For systematic review screening using AI, the PRISMA reporting guidelines are being updated to include AI-specific elements: the AI tool and version used, the screening prompt, the calibration procedure, the spot-check sample and error rate, and the decision rule for handling cases where AI and human judgments conflicted. Researchers conducting AI-assisted systematic reviews should monitor the latest PRISMA extensions and consult field-specific methodological guidelines.

Data and code sharing for AI-assisted research should include: the full prompt library used (typically as a supplementary document or in a repository); the analysis code that called AI APIs (with model version pinned in the code); and, where the AI outputs themselves constitute primary data (coded documents, AI-generated summaries that were analyzed), the AI outputs themselves. This transparency package allows the scientific community to evaluate the methods, verify the outputs, and assess the reproducibility of the AI-assisted components of the work.