Overview

Lesson 3.4: Open Science and AI Transparency

This lesson teaches you how to practice open science while incorporating AI systems: proactively sharing not just data and code (as traditional open science practices), but also the prompts, configurations, and interaction logs that document your AI use. You'll learn to support preregistration of AI-assisted analyses, maintain transparent communication with reviewers and readers about AI's role, and build scientific trust through comprehensive documentation of your AI processes.

Title

Lesson 3.4: Open Science and AI Transparency

Purpose

This lesson teaches you how to practice open science while incorporating AI systems: proactively sharing not just data and code (as traditional open science practices), but also the prompts, configurations, and interaction logs that document your AI use. You'll learn to support preregistration of AI-assisted analyses, maintain transparent communication with reviewers and readers about AI's role, and build scientific trust through comprehensive documentation of your AI processes.

Open Science in the Age of AI: Expanded Transparency Obligations

Open science is a set of practices aimed at making the process and products of research freely available, transparent, and reproducible. Its core commitments include sharing data, sharing code, preregistering analyses, publishing null results, and communicating methods with sufficient detail for others to replicate or evaluate them. These commitments emerged from recognition that science had developed a replication crisis, that publication bias was distorting the scientific record, and that restricted access to methods and data was impeding cumulative progress.

AI poses fresh challenges for open science that its practitioners are still working through. On one hand, AI has expanded what must be shared to achieve genuine transparency. A methods section that describes 'AI-assisted analysis' without specifying the model, version, prompts, parameters, and validation approach is not adequate for replication. It is the equivalent of describing a chemical experiment by saying 'reagents were combined.' The community is developing norms for what AI transparency requires, but advanced researchers should not wait for those norms to be mandated before adopting them.

On the other hand, AI creates genuine tensions with open science ideals. Commercial AI models may be modified, deprecated, or withdrawn by vendors, making reproducibility depend on third-party decisions outside researchers' control. Proprietary training data in foundation models means researchers cannot always share all the information needed for full replication. AI-generated content may complicate authorship and contribution attribution norms. These tensions do not excuse opacity; they require researchers to be honest about the limits of what can be reproduced and to document those limits clearly.

At the advanced level, open science practice in AI research means proactively sharing the full AI process documentation alongside your data and code, and engaging with your field's community to help develop the norms that will govern this sharing. It means treating your prompts as methods, your interaction logs as lab records, and your AI disclosure as a non-negotiable component of research transparency.

Preregistration of AI-Assisted Analyses

Preregistration is the practice of publicly committing your research hypotheses, design, and analysis plan before data collection or analysis begins. Its purpose is to prevent HARKing (Hypothesizing After Results are Known), the practice of presenting post-hoc exploratory analyses as if they were pre-planned confirmatory tests. Preregistration has become a strong norm in experimental psychology, clinical research, and is spreading to other fields.

AI complicates preregistration in important ways. The iterative, exploratory nature of many AI-assisted analyses, where prompt design, model selection, and analysis strategy evolve through interaction with the AI, does not fit neatly into the traditional preregistration model. Researchers may legitimately worry that preregistering a specific prompt template makes their preregistration too restrictive, since prompt refinement based on output quality is a valid methodological activity.

The solution is not to abandon preregistration for AI research, but to develop AI-appropriate preregistration forms. These should specify: the research question and hypotheses being tested (unchanged from standard preregistration), the AI approach at the level of the analysis type (e.g., 'LLM-assisted thematic coding of interview transcripts'), the criteria by which prompt quality will be evaluated in the development phase, the stopping rules for prompt development (e.g., 'we will finalize the prompt after validation on a held-out 10% sample meeting an agreement threshold of 85%'), the validation procedure for AI outputs, and how deviations from the planned approach will be reported. This approach captures the preregistration goal, a commitment made before confirmatory analysis, without the false rigidity of committing to a specific prompt before any pilot work.

Some disciplines are now developing AI-specific preregistration templates through platforms like the Open Science Framework (OSF) and the Center for Open Science. Engaging with these efforts in your field, contributing to template design, piloting them in your own research, is a way to advance community norms while benefiting from the credibility signal that comes with preregistration.

Sharing Prompts, Configurations, and Interaction Logs

The traditional open science triad is: share your data, share your code, share your materials. For AI-assisted research, this expands to include: share your prompts, share your AI configurations, share your interaction logs.

Prompts are methodological instruments in AI research the way questionnaires are instruments in survey research. Just as you would share a validated questionnaire in supplementary materials, you should share the prompts used in your AI analysis. This serves multiple purposes: it enables readers to assess whether the prompt was designed in a way that could introduce bias (e.g., leading language, restricted scope), it enables other researchers to replicate or adapt your approach, and it signals the level of methodological care you applied.

For large language models, prompt sharing should include: the system prompt or instructions given to the model, the user-turn prompt template, any few-shot examples provided, and any chain-of-thought or structured output specifications. The level of detail that has historically gone into describing survey instruments should now go into describing prompt instruments.

AI configurations, the model name, version, temperature setting, context window parameters, and any other settings that affect model behavior, are the equivalent of experimental settings. They should be reported in methods sections with the same specificity as you report statistical software version, statistical methods chosen, and thresholds used. A paper reporting AI-assisted analysis without specifying model version is methodologically opaque in a way that peer review should not accept.

Interaction logs, the full record of queries submitted and responses received, occupy more complex territory. They may be large, they may contain sensitive participant data (if participants' text was submitted as prompt context), and they may be proprietary or commercially generated in ways that create sharing complications. The practical guidance is: share interaction logs to the extent possible, with sensitive content appropriately redacted, in a citable repository. Where full logs cannot be shared (due to sensitivity or size), share representative examples and document why complete sharing was not possible. Acknowledging the limitation is part of transparency.

Communicating AI Use to Reviewers and Readers

Scientific communication norms around AI disclosure are evolving rapidly. Two years ago, many researchers did not know whether to disclose AI use in their methods sections. Today, most journals have explicit policies: some requiring disclosure, some requiring that AI not be listed as an author, some specifying what information the disclosure must include. Staying current with your target journals' policies is a basic professional obligation, not an optional best practice.

Beyond compliance with specific journal policies, researchers should aspire to disclosure standards that allow readers and reviewers to evaluate the role AI played in the research and to assess whether that role was appropriate. This means going beyond a perfunctory sentence ('AI tools were used to assist with analysis') to explain what AI did specifically, what its limitations were, and what validation was applied to its outputs.

For peer reviewers, adequate AI disclosure enables them to ask the right questions. A reviewer who knows that AI was used to generate a thematic coding of qualitative data can assess the validation methodology. A reviewer who doesn't know AI was used cannot evaluate the analysis appropriately. Hiding or minimizing AI involvement deprives reviewers of information they need to perform their function.

The concept of 'graduated disclosure' is useful here: the level of AI disclosure detail should scale with AI's centrality to the research. If AI was used peripherally, to format citations, to check grammar in supplementary materials, a brief note may suffice. If AI was a core analytical tool whose outputs are central to the paper's claims, detailed disclosure of model, version, prompts, validation, and limitations is required. Reviewers and readers deserve to understand exactly what they are evaluating.

A practical approach: draft your AI disclosure statement at the same time as your methods section, using the same level of specificity. Ask yourself: if a colleague wanted to exactly replicate my AI analysis, would this disclosure tell them everything they need to know? If not, add what's missing.

Building Scientific Trust Through AI Transparency

Scientific trust is foundational to research's social function. When readers and reviewers trust that a paper's methods are accurately described and its findings are genuinely supported, they can build on that work confidently. When trust erodes, through undisclosed methods, selective reporting, or inability to reproduce findings, science's cumulative nature breaks down.

AI poses trust risks that researchers must take seriously. AI hallucination (generating plausible but false information) can introduce errors into literature reviews, background sections, and summaries if AI output is not rigorously verified. AI-generated text that sounds authoritative but is factually wrong, cited without verification, can propagate errors through the scientific record. Researchers who use AI must implement and disclose verification workflows: how they checked AI outputs, what sources they verified against, what they found when they checked.

Prompt engineering can introduce analyst bias in subtler ways than traditional analysis. A prompt that frames a question tendentiously will generate tendentious outputs. Open disclosure of prompt text gives the research community the ability to identify such framing and evaluate its impact, which is exactly the function peer review is supposed to serve.

Transparency about AI limitations in your specific research context is also trust-building. Disclosing that 'the AI model used in this study was validated on [domain] data and may underperform on [specific population]' is an honest disclosure of scope. Disclosing that 'the model version used may no longer be available for exact replication, but the model configuration and all outputs are archived at [DOI]' is an honest disclosure of reproducibility constraint. These disclosures do not weaken papers. They strengthen trust by demonstrating that the researcher has thought carefully about what they know and what they don't.

Open science in the AI era ultimately requires researchers who see transparency not as a compliance burden but as a professional value, who disclose AI use not because journals require it but because they recognize that science's authority comes from its methods being open to scrutiny. Building this culture requires researchers at the advanced level to model these practices and to advocate for them in their fields.

Contributing to Open Science Norms for AI Research

Individual researchers practicing transparency contribute to collective norms by demonstrating that full AI disclosure is compatible with publishable, impactful research. But advancing field-level norms requires more than individual practice. It requires active engagement in the community processes that develop those norms.

This includes peer reviewing with AI transparency in mind: when you review papers, request the AI disclosures that are missing, just as you would request a missing control condition or an unconvincing statistical argument. Reviewers who routinely request adequate AI disclosure signal to authors that disclosure is expected, accelerating norm adoption.

It includes writing about methodology. Papers that describe and evaluate AI transparency practices, reporting on what approaches to AI disclosure are most informative, what preregistration templates work for AI research, what validation protocols successfully identify AI errors, are methodological contributions that advance the field's capacity for self-governance.

It includes contributing to open platforms. Sharing prompt libraries, validation frameworks, and AI interaction log archives in community repositories (the Open Science Framework, GitHub, domain-specific repositories) creates shared infrastructure that other researchers can use, cite, and build on.

Finally, it includes advocating within your institution. Research transparency is also an institutional practice: institutions should require AI disclosure in internal grant applications, should recognize prompt development and AI validation as legitimate research activities in evaluation criteria, and should support infrastructure (secure repositories, logging systems) that makes transparency feasible. Researchers who advocate for these institutional changes from a position of advanced practice are best placed to lead these conversations.