Overview

Lesson 1.1: Designing Systematic Search Protocols with AI

The literature search is the methodological foundation of any systematic review. Its design determines what evidence is retrievable, and therefore what evidence can be synthesized. A poorly designed search protocol introduces systematic gaps that no downstream quality of screening or synthesis methodology can compensate for: if relevant studies are never retrieved, they cannot be included. This asymmetry makes search protocol design the single most consequential methodological decision in a systematic review.

Traditionally, designing a comprehensive search protocol has required specialized expertise: deep familiarity with controlled vocabulary systems (MeSH, Emtree, CINAHL Subject Headings), knowledge of how different databases index the same concepts differently, and proficiency with Boolean logic to construct queries that retrieve relevant records without generating unmanageable false-positive noise. These skills are typically held by experienced medical librarians or information specialists, and many systematic review guidelines explicitly recommend involving an information specialist in search strategy development.

AI tools have substantially democratized access to this expertise. Large language models with training on biomedical and research literature can suggest MeSH terms, generate Boolean search strings, identify coverage gaps in draft strategies, and translate search strings across database syntax, tasks that previously required hours of specialized work. This lesson teaches you to leverage these capabilities while understanding their limitations. AI is an excellent search strategy development partner, but it cannot replace your judgment about research scope, cannot guarantee currency in database-specific controlled vocabulary, and cannot validate that a generated search string will behave as intended when executed.

Title

Lesson 1.1: Designing Systematic Search Protocols with AI

Purpose

This lesson teaches researchers how to independently design comprehensive, reproducible systematic search protocols using AI as a development partner. You'll learn to construct multi-database search strategies that comply with PRISMA standards while documenting AI involvement appropriately for journal submission.

Foundations: What Makes a Search Protocol Systematic

A systematic search protocol is distinguished from an ad hoc literature search by three properties: comprehensiveness (the protocol is designed to retrieve all relevant literature, not merely a representative sample), reproducibility (the protocol can be executed by another researcher and yield substantially the same results), and transparency (every component of the protocol is documented at a level of detail that enables replication and critical appraisal).

Comprehensiveness requires searching multiple databases. No single database covers all relevant literature for most research questions. PubMed/MEDLINE is the primary database for biomedical literature but has selective coverage of conference proceedings, gray literature, and non-English language publications. Embase has broader coverage of European and pharmaceutical literature. Web of Science provides strong interdisciplinary coverage. Scopus covers a wide range of academic disciplines. CINAHL is essential for nursing and allied health literature. PsycINFO is required for psychology and mental health topics. The appropriate database selection depends on the review topic, and database selection justification should be documented in the protocol.

Reproducibility requires specifying not just what databases were searched, but when (date of search execution), using what search string (fully documented, including all Boolean operators, wildcards, and syntax), with what filters (date limits, language restrictions, publication types), and with what additional sources (reference list checking of included studies, citation searching, gray literature sources). The PRISMA-S checklist, a 16-item extension of PRISMA specifically for reporting literature searches, provides a comprehensive framework for reproducibility documentation.

Transparency requires disclosing all decisions that shaped the search, including decisions to exclude databases or sources and the rationale for those decisions. A systematic review that searched only PubMed is not inherently invalid, but that decision must be disclosed and justified so readers can assess whether the limited database coverage introduces meaningful gaps for the specific review topic.

Using AI as a Search Strategy Development Partner

AI tools are most valuable in search strategy development during three phases: initial strategy drafting, strategy expansion and validation, and syntax translation across databases.

For initial strategy drafting, provide the AI with your PICO or equivalent framework, the specific population, concept, and context of your research question, and ask it to generate candidate search terms across all relevant dimensions. A useful prompt structure: 'I am conducting a systematic review of [specific topic]. The key concepts are [population: X], [intervention/exposure: Y], and [outcome: Z]. Please generate a comprehensive list of search terms for each concept, including synonyms, abbreviations, British and American spelling variants, and related concepts that might be used in the literature.' This prompt consistently generates useful term sets that would take much longer to develop manually.

For controlled vocabulary, ask the AI to suggest MeSH terms, Emtree terms, or other controlled vocabulary entries for your key concepts. Important caveat: AI suggestions about controlled vocabulary must be verified against the actual database thesaurus. MeSH terms can be verified in the MeSH browser (meshb.nlm.nih.gov); Emtree terms can be verified in the Embase thesaurus tool. AI-generated controlled vocabulary suggestions are excellent starting points but are not reliable without verification, LLMs can suggest plausible-sounding MeSH terms that don't exist, or suggest terms that exist but map to different concepts than intended.

For strategy expansion, ask the AI to review a draft search string and identify potential gaps: 'Review this search strategy and identify any synonyms, related terms, or concept variants that might be used in the literature but are not currently captured.' AI tools are particularly good at identifying terminology used in different disciplines for the same concept, for example, the same clinical concept might be called 'counseling' in US literature and 'counselling' in UK literature, or might be described using a diagnostic term in medical literature but a functional description in social science literature.

For Boolean logic construction, AI tools can take a list of terms and structure them into a Boolean search string with appropriate nesting: OR operators within concept groups, AND operators between concept groups, and parentheses structuring precedence correctly. Always verify that the parenthesization is correct, Boolean logic errors are a common source of search strategy failures that are difficult to detect without careful manual review.

Navigating Database-Specific Search Syntax

One of the most time-consuming aspects of multi-database searching is translating a search strategy developed for one database into the syntax of others. AI tools can substantially reduce this burden, but syntax translation requires careful validation.

Databases vary in their field code syntax, wildcard characters, phrase searching conventions, and controlled vocabulary implementation. PubMed uses [MeSH Terms] for controlled vocabulary and asterisk (*) for truncation. Embase uses '/exp' for exploded Emtree terms and an asterisk for truncation. CINAHL uses (MH 'term') for MeSH headings and a hash (#) for truncation. Web of Science uses TOPIC: field codes and an asterisk for truncation. Scopus uses TITLE-ABS-KEY field codes with asterisk truncation.

To translate a search strategy, provide the AI with the complete source strategy, the source database, and the target database: 'Here is my search strategy for PubMed. Please translate it to Embase syntax, replacing PubMed MeSH terms with equivalent Emtree terms and adjusting all field codes and operators to Embase conventions.' This prompt reliably produces useful translation starting points, but every translated strategy must be manually validated before execution.

Validation involves two steps: logic validation and execution testing. Logic validation means reading the translated string carefully to verify that all parentheses are correctly placed, all field codes are valid for the target database, and all controlled vocabulary terms were correctly translated. Execution testing means running the translated strategy in the target database and comparing the result count to what you would expect given the source strategy, extreme divergences (very high or very low result counts) signal errors requiring investigation.

Documentation of each database strategy must be complete and final-version specific. When you refine a strategy, update the documentation immediately. It is common for researchers to document an early draft of their strategy while continuing to refine it, creating a disconnect between the documented and executed strategies, a methodological error that can surface during peer review.

Gray Literature and Supplementary Source Searching

Database searching alone is insufficient for most systematic reviews. Gray literature, research produced by organizations outside traditional academic publishing channels, includes clinical trial registries, government reports, conference proceedings, organizational reports, unpublished theses, and preprints. Systematic reviews that exclude gray literature are subject to publication bias: the tendency for positive results to be published in academic journals at higher rates than null or negative results, meaning database-only searches overrepresent positive evidence.

Gray literature sources relevant to most systematic reviews include: ClinicalTrials.gov and WHO ICTRP for registered clinical trials; FDA and regulatory agency databases for drug approval documents; government health agency websites (NICE, CDC, WHO); relevant professional organization websites; conference abstract books from key annual meetings; ProQuest Dissertations for unpublished theses; and preprint servers (medRxiv, bioRxiv, SSRN, arXiv as relevant).

AI tools can help identify relevant gray literature sources for a specific review topic. Ask: 'What gray literature sources should I search for a systematic review on [topic]? Include clinical trial registries, government databases, organizational sources, and preprint servers relevant to this field.' AI responses on gray literature sources are generally reliable but should be supplemented by discipline-specific knowledge about which organizations are most relevant.

Handsearching, manually reviewing the table of contents or abstract books of specific high-yield journals or conferences, is another supplementary source. AI tools can identify which specific journals or conferences are most relevant: 'Which journals and conferences are the primary publication venues for research on [topic]?' This information guides decisions about whether handsearching specific venues would be cost-effective given their likely yield.

Citation checking, reviewing the reference lists of all included studies and of key reviews in the field, is a standard and often highly productive supplementary search method that consistently identifies relevant studies missed by database searches, particularly older studies that predate comprehensive electronic database indexing.

PRISMA Compliance and Search Documentation

Systematic review journals and evidence synthesis organizations uniformly require that search strategies be documented at a level sufficient for replication. The PRISMA-S checklist (an extension of PRISMA 2020 specifically addressing search reporting) covers 16 items including: the rationale for database selection, dates of all searches, complete search strategies for all databases, details of any search limitations applied, and a description of any additional sources searched.

Complete search strategy documentation means reporting the exact search string executed in each database, not a summarized or simplified version. This is a common reporting failure in published systematic reviews: authors describe their search approach in general terms ('we searched PubMed, Embase, and CINAHL using terms related to X and Y') without providing the actual executable strings. Reviewers and editors increasingly require full strategy appendices, and journals that have adopted PRISMA-S routinely require documentation of all executed strategies.

AI involvement in search strategy development must be disclosed, though the field has not yet converged on a single disclosure format. A reasonable disclosure statement: 'Search strategies were developed with assistance from [AI tool name and version], which was used to generate initial term lists and suggest controlled vocabulary. All AI-generated suggestions were verified against database thesauri and refined by the authors. The final executed search strategies are provided in full in Supplementary Table X.' This disclosure is transparent without implying that AI made final decisions about strategy design.

Protocol registration is a critical accountability mechanism. Registering your search protocol, and specifically your search strategy, in PROSPERO before beginning the systematic review creates a pre-specified record that prevents post-hoc strategy modification without disclosure. Outcome switching and search strategy modification based on results are forms of publication bias that protocol registration is designed to prevent. Document any deviations from the registered protocol with explicit justification.

Validating Your Search Strategy

Before executing your full search, validate the strategy against a known set of relevant studies. Assemble a set of 15-30 papers that are definitively relevant to your research question, ideally identified through preliminary scoping. Run your search strategy and check whether these known-relevant papers appear in the results. If key papers are missing, this is a signal that the search strategy has gaps.

Diagnose why missing papers are not retrieved: are they missing because the search misses a key synonym? Because the paper uses different terminology than your strategy anticipates? Because the paper is indexed under controlled vocabulary you haven't included? Each diagnosis points to a specific expansion needed in the strategy.

AI tools can assist in this diagnostic process: 'This paper [provide abstract] is relevant to my systematic review but is not being retrieved by my search strategy [provide strategy]. Analyze why this paper might not be retrieved and suggest modifications to the strategy to capture it.' This prompt can identify terminology, abbreviations, or controlled vocabulary entries that your strategy is missing.

Sensitivity and precision are the key performance parameters in search strategy design. A highly sensitive strategy retrieves nearly all relevant studies but also retrieves large numbers of irrelevant ones (high false positive rate). A highly precise strategy retrieves fewer irrelevant records but risks missing relevant ones (lower recall). For systematic reviews, sensitivity takes precedence over precision: it is better to screen many irrelevant records than to miss relevant ones. However, extremely low precision searches generate unmanageable screening burdens, so balance is needed.

The tradeoff is typically managed through iterative refinement: expand the strategy to improve sensitivity until coverage of known-relevant papers is satisfactory, then tighten filters (date limits, language restrictions, publication type restrictions) if the result set is unmanageably large. Document every iteration of the strategy development process, including the rationale for decisions to add or remove terms.

Summary

Designing a systematic search protocol is a structured, methodical process that determines the evidence base of your review. AI tools have made the technical aspects of search strategy development, term generation, controlled vocabulary suggestion, syntax translation, substantially more accessible, enabling independent researchers to develop searches of near-information-specialist quality.

The critical principles are: search multiple databases with selection justified by coverage of your review topic; document every component of the executed strategy at full PRISMA-S compliance; include gray literature and supplementary sources; validate the strategy against known-relevant papers before full execution; and disclose AI assistance transparently.

The AI is your development partner. It generates candidates, suggests expansions, and handles syntax translation. Your judgment determines scope, validates controlled vocabulary, evaluates tradeoffs between sensitivity and precision, and makes final decisions about strategy design. That division of labor produces search protocols that are both more comprehensive and more efficiently developed than either AI or human effort alone could achieve.