Overview

Lesson 5.2: When to Use and When Not to Use AI in Research

This lesson teaches researchers how to develop and apply a decision framework determining when AI assistance is appropriate and when human-only approaches are preferable. Effective AI use is not about maximizing AI involvement. It is about directing AI to tasks where it adds genuine value while reserving human judgment for tasks where it is irreplaceable. The lesson introduces the RISK matrix (Routine, Input-output definition, Stakes level, Knowledge verification feasibility) as a structured tool for evaluating any research task, walks through scored examples across the research workflow, and shows how hybrid approaches that combine AI and human strengths often outperform either approach alone. You will leave with a completed RISK matrix for your own most common research tasks and a personal AI use policy that makes your AI decisions deliberate rather than habitual.

Title

Lesson 5.2: When to Use and When Not to Use AI in Research

Purpose

This lesson teaches researchers how to develop and apply a decision framework determining when AI assistance is appropriate and when human-only research is preferable. You will learn to assess task characteristics, weigh benefits against risks, and make deliberate choices about AI use rather than defaulting to AI for all tasks.

By the end of this lesson you will be able to: (1) apply the RISK matrix to evaluate any research task for AI suitability; (2) classify tasks as AI-appropriate, hybrid, or human-primary based on their scores; (3) design a hybrid workflow that leverages AI for suitable sub-tasks while preserving human judgment for unsuitable ones; (4) perform an opportunity cost analysis to identify where AI use would free the most researcher time for high-value work; and (5) draft a personal AI use policy for your specific research context.

Core Concepts

The most common failure mode in AI-assisted research is not using AI too much or too little. It is using it indiscriminately, applying it by default to all tasks rather than evaluating each task on its own merits. Indiscriminate use leads to over-reliance on AI for tasks that require human judgment (introducing reliability and validity risks) and under-use on tasks where AI could save significant time (missing efficiency gains). A structured decision framework prevents both failure modes.

The RISK Matrix

The RISK matrix evaluates any research task across four dimensions, each scored 1-3:

R - Routine vs. Novel
Score 1 (highly routine): The task follows a well-established pattern that AI has been trained on extensively. Examples: generating alternative wordings for survey items, reformatting citations, summarizing a paper's abstract. Score 2 (moderately routine): The task has an established structure but requires some domain-specific adaptation. Score 3 (highly novel): The task requires original thinking that has no close precedent. Examples: developing a new theoretical framework, deciding how to interpret a paradoxical finding, designing a study for a newly emerging phenomenon.

I, Input/Output Definition
Score 1 (well-defined): Success criteria are clear and verifiable before the task begins. Examples: 'extract all gene names from this text', either a name is extracted or it is not. Score 2 (moderately defined): Some criteria are clear but others require judgment. Score 3 (highly ambiguous): Success cannot be defined until after the task is complete or requires deep contextual judgment. Examples: 'identify the most important theoretical contribution of this paper.'

S, Stakes Level
Score 1 (low stakes): Errors are easily caught and corrected; consequences are minor. Examples: a first draft that will be substantially revised. Score 2 (moderate stakes): Errors require effort to correct and may affect downstream work. Score 3 (high stakes): Errors are difficult to detect, have serious consequences, or are hard to reverse. Examples: primary outcome analysis in a clinical trial, ethical determinations about participant safety.

K, Knowledge Verification Feasibility
Score 1 (easily verifiable): A human can check the AI output against a primary source quickly. Examples: did the citation exist? did the code produce the correct number? Score 2 (moderately verifiable): Verification requires effort or expertise. Score 3 (verification-difficult): There is no reliable way to check whether the AI output is correct without approximately equal effort to doing the task manually. Examples: whether an AI-generated theoretical synthesis captures all relevant nuance.

Interpreting RISK Scores

Total RISK score ranges from 4 (maximally AI-suitable) to 12 (maximally AI-unsuitable):

Score 4-6: AI-appropriate. Use AI as primary tool with verification proportional to stakes.
Score 7-9: Hybrid. Use AI for well-defined sub-tasks; preserve human judgment for novel, ambiguous, or high-stakes elements.
Score 10-12: Human-primary - AI can inform or generate options, but a human should drive the task entirely.

Hybrid Approaches

Many research tasks score in the middle range and are best handled with hybrid workflows that decompose the task into sub-tasks and apply AI only to the AI-suitable sub-tasks. A literature synthesis is a good example: generating individual paper summaries (routine, well-defined, low stakes, easily verified, score 4-5) is AI-appropriate; synthesizing those summaries into a coherent theoretical narrative (novel, ambiguous, moderate-high stakes, hard to verify, score 9-11) is human-primary. The hybrid approach uses AI to generate summaries and human reasoning to synthesize them, capturing the efficiency gain while preserving intellectual quality.

Opportunity Cost Analysis

The full value of AI use is not just the efficiency gain on AI-assisted tasks. It is the opportunity cost saved: the researcher time freed for high-value human-primary tasks. A researcher who uses AI to handle 40 hours of literature screening frees 40 hours that can be spent on theoretical development, data interpretation, and writing, tasks where her unique expertise is irreplaceable. Opportunity cost analysis asks: which of my routine, AI-suitable tasks consume the most researcher time? Those are the highest-return candidates for AI assistance.

Practical Applications

The following worked examples illustrate RISK matrix scoring across the research workflow.

Example 1: Literature Search for a Systematic Review
R = 1 (highly routine, search patterns are established)
I = 1 (well-defined, inclusion/exclusion criteria are explicit)
S = 2 (moderate stakes, missing papers affects review quality but can be partially mitigated by database search verification)
K = 1 (easily verifiable, compare AI-identified papers to a traditional database search)
Total: 5, AI-appropriate. Decision: Use AI search with verification by traditional database search. Hybrid note: AI handles initial screening; human makes final inclusion/exclusion decisions for borderline cases.

Example 2: Developing a Novel Theoretical Framework
R = 3 (highly novel, no precedent; requires original conceptual work)
I = 3 (highly ambiguous, 'a good theoretical framework' cannot be defined in advance)
S = 3 (high stakes, theoretical contributions define a researcher's scholarly identity and are scrutinized by reviewers)
K = 3 (verification-difficult, no external reference to check whether the framework is correct or complete)
Total: 12, Human-primary. Decision: AI can generate literature overviews and summarize competing frameworks to inform the researcher's thinking, but the theoretical contribution must be the researcher's own.

Example 3: Drafting a Literature Review Section
R = 2 (moderately routine, structure is established but synthesis requires judgment)
I = 2 (moderately defined, section structure is clear; synthesis quality is not)
S = 2 (moderate stakes, will be reviewed; errors affect credibility but are correctable)
K = 2 (moderately verifiable, factual claims can be checked; synthesis quality requires expert judgment)
Total: 8, Hybrid. Decision: AI drafts individual paragraph summaries and organizes them by theme; researcher synthesizes the theme-level narrative, identifies connections across sources, and makes interpretive claims.

Example 4: Generating Survey Item Alternatives
R = 1 (routine)
I = 1 (well-defined, alternatives to a specific item are clearly bounded)
S = 1 (low stakes, alternatives will be reviewed and selected from)
K = 1 (easily verifiable, researcher reviews each alternative)
Total: 4, AI-appropriate. Decision: Ask AI to generate 10-15 alternatives; researcher selects and refines.

Example 5: Interpreting an Unexpected Finding
R = 3 (highly novel, unexpected findings by definition lack precedent)
I = 3 (highly ambiguous, correct interpretation depends on context, theory, and methodology)
S = 3 (high stakes, misinterpretation affects conclusions and downstream work)
K = 3 (verification-difficult, no external reference to confirm the interpretation is correct)
Total: 12, Human-primary. Decision: AI can retrieve relevant literature, generate alternative explanations, and identify comparable cases in the literature. The interpretive judgment is entirely the researcher's.

Building Your Personal RISK Matrix

Step 1: List 15-20 tasks that represent your most common and most time-consuming research activities.

Step 2: Score each task on R, I, S, K and sum the scores.

Step 3: Cluster tasks into three groups: AI-appropriate (4-6), Hybrid (7-9), Human-primary (10-12).

Step 4: For your top 5 time-consuming AI-appropriate tasks, estimate how many hours per month you currently spend on them. This is your opportunity cost available for reallocation.

Step 5: For each Hybrid task, decompose it into sub-tasks and apply the RISK matrix to each sub-task individually to identify exactly which components should be AI-assisted and which should be human-primary.

Step 6: Draft your personal AI use policy using the task clusters from Step 3 as the structural framework.

Key Takeaways

Explicit decision frameworks make AI use deliberate rather than habitual. The risk of indiscriminate AI use is not just efficiency loss. It is the displacement of human judgment from tasks that require it, which compromises research quality in ways that are difficult to detect.

The RISK matrix provides a structured evaluation across four dimensions: how Routine the task is, how well-defined the Input/output relationship is, the Stakes level if errors occur, and how feasible Knowledge verification is. Tasks scoring 4-6 are AI-appropriate; 7-9 are hybrid; 10-12 are human-primary.

Hybrid approaches that combine AI and human strengths often outperform either approach alone. Most moderate-scoring research tasks can be decomposed into sub-tasks that score differently, AI handles the routine, well-defined, low-stakes sub-tasks while humans handle the novel, ambiguous, high-stakes ones.

Opportunity cost analysis reveals that AI's full value includes not just efficiency gains but time freed for irreplaceable human-primary work. The highest-return AI uses are those that save the most researcher time on tasks that genuinely do not require human judgment.

Decision frameworks should be treated as living documents, not fixed rules. Update your RISK matrix as AI capabilities change, as your own research tasks evolve, and as you accumulate experience with which AI applications produce reliable results in your specific workflow.

The practical outcome of this lesson is a personal AI use policy: a written document that specifies which tasks you will use AI for, which tasks are human-primary, and what hybrid workflows you will apply for tasks in the middle range.

RISK Matrix Reference: Common Research Tasks

The following table provides pre-scored RISK matrix assessments for the most common research tasks. Use these as starting points and adjust based on your specific context.

AI-Appropriate Tasks (Score 4-6)
- Reformatting and standardizing citations [Score: 4]
- Generating alternative wordings for survey items [Score: 4]
- Summarizing individual paper abstracts [Score: 5]
- Extracting structured information (author, year, sample size) from papers [Score: 4]
- Generating initial data cleaning code from a description [Score: 5]
- Screening titles and abstracts against explicit inclusion/exclusion criteria [Score: 5]
- Creating study flashcards and review materials [Score: 4]
- Translating text (with verification) [Score: 5]
- Generating visualization code from a description [Score: 5]

Hybrid Tasks (Score 7-9)
- Drafting a literature review section [Score: 8]
- Generating survey instruments (draft items for human review and pilot testing) [Score: 7]
- Exploratory data analysis (generating code + visualizations; human interprets) [Score: 7]
- Writing a methods section (AI drafts standard procedures; human verifies and adapts) [Score: 7]
- Generating grant narrative drafts (AI drafts; domain expert verifies all scientific content) [Score: 8]
- Qualitative coding (AI suggests codes; human reviews and refines) [Score: 8]

Human-Primary Tasks (Score 10-12)
- Developing novel theoretical frameworks [Score: 12]
- Interpreting unexpected or paradoxical findings [Score: 12]
- Making ethical decisions about participant welfare [Score: 12]
- Selecting a dissertation research question [Score: 11]
- Evaluating whether a study design is adequate for a research question [Score: 10]
- Deciding whether to submit, revise, or abandon a paper [Score: 11]
- Peer review of others' manuscripts [Score: 11]