Lesson 1.3: AI Capabilities and Limitations

Purpose

You now understand what AI does (pattern recognition and generation) and how it works (probabilistically, token-by-token). Now you need a practical map: what can AI actually be relied on to do for managerial work, and what should you never ask it to do?

This lesson provides that map. It's a reference you'll return to as you encounter new tasks.

Why This Matters for Managers

A common mistake is viewing AI as either "magical solution" or "useless hype." The truth is more nuanced: AI is exceptionally good at some things, terrible at others, and uncertain in the middle.

Your job is to match tasks to capability. Assigning something to AI that it's bad at wastes time, creates errors, and breeds skepticism. Using AI where it excels multiplies your effectiveness.

The stakes include:

• Credibility (you look uninformed if you ask AI to do something it can't)
• Productivity (using AI on the right tasks saves significant time)
• Risk (some tasks delegated to AI create real problems)
• Team trust (your team needs to see AI used appropriately)

Core Concepts

AI Reliability Scale

AI isn't uniformly good or bad. Think of it on a spectrum:

TIER 1: Very High Reliability (Use Confidently)

• Tasks where output quality is immediately visible to human eyes
• Tasks with multiple acceptable answers
• Tasks where errors are caught before deployment
• Generally: transformative tasks (reshaping existing material)

TIER 2: High Reliability (Use with Review)

• Tasks with clear success criteria
• Tasks where humans verify before use
• Tasks that save time even if 20% of output needs fixing
• Generally: generative tasks with human oversight

TIER 3: Moderate Reliability (Use Cautiously)

• Tasks requiring judgment or context
• Tasks with high consequences for errors
• Tasks where nuance matters
• Generally: analytical tasks with human final decision

TIER 4: Low Reliability (Don't Use)

• Tasks requiring guaranteed accuracy
• Tasks involving proprietary or sensitive information
• Tasks with irreversible consequences
• Tasks requiring value judgments
• Generally: things where AI errors create real harm

What AI Can Reliably Do (Tier 1 & 2)

DRAFT AND REFINE TEXT

What AI does exceptionally well:

• Generates first drafts of emails, memos, proposals, announcements
• Takes rough notes and creates polished paragraphs
• Rewrites for tone, formality, conciseness
• Expands bullet points into prose
• Adapts the same content for different audiences

Why it works:

• Text generation is what AI was built for
• You immediately see the output and can judge quality
• Variations in "good" writing means multiple correct answers
• The human refines, so errors don't escape

Manager use: "Draft an email to the team about the office reopening, friendly and encouraging tone."

Success rate: Very high. Almost always usable with light editing.

SUMMARIZE INFORMATION

What AI does exceptionally well:

• Extracts key points from long documents
• Condenses meeting notes into action items
• Summarizes email threads or conversations
• Pulls relevant information from articles
• Identifies main arguments and supporting details

Why it works:

• Summarization follows consistent patterns
• Length and format constraints are clear
• You can compare the summary to the original and verify accuracy
• For partial inaccuracy, human correction is quick

Manager use: "Here's a transcript of a 90-minute board meeting. Summarize in 10 bullet points: key decisions, action items, and owners."

Success rate: Very high for factual summaries. Lower if summary requires subjective interpretation of importance (you fix it then).

EXTRACT AND CATEGORIZE

What AI does exceptionally well:

• Pulls specific information from text (names, dates, decisions)
• Categorizes items based on clear criteria
• Structures unstructured information
• Tags or labels content
• Organizes lists

Why it works:

• The task has clear success criteria
• You can verify the work by spot-checking
• Errors are visible and easy to correct

Manager use: "I have 50 customer feedback comments. Categorize each as: bug report, feature request, or praise. Also extract the product mentioned."

Success rate: Very high. AI is good at pattern matching.

BRAINSTORM AND GENERATE IDEAS

What AI does exceptionally well:

• Generates multiple options or approaches
• Asks questions you might not have considered
• Suggests angles or perspectives on a problem
• Creates variations on a theme
• Helps think through scenarios

Why it works:

• No single "right answer" expected
• Quantity is valuable even if not all ideas are useful
• Human filtering is built in
• Exploration mode, not decision mode

Manager use: "We want to improve manager-to-IC communication. What are 10 approaches we could try?"

Success rate: High. Some ideas will be obvious or impractical. That's okay. A few good ones make the exercise valuable.

EXPLAIN AND TEACH

What AI does exceptionally well:

• Explains concepts clearly at different levels
• Breaks down complex topics into steps
• Answers how/why questions about concepts
• Provides examples and analogies
• Suggests learning paths

Why it works:

• Explanation follows patterns AI learned from educational content
• Multiple good explanations exist (no single right answer)
• You can evaluate clarity for your needs
• Errors in explanation are often caught by understanding gaps

Manager use: "Explain the difference between Type I and Type II errors in hypothesis testing. Explain it to someone who hasn't studied statistics."

Success rate: High for conceptual explanations. Lower for technical accuracy in specialized domains.

ANALYZE TEXT AND IDENTIFY PATTERNS

What AI does exceptionally well:

• Identifies sentiment in customer feedback
• Spots recurring themes in responses
• Analyzes writing style or tone
• Identifies bias or assumptions in text
• Categorizes by theme or topic

Why it works:

• Pattern identification is AI's core strength
• Output is usually correct or close enough for human judgment
• You can verify by sampling

Manager use: "Analyze these 30 customer testimonials. What are the three most common complaints? What do they want from us?"

Success rate: High. You might need to refine the categorization, but AI identifies themes quickly.

REFORMAT AND STRUCTURE

What AI does exceptionally well:

• Converts lists into tables
• Turns prose into bullet points
• Reorganizes information for different purposes
• Reformats documents for different contexts
• Translates between formats (email to memo, article to summary, etc.)

Why it works:

• Structural transformation is straightforward
• Errors are immediately visible
• The original is available for reference

Manager use: "I have a paragraph-form process document. Reformat it as a numbered step-by-step guide with clear headings."

Success rate: Very high.

What AI Can Probably Do (Tier 2 - Use with Review)

WRITE ANALYTICAL SUMMARIES

What AI can do:

• Compare two approaches and list pros/cons
• Evaluate options against criteria you define
• Analyze trends in data
• Interpret data and suggest implications

What to watch for:

• AI might miss important context
• Analysis might be surface-level
• Might overstate confidence
• Might reflect biases in training data

How to use: Always review analysis. Ask "What are you basing this on? What could I be missing?"

Manager use: "Compare our two vendor options based on: cost, integration capability, support quality. What's your analysis?"

Success rate: Moderate-to-high. Good for frameworks. Verify conclusions.

HELP WITH PLANNING AND PROJECT MANAGEMENT

What AI can do:

• Suggest project timeline and milestones
• Identify potential risks
• Propose resource allocation
• Create checklists for complex tasks
• Suggest questions to ask stakeholders

What to watch for:

• AI lacks knowledge of your organization's constraints
• Doesn't know the politics or relationships
• May be generic (works for any organization)
• Doesn't know hidden dependencies

How to use: Use as a starting framework. "This AI-generated timeline assumes we have full team availability. We don't. Here's what we're adjusting."

Manager use: "I'm managing a system migration. Generate a timeline, key milestones, and potential risks to watch for."

Success rate: Moderate. Good skeleton. You provide the customization.

HELP WITH DECISION MAKING

What AI can do:

• List considerations for a decision
• Suggest frameworks for thinking through options
• Play devil's advocate
• Ask clarifying questions
• Generate scenarios

What to watch for:

• AI can't make the decision (that's yours)
• AI doesn't know what you actually care about
• AI might miss your organization's values
• AI can't account for relationships or politics

How to use: AI is a thinking partner, not a decision maker. "Help me think through this. Here's the situation [describe]. What am I not considering?"

Manager use: "Should we promote Sarah or hire externally? Lay out the considerations."

Success rate: Moderate. Useful for expanding your thinking. You make the call.

What AI Is Bad At (Tier 3 - Use Cautiously, If At All)

GUARANTEE FACTUAL ACCURACY

Why it's risky:

• AI hallucinate confidently
• Training data may be outdated
• AI doesn't distinguish between certain and uncertain knowledge

Better approach:

• Use AI to draft, then verify facts against authoritative sources
• Ask AI to cite sources (it can try, but verify the sources)
• For fact-dependent work, use AI as accelerator, humans as verifier

Manager use (wrong): "Look up Q3 industry benchmarks for software engineer salaries and put them in this report."

Manager use (right): "Here are Q3 benchmarks I found. Summarize this into a paragraph for our report."

UNDERSTAND ORGANIZATIONAL CONTEXT

Why it's risky:

• Proprietary information wasn't in training data
• AI doesn't know the politics, relationships, or history
• AI can't read the organization's values accurately from a single description

Better approach:

• Always provide explicit context
• Use AI to draft based on context you give it
• Have humans review anything client-facing or high-stakes for tone/appropriateness

Manager use (wrong): "Write a message to our board about why we're shifting strategy."

Manager use (right): "The board has concerns about our [specific concern]. Here's the situation [describe]. Draft a message explaining our thinking."

MAKE VALUE JUDGMENTS

Why it's risky:

• AI has no actual values or judgment
• It mimics values from training data
• High-stakes decisions need human judgment

Better approach:

• Use AI to present options and analysis
• Make judgment calls yourself
• Be transparent that you made the judgment, not delegated it to AI

Manager use (wrong): "AI, should I give Sarah the promotion or the raise? Decide."

Manager use (right): "Here are two career options I'm considering for Sarah. What should I think about? Then you decide with your judgment about what's best for her development and our team.

REMEMBER CONTEXT OVER TIME

Why it's risky:

• AI doesn't learn from conversations
• Context windows have limits
• Tomorrow's conversation starts fresh

Better approach:

• Save important information
• Repeat key context in subsequent conversations
• Document decisions so you can reference them

What AI Should NOT Do (Tier 4 - Don't Use)

These are tasks where AI should never be the primary decision-maker, and where human verification is insufficient.

PERFORMANCE EVALUATION WITH CONSEQUENCES

Why not:

• Evaluation requires deep contextual knowledge
• AI can't assess potential, growth trajectory, or hidden strengths
• Biases in historical data reproduce problematic patterns
• If evaluation affects pay, promotion, or employment, this is high-stakes
• Legal and ethical responsibility is entirely on you

What you can do: Use AI to organize information ("Here's feedback about Sarah from three people. What themes emerge?"). But the evaluation judgment is entirely yours.

SENSITIVE HR COMMUNICATION

Why not:

• Requires understanding of employment law
• Requires psychological awareness (how will this land with the employee?)
• Requires organizational knowledge (what precedent does this set?)
• Errors have legal consequences
• Tone is critical and context-dependent

What you can do: Draft with AI, but have legal/HR review before sending anything about termination, accommodation, performance management, or sensitive workplace issues.

COMPENSATION DECISIONS

Why not:

• AI doesn't know market rates accurately
• AI doesn't know compensation philosophy
• Errors affect people's livelihoods
• Legal implications (equity, discrimination)

What you can do: Use AI to research general market information, then use your judgment informed by your organization's compensation approach.

DECISIONS WITH IRREVERSIBLE CONSEQUENCES

Why not:

• You can't undo some decisions
• If high stakes, human judgment is essential
• AI has no accountability for the consequences

Examples to avoid delegating:

• Who to hire/fire
• What to communicate publicly about serious issues
• Data privacy decisions
• Ethical trade-offs
• Strategy changes with major impact

What you can do: Use AI to analyze, summarize, and present options. You make the decision.

ANYTHING WITH REAL LEGAL CONSEQUENCES

Why not:

• AI isn't a lawyer
• Legal advice is protected professional work
• AI can confidently advise incorrectly
• You have liability for the decision

What you can do: Use AI to summarize legal concepts or draft documents for your lawyer to review. Never treat AI as legal counsel.

DECISIONS REQUIRING ETHICAL JUDGMENT

Why not:

• Ethics requires values and judgment
• AI has no genuine values
• High-stakes ethical decisions need human reasoning
• You bear responsibility

What you can do: Use AI to think through implications ("If we take approach X, what might the consequences be?"). The ethical judgment is yours.

Practical Managerial Use Cases

Case Study 1: Performance Review (What NOT to Do)

Scenario: Your organization has collected feedback on a direct report. You ask AI to write the review.

What goes wrong:

• AI can write something grammatically perfect and completely miss the actual issues
• AI might reproduce biases from how reviews are typically written
• If the review is negative, it needs your empathy and knowledge of context
• You're responsible for the contents and consequences—AI isn't

Better approach:

• Provide context to AI: "Here's specific feedback from three people [list it]. What themes do you see?"
• AI highlights themes
• You decide which themes are accurate and fair
• You write the review, informed by AI's pattern-spotting but shaped by your judgment
• Alternative: Draft with AI, then completely rewrite based on your knowledge

Case Study 2: Draft Communication (What AI IS Good At)

Scenario: You need to communicate a policy change to your team.

Better approach:

• Give AI the facts: "We're shifting the WFH policy from 2 days to 3 days. Here's the reasoning [explain]. Draft a brief announcement."
• AI generates something you review
• You adjust for tone ("More warm, less corporate") or missing details
• You send it

Outcome: 20 minutes saved, quality maintained, and you made the judgment calls about messaging.

Case Study 3: Data Analysis (Moderate Reliability)

Scenario: You have customer feedback and want to understand patterns.

Good use:

• "Here's 50 pieces of feedback. What are the five most common themes?"
• AI categorizes and summarizes
• You spot-check a few to verify categorization makes sense
• You identify which themes matter most to your business

Risk to watch: AI's themes might not align with what's most important to your business. Review and adjust.

Case Study 4: Legal/Compliance (Don't Use AI Directly)

Scenario: You're unsure about a remote work arrangement and employment law.

Wrong: Ask AI "Are we legally required to provide a laptop for remote workers?"

Right: Ask your legal/HR team. After they advise, use AI to help you understand the concept: "Help me understand the distinction between contractor and employee for tax purposes."

Anti-Patterns / Misuse Risks

Risk 1: Treating AI Output as Final

"The AI wrote it, so it's done."

Why it fails: AI output is almost always a draft, not a final product. For anything high-stakes, you need review.

Risk 2: Expecting AI to Know Your Organization

"AI should understand our culture."

Why it fails: Your organization's culture isn't in the training data. You have to explain it.

Risk 3: Asking AI for Judgment on Subjective Matters

"The AI should decide if this email sounds professional enough."

Why it fails: AI can analyze patterns of professional writing, but "appropriate for your organization's culture" requires your judgment.

Risk 4: Assuming AI Consistency

"If the AI said this yesterday, it should say it today."

Why it fails: Probabilistic generation means slight variations. No memory between sessions.

Risk 5: Over-Relying on AI for Risk Assessment

"The AI said the risk is low, so we're good."

Why it fails: AI analyzes patterns in what it was trained on. Risks specific to your situation might not be obvious from training data.

Human Judgment Checkpoints

Before using AI output, ask:

1. Is the output factual or creative? If factual, verify.
2. Does it require context AI wouldn't have? If yes, review carefully.
3. Are there consequences if it's wrong? If yes, verify or escalate.
4. Could this affect someone's career or livelihood? If yes, add multiple reviews.
5. Is this irreversible? If yes, be very careful.

The more "yes" answers, the more human judgment needed.

Responsible AI Considerations

Knowing Your Limits

Be honest about what you can verify. If you don't have expertise to verify something, don't present it as verified. "The AI drafted this. I'm not an expert in this area, so take it as a starting point, not a final source" is responsible transparency.

Maintaining Accountability

Remember: You're responsible for anything you send that was AI-assisted. The AI has no accountability. That's on you. This should inform how much review you do.

Communicating Capability Honestly

When pitching AI to your team, be honest: "AI is great at summarizing and drafting. It's not good at understanding our specific situation or making judgment calls. We use it to save time on the first draft."

Practice / Reflection Prompts

1. Your Current Work: List 5 recurring tasks you do. For each, is it Tier 1 (use confidently), Tier 2 (use with review), Tier 3 (use cautiously), or Tier 4 (don't use)?

1. Verification Plan: For a task you want to delegate to AI, what would verification look like? What could go wrong? How would you catch it?

1. Context and Judgment: Think of a decision you're facing. What context would AI lack? What judgment is required?

1. Team Communication: How would you explain to your team what AI is good at and bad at in your organization?

1. Risk Assessment: What's one task that sounds like AI could help but actually carries hidden risk? Why?

Key Takeaways

1. Match tasks to reliability tiers. Use AI where it excels (drafting, summarizing). Avoid or heavily review where it's risky (judgment, accuracy).
2. Tier 1 tasks are safe and high-value. Drafting, summarizing, and extracting are where AI shines.
3. Tier 2 tasks need review. Analysis and planning are useful with human oversight.
4. Tier 3 tasks need careful judgment. Some analytical and decision work can use AI as input, but you decide.
5. Tier 4 tasks are off-limits. High-stakes decisions, legal/ethical issues, and irreversible consequences belong to humans entirely.
6. Context is critical. AI lacks organizational knowledge. Explicit context is always necessary.