Lesson 3.3: Evaluating AI Output

Purpose

You've written a prompt and gotten a response. Now you need to assess: Is this good enough to use? What needs fixing? What should I watch for?

This lesson teaches the verification habit—the human judgment layer that makes AI safe and useful.

Why This Matters for Managers

This is critical. The biggest mistake is trusting AI output without checking it.

Poor verification leads to:

• Sending emails with tone-deaf sections
• Presenting information that's slightly wrong
• Making decisions based on inaccurate summaries
• Damaging credibility when AI mistakes are discovered

Good verification means you catch problems before they escape.

Core Concepts

The Verification Checklist

Before using AI output, run through this checklist. It takes 2-5 minutes and saves mistakes.

Dimension 1: Accuracy

The question: Is the factual content correct?

How to check:

• For facts you know well, verify accuracy
• For facts you're not sure about, be skeptical (might be hallucinated)
• For statistics or numbers, extra scrutiny (AI confidently invents numbers)
• For content based on current events, verify (AI training data is old)

Red flags:

• Specific numbers that sound plausible but you're not sure about
• References to events or studies you don't remember
• Very specific claims ("The average manager works 47 hours per week" - is this made up?)

What to do:

• For factual claims, verify against reliable source
• If uncertain, note the claim in brackets: "[VERIFY: AI says X but I should check this]"
• Never send out factual claims you haven't verified (especially if they'll be read by others)

Example:

AI writes: "Research shows 73% of managers report feeling overwhelmed."

Your check: I've never seen this number. It sounds plausible but might be fabricated. What should I do? Either verify it with a search, or rewrite: "Many managers report feeling overwhelmed" (which doesn't claim a specific number).

Dimension 2: Tone and Appropriateness

The question: Does this sound right for the context? Is it too formal, too casual, too harsh, too soft?

How to check:

• Read it once as someone receiving it
• Ask: "Does this match my voice and the relationship?"
• Imagine the recipient reading it

Red flags:

• Sounds corporate or generic
• Missing warmth in a context where warmth matters
• Too direct or too soft for the situation
• Sounds "AI-like" (overly polished, slightly off)

What to do:

• If tone is off, rewrite or ask AI to adjust ("Make this warmer" or "This is too formal")
• Don't send if tone feels wrong, even if content is right
• Remember: Tone matters more than perfection

Example:

AI writes: "It is requested that you expedite the completion of the aforementioned project."

Your check: This is way too formal. Your actual team would find this absurd. Rewrite or ask AI: "Make this more conversational and friendly."

Dimension 3: Completeness

The question: Did AI miss anything important? Is anything left unsaid?

How to check:

• Compare to what you actually want to communicate
• Ask: "If I sent this, would the recipient have everything they need?"
• Are there gaps in logic or missing steps?

Red flags:

• Key information that should be included but isn't
• Instructions that skip steps
• Information that would confuse without context
• No clear call to action when one is needed

What to do:

• Add missing information
• Rearrange to make flow clearer
• Ask AI to expand ("This is good but missing [specific thing]. Can you expand?")

Example:

AI drafts meeting agenda. You notice: "This includes the topics but no time allocations. Without timing, people won't know if they have 5 minutes or 45 minutes per topic. Let me add that."

Dimension 4: Relevance to Your Context

The question: Does this account for your specific situation, relationships, or constraints?

How to check:

• Does it know about your team dynamics?
• Does it reflect your organization's culture?
• Does it account for constraints or issues you know about?

Red flags:

• Suggests something that would never work in your organization
• Misses a key constraint or relationship
• Sounds generic, not customized to you
• Ignores something obvious about your situation

What to do:

• Customize and add your context
• Rewrite sections that miss your situation
• Ask AI for revision with more context

Example:

AI suggests a brainstorming approach. But you know your team: "This suggestion assumes people are motivated by competition. My team is collaborative. This approach would backfire. Let me reframe it."

Dimension 5: Alignment With Your Values

The question: Does this communicate what you actually believe or want to communicate?

How to check:

• Read it. Does it reflect your values and approach?
• Would you stand behind this if questioned?
• Does it align with how you see your role?

Red flags:

• Language that doesn't match your values
• Tone that's inconsistent with your principles
• Framing that you disagree with
• Anything you wouldn't be comfortable explaining later

What to do:

• Rewrite to match your values and voice
• Don't send if it feels inauthentic
• Use AI output as first draft, not final

Example:

AI drafts feedback: "Your performance has been inconsistent and below expectations in several areas."

Your check: This is harsh and doesn't match my coaching approach. I'd say: "I've noticed some inconsistency. Let's talk about what's going on and how I can support you better."

The Verification Speed

Good news: You get faster at verification.

First time: 5-10 minutes (reading carefully, checking multiple dimensions)

After 10 attempts: 2-3 minutes (you know what to look for)

After 50 attempts: 30 seconds to 1 minute (you spot problems quickly)

The habit of verification is quick once you develop it.

Practical Managerial Use Cases

Case Study 1: Verifying an Email Draft

Scenario: You asked AI to draft an email to your team about a deadline change.

AI output:

"Hello team, due to unexpected circumstances, we have had to modify the project deadline. The new completion date is Friday, March 20th. This may require some adjustment to your schedules. Please update your plans accordingly and reach out if you have concerns."

Verification checklist:

1. Accuracy: Is "Friday, March 20th" correct? Check calendar. Yes. Are the facts right? Yes, deadline did move. Check mark.

1. Tone: Is this appropriate? For your team, you usually sound warmer and more collaborative. This sounds corporate. Missing personal touch.

1. Completeness: Does this explain WHY the deadline moved? Does it acknowledge the impact on the team? Does it offer support? Missing some of this.

1. Context: Does this reflect your team's situation? You have three people who are already overloaded. Does this acknowledge that? No.

1. Values: Would you actually say "unexpected circumstances"? Not really. You're usually more direct about what's happening.

Your decision: Good skeleton, but needs customization. You rewrite:

"Team, we need to adjust the project deadline to Friday, March 20th. I know you're already pushing hard, and this puts more pressure on everyone. Here's why we need to move it [explain reason]. I want to help figure out what we can do to make this manageable. Let's talk about scope, resources, or whatever support you need. Thanks for staying flexible."

Result: Much better. Reflects your voice, acknowledges impact, offers support.

Case Study 2: Verifying a Summary

Scenario: You asked AI to summarize feedback from customer meetings.

AI output:

"Customers express satisfaction with product features and pricing. Main requests are for additional integrations and faster support response time. One customer mentioned invoicing updates."

Verification checklist:

1. Accuracy: Was "one customer mentioned invoicing"? You attended the meetings. Actually, invoicing came up multiple times and seemed important. AI missed the significance.

1. Tone: N/A (it's a summary)

1. Completeness: Did you ask for specific categories? You wanted: (1) what's working well, (2) what needs improvement, (3) what customers want next. This summary doesn't organize around those categories.

1. Context: Did it capture the tone/sentiment? Three customers were frustrated about support times. This summary makes it sound like a minor mention.

1. Values: N/A

Your decision: Accurate on surface, but misses what actually matters. You ask AI: "I want more nuance. Support response time came up as a pain point, not just a request. Can you reorganize around: (1) What's working well, (2) What's frustrating them, (3) What they want next? Add sentiment too—what did they sound frustrated about vs. neutral about?"

Result: AI provides better-organized summary with sentiment.

Case Study 3: Verifying Analysis

Scenario: You asked AI to analyze feedback themes from your team.

AI output:

"Main themes: 3 people mentioned wanting more clarity on career paths (27%), 2 people wanted better work-life balance (18%), 1 person wanted higher compensation (9%), and miscellaneous other concerns (46%)."

Verification checklist:

1. Accuracy: Let me check. Yes, 3 people did mention career clarity. 2 mentioned work-life balance. Percentages add up. Accurate.

1. Tone: N/A

1. Completeness: Wait, is "miscellaneous other concerns" really the majority of feedback? That seems like AI missed something. Let me read the original feedback. Actually, several people mentioned wanting more feedback and more connection time with you. Why isn't that a category?

1. Context: The "miscellaneous" concerns are actually important. If you miss them, you'll miss what your team actually needs.

1. Values: N/A

Your decision: AI missed important patterns. You ask AI: "I noticed you grouped many feedback items as miscellaneous. Can you tell me specifically what those were? I want to make sure we're not missing key themes."

Result: AI clarifies and reveals that feedback/connection is actually a major theme, similar in importance to career clarity.

Anti-Patterns / Misuse Risks

Misuse Risk 1: Skipping Verification to Save Time

"AI's usually good. I'll just send this without checking."

Why it fails: One un-verified mistake can damage credibility or cause real problems.

Misuse Risk 2: Trusting AI on Factual Claims

"AI said this statistic, so it must be true."

Why it fails: AI confidently generates false statistics. This is a known problem.

Misuse Risk 3: Not Customizing When Needed

"AI's output is good enough. I'll send it as-is."

Why it fails: Generic output doesn't reflect you, your values, or your situation. It feels inauthentic.

Misuse Risk 4: Assuming AI Understands Your Context

"AI knows what I need. This must be what I wanted."

Why it fails: AI doesn't know your team, your situation, or your values unless you explained them.

Human Judgment Checkpoints

As you verify AI output, ask:

1. Accuracy: Are the facts correct? Should I verify before using?
2. Tone: Does this sound like me? Does it feel right for this context?
3. Completeness: Is anything missing? Did AI understand what I wanted?
4. Context: Does this account for my specific situation?
5. Authenticity: Would I be comfortable putting my name on this?

If you answer "no" to any question, refine before using.

Responsible AI Considerations

Building the Verification Habit

Verification is a muscle. The first few times take longer. But it's critical for using AI responsibly.

Recognizing When to Start Over

Sometimes AI output is so far off that refining it takes longer than starting fresh. Recognize when to abandon a prompt and try again.

Maintaining Accountability

Your verification is your safeguard. You're responsible for what you send. Verification is how you maintain that responsibility.

Protecting Your Reputation

A single AI mistake that you didn't catch can damage credibility. Verification protects your professional reputation.

Practice / Reflection Prompts

1. Verification Practice: Take an AI output you've recently received. Run through the five-dimension checklist. What would you change?

1. Accuracy Check: Find a factual claim in an AI output. Can you verify it? Is it actually true?

1. Tone Audit: Read an AI output as if you received it. Does it sound like it came from someone you know? What's missing?

1. Context Customization: Take AI output and customize it for your specific situation. How much better is it after customization?

1. Time Investment: How long did verification take? Was it worth it to catch problems?

1. Your Red Flags: What are the most common problems you find in AI output? Make a personal checklist.

Key Takeaways

1. Always verify before using. Even good AI output needs a human check.
2. Use the five-dimension checklist: Accuracy, tone, completeness, context, values.
3. Expect to customize. AI output is rarely perfect for your situation.
4. Verify facts especially carefully. AI confidently invents numbers and statistics.
5. Fast verification is possible. After a few rounds, you'll spot problems quickly.
6. Verification is not optional. It's the safeguard that makes AI use responsible.