Overview

Lecture URL: https://skill.re/learn/manager/coaching-team-members-on-ai-quality-standards.php

AI FOR MANAGERS CERTIFICATION

Responsible AI Oversight (Level 2) | Responsible AI Oversight

LECTURE: Coaching Team Members on AI Quality Standards

Lesson 2.5.3 | Estimated Duration: ~22 minutes

Welcome to the AI for Managers certification program. I am your instructor, and today we are covering one of the most important lessons in the Responsible AI Oversight module: Coaching Team Members on AI Quality Standards.

This is Lesson 2.5.3 in Level 2, the Responsible AI Use track. Whether you are managing a team newly using AI tools, a director building organizational AI capability, or a VP responsible for quality standards, the material in this session is designed to meet you where you are.

In Level 1, we covered how to select and adopt AI tools. Now in Level 2, we focus on ensuring your team uses those tools responsibly and produces high-quality outputs.

The goal is not to become a micromanager policing every AI use. The goal is to build your team's judgment about when to use AI, how to verify its output, and when to escalate concerns.

Before we begin, I encourage you to think about a tool your team uses and whether people are using it in ways you would expect or approve of. By the end of this session, you will have coaching frameworks and techniques.

Let us get started.

Lesson 2.5.3: Coaching Team Members on AI Quality Standards

Purpose

Your team now has access to AI tools. The question is: Are they using them well? Are outputs meeting quality standards? Are they using AI appropriately, or are there edge cases where AI should not be used?

This lesson teaches you how to coach your team toward consistent, responsible AI use.

Why This Matters for Managers

A common mistake is assuming that people will automatically use AI tools well. In reality, people need coaching. Some will over-rely on AI and not verify outputs. Others will use AI for things it should not be used for. Some will not use it at all because they misunderstand when it is appropriate.

The stakes include:

• Quality (if AI outputs are not verified, quality suffers)
  - Risk (if people use AI inappropriately, organizational risk increases)
  - Team learning (without coaching, people do not develop judgment)
  - Efficiency (people who use AI well are much more productive)

Your job is to build your team's judgment through coaching.

Establishing Quality Standards

Step 1: Define What "Good" Looks Like

Before coaching others, you need to define what good AI use looks like for your team and context.

Questions to answer:

• For outputs from AI, what level of accuracy is required? 100%? 90%? 80%?
  - What verification process is necessary before using AI output?
  - Which tasks are safe to use AI for? Which are not?
  - What kind of output editing/refinement is acceptable vs. required?
  - How should people document that they used AI?

Example standards for email drafting:

• AI drafts emails. You review and edit for tone and accuracy. You are responsible for the final message.
  - For routine emails (status updates, scheduling), 1-2 minutes of review is typical.
  - For sensitive emails (difficult feedback, policy announcements), full rewrite may be necessary.
  - You may use AI but must verify all facts against source documents.

Example standards for code review AI:

• AI flags potential issues. You manually review all flagged code sections.
  - AI is not a substitute for human code review. It is an accelerator.
  - For security-critical code, additional human review is required beyond the AI output.
  - Document in code review comments if AI was used as part of the review.

These standards become the foundation for your coaching.

Step 2: Communicate Standards Clearly

Do not assume people understand standards. Communicate explicitly.

Format: One-page standard for each common AI use case in your team.

Example document:

"AI Email Drafting Standard

• Use AI to draft initial emails for routine communication
  - Always review the draft for tone, accuracy, and completeness
  - For sensitive topics (performance, legal, major announcements), rewrite as needed
  - You are responsible for the final email. It represents you.
  - Estimated time savings: 10-15 minutes per email
  - If the draft is wrong more than 20% of the time for a use case, let's discuss a different approach"

Share these standards with your team. Post them. Reference them regularly.

Step 3: Coach by Example

The most powerful coaching is showing, not telling.

Demonstrate:

• Draft an email using AI. Show the team your process: prompt -> output -> review -> final email
  - Explain your thinking: "I used AI here because it is a routine email. I spent 2 minutes reviewing for tone. I changed one phrase."
  - For a sensitive email: "This required more thought. I rewrote most of it because the AI draft was too formal for our culture."
  - For code: "The AI flagged this potential null pointer issue. Let me check... yes, that is a real issue. Good catch by the AI."

When you demonstrate, the team sees not just what good looks like, but your reasoning.

Identifying and Addressing Common Issues

Issue 1: Trusting AI Output Without Verification

Symptom: A team member uses AI output directly without review.

Why it happens:

• They are busy and take shortcuts
  - They trust AI more than they should
  - They do not understand the quality requirements
  - They have not developed a verification habit

How to coach:

1. Point out the issue gently: "I noticed you used the AI draft directly. Did you have a chance to review it?"
2. Show the problem: "Look at this section. The AI got the fact wrong. It sounds right but it is incorrect."
3. Teach the habit: "Here is the process: Draft with AI, then verify against your source material."
4. Make it easy: "Set a 3-minute review as a habit. Read through once for accuracy."
5. Follow up: Check in later. "How is the review habit going? Finding issues?"

Issue 2: Using AI for Inappropriate Tasks

Symptom: A team member uses AI for something it should not be used for.

Examples:

• Using AI to write performance reviews without human judgment
  - Using AI for sensitive HR decisions
  - Using AI with proprietary or confidential data
  - Using a consumer AI tool for regulated industry work (HIPAA, financial)

How to coach:

1. Understand why: "Help me understand why you used AI for this. What problem were you solving?"
2. Make it clear going forward: "Going forward, please check with me before using AI for sensitive HR or compliance work."
3. Document the standard: Add this to your team's AI use standards.

Issue 3: Over-Reliance on AI

Symptom: Someone is using AI for things they could do faster themselves.

Example: A manager asks AI to draft a 3-line message that takes longer to prompt than to write.

How to coach:

1. Point out the inefficiency: "I noticed you used AI to draft that message. Did it save time compared to writing it yourself?"
2. Build judgment: "AI is great for longer drafts and complex writing. For short messages, you are usually faster. Judgment here matters."
3. Help them decide: "When in doubt, ask: Will this take more than 5 minutes to write? If yes, use AI. If no, just write it."
4. Build the habit: "Over time, you will develop a feel for when AI is worth using. It comes from experience."

Issue 4: Not Documenting AI Use

Symptom: Someone uses AI but does not disclose it.

Why it matters:

• For regulatory/compliance work, disclosure might be required
  - For client work, transparency is important
  - For team context, knowing AI was used informs how much to trust the output

How to coach:

1. Explain why: "When you use AI, it is good practice to note it. For our clients, transparency matters. For our team, it helps context."
2. Model it: Do this yourself. Show the example.
3. Build the habit: Remind early. It becomes automatic.

Issue 5: Insufficient Skill Development

Symptom: Someone uses AI but never gets better at prompting or verifying.

Why it happens:

• They have not been taught good practices
  - They do not know advanced features of the tool
  - They have not practiced enough to develop judgment

How to coach:

1. Assess: "Tell me how you typically use the AI tool. Walk me through an example."
2. Identify gaps: "I notice you always use the default settings. Let me show you a trick..."
3. Teach: Show them how to write better prompts, how to iterate, how to verify more efficiently
4. Build on success: "That worked well! Here's the next level of sophistication..."

Coaching Frameworks

Framework 1: The Recognition Conversation

Use this when you see something going well:

"I noticed you used AI to draft the client proposal. I liked how you verified all the facts against our project docs before sending. That is exactly the right approach. The quality of your output is strong. Keep doing that."

Why this works:

• Recognizes the specific behavior (verification)
  - Explains why it is right (quality, reliability)
  - Encourages continuation

Framework 2: The Redirecting Conversation

Use this when you see something that needs adjustment:

Why this works:

• Acknowledges their thinking (easy tool)
  - Explains the constraint (compliance, trust)
  - Provides a clear alternative

Framework 3: The Teaching Conversation

Use this when someone is missing a skill:

Why this works:

• Shares a specific skill
  - Demonstrates with example
  - Creates opportunity for practice

Framework 4: The Judgment Development Conversation

Use this when someone needs to develop judgment:

Why this works:

• Invites their thinking
  - Validates their uncertainty as normal
  - Shares your reasoning
  - Positions judgment development as ongoing

Building Team Culture Around Quality

Beyond individual coaching, build a team culture around quality:

Normalize discussion:

• In team meetings, ask: "Who has been using AI? What have you learned?"
  - Share examples: "Sarah found a clever way to use AI for customer summaries. She reduced time by 50%. Sarah, will you share what you did?"
  - Celebrate learning: "We made mistakes with AI last month and corrected course. That is how we get better."

Create peer coaching:

• Pair experienced AI users with newer users
  - Have experienced users review AI outputs of newer users
  - Create a culture where people ask each other for feedback

Build standards incrementally:

• Start with loose standards
  - As you learn what works, tighten standards
  - Involve the team in updating standards
  - Communicate updates: "Based on what we have learned, here is how we are refining our standards..."

Document and share learning:

• When someone discovers a useful technique, document it
  - When someone makes a mistake, discuss it (without blame)
  - Create a team knowledge base: "Here are the most effective ways we use AI tools"

ANTI-PATTERNS

Anti-Pattern 1: Overly Restrictive Standards

"Nobody can use AI without manager approval."

Why it fails: You become a bottleneck. Adoption slows. People lose autonomy.

Better: Set clear standards. Trust people to follow them. Spot-check periodically.

Anti-Pattern 2: No Standards at All

"Everyone use AI however they want."

Why it fails: Quality suffers. Some people use it inappropriately. Organizational risk increases.

Better: Clear standards communicated to everyone. Coaching to ensure compliance.

Anti-Pattern 3: Coaching in Public When Correcting

"You used AI wrong. Here is what you should have done." (Said in a team meeting)

Why it fails: Person feels embarrassed. They become defensive. Learning is slower.

Better: Private conversation first. Redirect respectfully. Make it a learning moment, not a criticism.

Anti-Pattern 4: Assuming Understanding

"I explained the standard once. Everyone should understand it now."

Why it fails: Understanding develops over time with practice and feedback, not just explanation.

Better: Communicate standard, then coach individuals as they practice.

Anti-Pattern 5: Not Adjusting Standards as You Learn

"We set these standards at the beginning. We are sticking with them."

Why it fails: You learn new things about what works and what does not. Standards should evolve.

Better: Review standards quarterly. Adjust based on learning and feedback.

PRACTICE PROMPTS

1. Standard Definition: For a common AI use case in your team, write a one-paragraph standard defining what good looks like, what verification is needed, and what is off-limits.
2. Issue Identification: Think about your team's current AI use. What issue (from the list above) are you seeing? Draft how you would coach someone on that issue.
3. Demonstration Plan: Plan how you would demonstrate good AI use for your team. What would you show? What would you explain about your thinking?
4. Conversation Drafting: Using one of the four coaching frameworks, draft a coaching conversation for a specific situation in your team.
5. Team Culture: Identify one way you could build peer coaching or knowledge sharing about AI use in your team.

KEY TAKEAWAYS

1. Define clear quality standards for each AI use case in your team. These are the foundation for coaching.
2. Coach by example. Demonstrate good AI use, showing your thinking and verification process.
3. Address issues early with a teaching mindset, not a blame mindset. Frame coaching as helping people develop judgment.
4. Build team culture around learning. Share discoveries. Celebrate good use. Discuss mistakes without blame.
5. Evolve standards over time. As your team gets better at AI use, your standards can become more sophisticated.

GLOSSARY

Quality Standard: A defined expectation for acceptable quality and verification of AI output for a specific task.

Verification: The process of checking AI output for accuracy, appropriateness, and alignment with standards before using it.

Judgment: The human ability to decide when AI should be used, how much to trust its output, and when human oversight is essential.

Coaching: A development conversation focused on building skills, judgment, and understanding rather than correcting mistakes.

Peer Coaching: Team members helping each other learn and develop. Often more effective than manager-only coaching.

[SYNTHESIS AND APPLICATION]

Let us step back and look at the bigger picture of what we have covered in this session on Coaching Team Members on AI Quality Standards.

Many managers adopt AI tools, then assume people will use them well. But responsibility for quality and appropriate use requires active coaching and culture building.

Here is what I want you to take away from this session:

First, standards. Define what good looks like for your team's context. Different teams have different standards. That is okay.

Second, coaching. Use coaching frameworks to help people develop judgment, not just follow rules.

Third, culture. Build an environment where people ask questions, share learning, and help each other develop capability.

[REFLECTION EXERCISE]

Before we close, I would like you to spend two minutes on this reflection:

Think about your team's current AI use. What is one thing going well that you want to recognize? What is one thing that needs coaching? How would you approach that coaching conversation?

Write down your answer. That reflection guides your next coaching effort.

[CLOSING REMARKS]

In our next lesson, we will explore establishing review checkpoints in workflows, which is a complementary approach to coaching: building systems and processes that ensure quality and responsibility.

This has been Lesson 2.5.3: Coaching Team Members on AI Quality Standards, part of the Responsible AI Oversight module in Level 2: Responsible AI Use of the AI for Managers certification.

Remember: the goal is not to police your team's AI use. The goal is to help them develop the judgment to use AI well, responsibly, and effectively.

Thank you for your time, your attention, and your commitment to coaching your team toward responsible, high-quality AI use.

END OF TRANSCRIPT

AI for Managers Certification Program

Level 2: Responsible AI Use | Responsible AI Oversight | Lesson 2.5.3

A SkillsClinic initiative by No Worker Left Behind and The Work Company.

Duration: ~22 minutes | Word Count: ~3447