Overview

The organizational mistake that dooms AI: You build a "Center of Excellence." It's separate from product teams. It does research. It builds frameworks. Six months later, the product teams ignore it and build their own AI solutions in a dozen different ways.

Or worse: the CoE becomes a silo. It builds amazing things that nobody uses. It becomes theater.

The winning structures are different. They're organized around problems, not around "AI." They have power and responsibility. They ship.

This is about building organizational structures that actually move the needle.

The Wrong Way to Organize (and Why It Fails)

The Centralized CoE Model:
- Single "AI team" in a separate org
- Reports to CTO or VP of Engineering
- Builds ML platforms and makes them available to product teams
- Product teams "can use" the AI but don't have to

Why it fails:
- Product teams have incentives to move fast, not use the "right" solution
- CoE becomes a service org with no power
- CoE builds for the generic case, not specific problems
- Knowledge doesn't spread. AI becomes mysterious.
- When CoE fails, product teams are stuck

Case Study: Centralized Failure: A large fintech company built a separate ML/AI org of 15 people. They created a beautiful feature store, built reusable models, documented best practices. Product teams ignored them. One team built their own recommendation system because it was faster than getting approval from the CoE. Another team shipped a customer segmentation model that violated the CoE's bias testing standards. Six months in, the CoE realized no one was using their platform. They had built for a centralized, coordinated company. But the company was decentralized and empowered. The CoE eventually merged back into product, much smaller, with less ambitious scope.

The Siloed Teams Model:**
- Every product team has its own AI person
- No coordination
- Lots of reinvention
- Inconsistent approaches
- Technical debt everywhere

Why it fails:
- Duplication of effort
- Inconsistent quality
- Expensive (hiring 50 people instead of 10)
- Hard to retain talent (small teams are less interesting than big teams)

Case Study: Siloed Failure: A SaaS company with 4 product teams each hired an ML engineer. All four built ranking systems. All four used different approaches (one used Elasticsearch, one used a custom tree search, one used a neural network, one used heuristics). When customer needs changed, they couldn't share solutions. When one engineer left, their team was stuck. Cost: 3-4 engineers worth of effort duplicated. Quality: inconsistent, so the product felt disjointed.

The Theater Model:**
- You have an AI strategy
- You announce Center of Excellence
- You hire a VP
- You build frameworks and platforms
- Nothing actually ships
- The VP leaves and the whole thing collapses

Why it fails:
- No accountability for results
- Status over substance
- Organizational resistance
- Board pressure creates pressure to announce instead of to ship

Case Study: Theater Failure: A B2B company hired a VP of AI with a mandate to "build the center of excellence." She brought in 6 people. Spent 3 months designing governance. Built a beautiful data architecture. Created an ML platform. In month 6, leadership asked: "Where's the business impact?" The CoE had shipped zero customer-facing capabilities. They had built infrastructure that no one was using. The VP left. The team scattered. The infrastructure sat unused.

The Organization Principle:** Organize around problems, not technology. Give AI teams budget and accountability. Measure them on shipped capabilities, not on frameworks built.

The Winning Structures

Model 1: Embedded AI Team (Small Companies)

You have 5-20 engineers total. You hire 1-2 AI specialists. They work directly with product teams. No separate CoE.

Structure:
- AI specialist reports to product manager (not separately to engineering)
- They work on real product problems with timelines and OKRs
- They upskill the team around them
- They set standards but ship, not just design

Advantages:**
- Fast. No process overhead.
- Accountability is clear.
- AI is connected to product.
- Spreads knowledge quickly.

Disadvantages:
- Limited scale for very complex AI work
- Knowledge could be more systematized

When to use it:** Companies under 100 engineers. You're building AI features, not an AI company.

Case Study: Startup Using Embedded Model

An early-stage B2B SaaS company (12 engineers) hired one AI engineer to add smart recommendation features. She didn't form a team. She worked with product, sitting in their standups, shipping features on their roadmap. Within 6 months: deployed recommendations, built user segmentation, added predictive churn. She upskilled the team, senior engineers paired with her, learned the patterns. By year two: hired second AI engineer, not as manager, but as peer. Team had shipped 8 AI features. By year three: 4 out of 12 engineers could build AI features competently. The two AI specialists shifted to harder problems (fine-tuning on company data). AI was embedded in product DNA, not siloed. CoE wasn't a meeting. It was how they worked. Recruiting became easier, people wanted to work there because they'd learn AI on the job.

Model 2: Hub and Spoke (Medium Companies)

You have 100-500 engineers. You have 1-2 core AI teams (the hub). Product teams have AI-focused people (the spokes).

Structure:
- Core AI team: 5-10 people doing the hardest AI work, building infrastructure, setting standards
- Spoke people (1-2 per major product area): embedded in product teams
- Hub and spoke meet weekly to sync, share learning
- Hub has authority on technical decisions (standards, model selection, etc.)

Governance:**
- Hub owns: model selection, architecture patterns, data pipelines, monitoring standards, evaluation frameworks
- Spokes own: product-specific features, local optimization, user experience
- Both own: shipping working systems

Advantages:**
- Scales well (you're not hiring a separate AI person per feature)
- Consistency without stifling
- Knowledge flows both directions
- Spokes feel empowered, hub provides support

Disadvantages:**
- Requires clear governance (or it falls apart)
- Spoke people need to have foot in both worlds

When to use it:** Companies with 100-500 engineers building multiple products where AI matters in each.

Case Study: Hub and Spoke Success

A Series B company (80 engineers, 3 product areas) started with AI siloed in individual teams. Each team picked different models, different data approaches, different monitoring. Teams were reinventing solutions. Progress was slow. They created a hub (4 people) and embedded spoke people (1-2 per major product area). Hub: two senior ML engineers, one data engineer, one ML Ops person. Spokes: product engineers with AI interest.

Within 6 months: standardized on shared infrastructure (feature store, model evaluation framework, monitoring). Reduced duplication, the company went from 8 models to 4 cohesive models across products. Feature shipping doubled. Spoke people learned from hub expertise but stayed connected to product (reported to product PM, not hub). Hub people shipped features, not just frameworks (hub engineers spent 30% time on product features, 70% on platform). By year two: gone from 3 distinct AI architectures to 1 unified platform. Doubled shipped AI capabilities. Hub became trusted, product teams wanted hub input because the hub shipped things, not just designed things.

Model 3: Product AI Teams (Large Companies)

You have 500+ engineers. Each major product area gets dedicated AI investment. There's a loose AI guild for consistency.

Structure:
- Each major product (search, recommendations, customer support) gets dedicated AI team (5-15 people)
- These teams report into the product org, not a separate AI org
- AI guild (4-6 senior people) meets monthly to share patterns and set standards
- Guild doesn't have authority to block decisions, but has influence through expertise and examples

Governance:**
- Product teams own their AI decisions
- Guild provides guidance on standards, tools, approaches
- Guild escalates conflicts (usually about model selection or data strategy)

Advantages:**
- Scale (AI is embedded in product DNA)
- Each team optimizes for their specific problem
- Talent stays engaged (building interesting products, not frameworks)
- Clear accountability

Disadvantages:**
- Risk of divergence (different models, different data pipelines)
- Less knowledge sharing than hub and spoke
- Requires strong senior people in guild role (they can't be deadweight)

When to use it: Large companies where AI is product differentiation. Companies where different product areas have very different AI needs.

The Governance Layer (That Doesn't Strangle Innovation)

You need some governance. But governance kills innovation if it's too heavy.

What to Govern (and How Lightly)

Model selection:
- Standard: use proven models (Claude, GPT-4, Llama)
- Exception: use new models only after testing
- Who decides: Hub lead + product lead
- Friction: low, requires written justification
- "Low friction" means: 2-3 days max for decision

Data strategy:
- Standard: use company data warehouse with feature store
- Exception: use external data only with legal/compliance review
- Who decides: Hub lead + data team
- Friction: medium, requires data agreement
- Timeline:
The Governance Principle: Make governance lightweight and fast. It's about consistency and avoiding mistakes, not about process. If it takes more than 2 weeks to get approval, it's too heavy.

Avoiding the CoE Theater Trap

Red flags that you're building theater:**
- Your CoE spends more time on frameworks than on shipping
- People talk about the "Center of Excellence" but most teams ignore it
- You've got a VP but no shipped capabilities in the last 6 months
- The hub builds things that only the hub uses
- You hired people who are good at PowerPoint but not good at shipping

How to stay real:**
- Measure the CoE on shipped capabilities, not on frameworks
- CoE people spend 50% of their time working directly on product problems
- Friction to use CoE standards is low (less than 20% overhead)
- When the CoE is wrong about something, product teams can override (with documentation)
- Senior engineers in the hub have to code, not just architect

The test: If you removed the CoE tomorrow, would product teams be able to ship AI features? If the answer is "no," you've built a critical dependency. If the answer is "yes but slower," that's right. That's the sweet spot.

Failure Modes: When Centers of Excellence Go Wrong

Failure Mode 1: The Invisible CoE

You build a beautiful platform. Governance frameworks. Standard patterns. Shared infrastructure. No one uses it. Why? The CoE didn't address real pain. Product teams solved problems faster with direct integration. The platform didn't make them faster, just more bureaucratic.

Antidote: Build platforms that solve real problems. Start with one team that has concrete pain. Fix their problem fast. They become advocates. Expand from there.

Failure Mode 2: The Bottleneck CoE

Everything goes through the CoE. Can't ship a model without hub approval. Can't pick a model without a committee. Hub approval takes 6 weeks. Product teams are blocked.

Antidote: Governance should accelerate shipping, not slow it. Hub approval should take 2-3 days for most things. Only high-risk decisions get long reviews. Default to "yes, unless there's a real problem."

Failure Mode 3: The Divorced Hub

Hub people build infrastructure. Product people ship features. Hub and product never talk. Hub builds for an imaginary future. Product teams don't use it. Hub becomes resentful ("they don't appreciate our work"). Product teams resent hub ("they don't understand real problems").

Antidote: Hub engineers work directly on product problems 30-50% of the time. They stay connected. They understand what's hard. They build things product teams actually want to use.

Reorganizing from a Bad Structure

Maybe your AI organization is broken. You have siloed teams duplicating work. Or you have a centralized CoE that's become invisible. How do you reorganize without losing people and momentum?

Step 1: Be honest about the problem. "We've organized as [structure]. It's not working because [specific reasons]. We're reorganizing to [new structure]. Here's why. Here's what it means for you."

Step 2: Make it a growth opportunity, not a demotion. "We're consolidating AI engineering into a hub and spokes model. Some of you will move to the hub (more interesting technical problems). Some will become spokes (stay close to customers, more influence on roadmap). All roles are valuable. We'll place you where your skills and interests align."

Step 3: Protect seniority and compensation. When you reorganize, people worry: "Will I be demoted? Will my title change? Will my pay go down?" Address this explicitly. "Your level stays the same. Your compensation stays the same. Only your team and reporting line changes."

Step 4: Get key people involved in design. Don't reorganize to them. Reorganize with them. Ask your best AI people: "How should we organize to be effective?" They'll have good ideas. They'll also buy into a design they helped create.

Step 5: Run it like a project. Give it a timeline and milestones. "Week 1-2: Publish new org chart. Week 3-4: One-on-ones with every AI person. Week 5-6: Form new teams. Week 7+: New structure operational. Everyone knows what's happening. Surprises are minimized."

Step 6: Create early wins in the new structure. First 3-4 months, the new structure will feel disruptive. Things will move slower. Compensate by having the hub deliver something visible. "First quarter, hub delivered new feature store saving spoke teams 30% development time." That shows the new structure works.

What to Do Monday Morning

Step 1: Map your current org. Where are AI people? Are they siloed in separate teams? Embedded in product? Mixed? List them out. Map who reports to whom.

Step 2: Assess whether it's working. Are you shipping? Are people collaborating or duplicating work? Is it fast or slow? Are people happy? Interview 5 engineers and ask: "Is the AI organization helping you or slowing you down?"

Step 3: Choose your model. Based on company size and AI importance: Embedded (under 100 eng, AI is one feature type). Hub and spoke (100-500 eng, AI matters across products). Product teams (500+, AI is core differentiator).

Step 4: Design the transition. How do you move from current state to target state? How do you avoid losing people? Be explicit: "We're reorganizing from [old] to [new]. Here's what that means for you. Here's what doesn't change." Retention is critical. You can't afford to lose your AI people during the transition.

Step 5: Set up governance lightly. What needs coordination across teams? (Model selection, data strategy, monitoring standards.) What should be local decisions? (Specific feature approaches, UI design, user-facing tradeoffs.) Write it down. Don't over-govern. Set clear decision timelines (2-3 days for most decisions).

Step 6: Establish success metrics. How will you know the organization is working? Shipped features per quarter. Time from idea to launch. Developer satisfaction. Cost per capability. Pick 2-3 metrics and track them quarterly. Adjust organization based on metrics.

FAQ: Organizational Questions

Q: Should AI teams report to product or engineering?**

A: Usually engineering. But they need tight alignment with product (the PM sits in the room during technical decisions). Avoid AI teams reporting to data, analytics, or research. They'll drift away from shipping.

Q: How do we handle conflicts between hub and spoke?**

A: Hub sets defaults. Spoke can override with justified reason. Hub learns from override. Next time, maybe the default is different.

Q: What if one product team is much further ahead than others?**

A: Great. Have them lead. Put the team lead on the guild. Have them do talks. They set the standard. Bring other teams up rather than holding the leader back.

Key Takeaway

Don't build a separate AI org. Embed AI in product teams with a light coordination hub. Measure on shipped capabilities, not frameworks. Make governance lightweight and fast. Avoid theater, if the hub disappeared, product teams should still be able to ship.

Organization as Strategy

How you organize determines what you ship. Organize around problems and you ship solutions. Organize around technology and you ship frameworks nobody uses.

Get the organization right and everything else flows from that.

On This Page

Introduction
Wrong Approaches
Winning Models
Case Study: Embedded
Case Study: Hub and Spoke
Governance
Avoiding Theater
Failure Modes
Reorganizing from Bad Structure
Monday Morning Action
FAQ
Key Takeaway

Chapter Details

Part ofCh 3: AI Talent and Team Strategy