The Testing Truth

Writing tests is necessary and tedious. You write the feature, then you write tests for it, then you maintain the tests as the feature changes. The test code often exceeds the feature code in complexity. Yet many tests are mediocre: they test the happy path but miss edge cases, or they're so brittle that any refactoring breaks them. In a typical SaaS product, 40-50% of the codebase is tests, but many organizations struggle to identify which tests actually prevent bugs.

The promise of AI test generation is attractive: write the feature, the AI generates comprehensive tests, you ship. Reality is more nuanced. AI is excellent at some testing tasks and terrible at others. AI can write 100 tests for a utility function in 30 seconds. But it can't understand which tests actually matter, and 80% of those tests might be noise.

The Test Generation Reality: AI test generation is like spell-check for tests: useful for catching obvious issues, useless for understanding what you're actually trying to test. The tool works for mechanics, not semantics.

What AI is Good At

• Generating happy-path tests quickly
  - Creating test fixtures and mocks
  - Generating basic edge case tests
  - Creating test structure and boilerplate
  - Generating load tests and benchmarks
  - Testing error handling patterns

What it's not good at:

• Identifying what tests actually matter for your business
  - Understanding domain-specific edge cases
  - Testing integration scenarios
  - Evaluating whether a test is actually valuable
  - Understanding user workflows and critical paths

The best approach: AI handles the mechanical test generation. Humans provide judgment about what tests matter.

The Testing Reality: More tests aren't better. Better tests are better. AI can write a lot of tests fast. You need to determine which tests actually add value.

When to Use AI for Tests

Good Use Case 1: Utility Functions

"Generate tests for this utility function:
```javascript
function calculateCompoundInterest(principal, rate, years) {
return principal * Math.pow(1 + rate / 100, years);
}
```

Test cases:
- Normal case (positive values)
- Zero principal
- Zero years
- Negative rate
- Very large numbers
- Decimal values"

AI generates comprehensive tests that cover the obvious cases. You run them. They pass. Done. Fast way to get test coverage on well-defined functions. A fintech company used AI test generation on 40 utility functions (math, formatting, validation). It generated 480 tests in 3 minutes. After review (removing 60 duplicate or trivial tests), they had 420 valuable tests. Manual writing would have taken 8+ hours. Time saved: 7.5 hours. Bugs caught by those tests in first 3 months: 12 real edge cases that would have hit production.

Good Use Case 2: Boilerplate and Fixtures

"Generate test fixtures for our User model:
- Standard user
- Admin user
- User with no permissions
- Deleted user
- User from API response

Include factory functions to create variations."

The AI generates repetitive fixture code faster than you would manually.

Good Use Case 3: Error Handling Tests

"Generate tests for error cases in this function:
[function code]

Expected errors:
- Invalid input
- Missing parameters
- Database connection failure
- Timeout
- Authorization failure

For each, generate a test that verifies the correct error is raised."

AI systematically tests error paths that are tedious to write manually.

Bad Use Case 1: Business Logic

Don't use AI to generate tests for complex business logic. The business logic is what the tests are meant to verify. AI might generate tests that pass because the AI doesn't understand what "correct" means.

Example: AI generates a test for a payment processing function. The test might check "function returns without error" but not check "the payment amount is correct" or "the customer was charged the right amount." A team tried this and ended up with 45 passing tests for their pricing logic, but production revealed the tests never validated that customers were charged the right amount. The tests were measuring implementation, not behavior. Lesson: critical business logic needs human-written tests that verify the actual business requirements.

Bad Use Case 2: Integration Tests

Integration tests involve multiple systems. AI struggles because it doesn't understand the integration contract. You need to specify it. And once you've specified the contract, writing the test is straightforward. Example: testing database interaction with caching. AI can generate "write to DB, read from cache" but doesn't understand the contract: "if cache is stale (>5 seconds), invalidate it." That contract is domain knowledge only you have.

Bad Use Case 3: User Workflow Tests

Testing a signup flow: AI might test "user creates account" and "user logs in" as separate tests. But the critical test is "user creates account, verifies email, logs in, can use the system." That requires understanding the actual user workflow, which AI doesn't have. What if there's a race condition where the verification email is lost if the user clicks verify too fast? Or if the system doesn't sync user permissions to the cache? Those aren't edge cases. They're bugs in prod that only human judgment catches.

Test Generation Red Flag: If the test isn't obvious to a human domain expert, AI definitely won't generate it. Focus AI on mechanical tests (happy path, boundary conditions, error handling). Focus humans on critical business logic, workflows, and edge cases that require domain knowledge.

The Test Generation Workflow

Step 1: Write the Function/Feature

Code first, then tests.

Step 2: Ask AI for Test Suggestions

"Generate tests for this function. Include:
- Happy path cases
- Edge cases
- Error cases
- Boundary conditions

Here's the function:
[code]"

Step 3: Review and Filter

The AI generates 20 tests. You review:
- Which ones test things that actually matter?
- Which ones are testing the implementation instead of the behavior?
- Which ones are missing (domain-specific edge cases)?

Keep the valuable ones. Delete the cruft.

Step 4: Add Domain Knowledge

Add tests for things only you know matter:

• "This function is used in the payment path. Test that it's idempotent."
  - "This function is called with data from an untrusted source. Test with injection attempts."
  - "Users report occasional 'phantom failures' with this code. Add regression test for race condition."

Step 5: Verify Coverage

Run tests. Check coverage. Identify uncovered lines. Ask AI to generate tests for uncovered code.

Test Quality Metrics

Not all tests are equal. Some catch bugs. Some are noise.

Valuable Test Characteristics

• Tests behavior, not implementation
  - Fails when the behavior changes incorrectly
  - Passes when the behavior is correct
  - Fast to run
  - Doesn't depend on external systems (unless testing integration)
  - Has a clear purpose (comment explains why this test matters)

Warning Signs

• Tests change when you refactor (testing implementation)
  - Tests check output format instead of behavior
  - Tests depend on execution order
  - Tests are flaky (sometimes pass, sometimes fail)
  - Test name doesn't explain what it's testing

When AI generates tests with these characteristics, delete them and write better ones.

Smart Test Generation Practices

Pattern 1: Happy Path + Boundaries

"Generate tests for this function. Focus on:
- What it's supposed to do (happy path)
- What happens at boundaries (empty input, max values, negative values)
- What it should reject (invalid input, null, wrong types)

Here's the function:
[code]"

Pattern 2: Parameterized Tests

"Generate parameterized tests (multiple cases in one test) for:
[function]

Cases:
- Input: X, Expected: Y
- Input: X, Expected: Y
..."

This reduces test duplication while maintaining readability.

Pattern 3: Snapshot Tests

"Generate a snapshot test for this function's output:
[function]

This is useful for regression testing, if the output changes unexpectedly, the test fails."

Coverage Without Quality

A warning: high test coverage doesn't mean good tests. A critical insight from the test industry: coverage is necessary but not sufficient. You need coverage in the right places.

You can have 90% coverage with all tests being garbage (testing implementation details, not behavior). And you can have 70% coverage with all tests being valuable (catching real bugs, testing behavior). The difference: the 70% tests are testing critical paths and failure modes. The 90% tests might be testing getters and trivial code paths.

Real example: A web app had 92% test coverage. A security vulnerability in the auth layer wasn't caught because the tests were testing "user is logged in" but not testing "unauthorized user is rejected." The coverage was high; the test quality was low in the area that mattered. They reorganized to: 100% coverage on critical paths (auth, payments, data access), 60% on UI, 40% on admin features. Bugs in critical paths dropped 88%. Bugs in UI stayed the same.

Don't optimize for coverage percentage. Optimize for "does this test catch bugs in the parts that matter?"

Continuous Test Improvement

Tests degrade over time as code changes:

"Review these tests. Which ones are:
- Testing implementation (brittle)?
- No longer relevant?
- Missing important cases?

Suggest improvements."

Periodically audit your tests with AI assistance. Keep tests valuable.

Key Insight

AI excels at generating tests for well-defined, mechanical scenarios. Human judgment determines which tests matter and whether they're actually valuable.

What to Do Monday Morning

• Find a utility function your team wrote that's not tested. Ask the AI to generate tests. Review them. Keep the good ones. See how much faster this is than writing tests manually.
  - For a complex function with some tests, ask the AI to generate additional tests. Which ones are valuable? Which are noise? Use this to calibrate your judgment on test quality.
  - Measure your test coverage by critical path. Ask AI to help write tests for uncovered code. Prioritize testing the critical path (auth, payments, core features) to 85%+ coverage.
  - Review a brittle test suite. Ask AI: "Which tests are testing implementation instead of behavior?" Refactor those tests to test behavior instead. Measure: do the refactored tests catch more bugs?
  - Create a test generation checklist for your team. "Use AI for: utility functions, error handling, fixtures. Use humans for: business logic, workflows, domain-specific edge cases."

FAQ

Q: Should all code be tested?

A: No. Test the code that breaks things when it's wrong. Trivial code (getters, obvious implementations) doesn't need tests. Focus on: anything in the critical path (payments, auth, data), anything with complex logic, anything that's changed before, anything users rely on.

Q: What's a good coverage target?

A: 80-90% for critical paths, 60-70% for typical code, 30-50% for UI/admin. Don't obsess over the percentage, obsess over whether tests catch bugs. Coverage ≥80% but with flaky tests is worse than coverage 60% with solid tests.

Q: AI-generated tests, should I trust them?

A: Trust them to be syntactically correct and to run without errors. Don't trust them to actually test what you care about or to not have false positives. Always review: does the test check the right thing? Is it testing behavior or implementation?

Q: How often should I run tests?

A: On every commit (pre-commit hook). On every PR. Before deployment. Ideally 5 min) discourage frequent testing and should be moved to nightly/CI-only.

Q: What about test maintenance?

A: Tests are code and need maintenance. Budget 10-20% of sprint time for test maintenance. When a test breaks during refactoring: if you're testing implementation, fix the test. If you're testing behavior, fix the code. If you're not sure, the test is probably bad.

Q: How do I know if my tests are actually good?

A: Mutation testing: deliberately break the code slightly. Do the tests catch the break? If not, your test coverage is illusory. Run this on 10% of your code quarterly. If

Case Study: AI-Generated Tests in Practice

A fintech company had a codebase with 45% test coverage. They wanted to improve. They tried two approaches:

Team A: Manually wrote additional tests targeting uncovered code. Effort: 120 hours. Result: coverage improved to 72%. Bugs caught: 8 over 3 months.

Team B: Used AI to generate tests for the same uncovered code, then reviewed and refined. AI generated 250 test cases. Team reviewed, keeping 160 (64% retention, the rest were too trivial or testing implementation). Effort: 60 hours (half of Team A). Result: coverage improved to 75% (better than Team A). Bugs caught: 14 over 3 months (better than Team A).

Why Team B was better: (1) AI didn't miss cases that humans might miss (thoroughness), (2) humans focused on review and filtering, which is higher-value than writing boilerplate, (3) better test variety (AI thought of scenarios humans wouldn't).

Key lesson: AI test generation works best when paired with human review. The human's job shifts from "write tests" to "decide which generated tests are valuable." This is faster and often produces better results.

Advanced: Mutation Testing with AI

Mutation testing reveals whether tests are actually good. Tools like Stryker introduce mutations (small changes to code) and check if tests catch them. "Mutation score" = percentage of mutations caught.

You can use AI to improve mutation scores: "Here's my code with a mutation. Here's my test. The test didn't catch this mutation. Generate an additional test that would catch it." Over multiple iterations, you get test suites that actually catch bugs.

Effort: Higher than just generating tests. Payoff: Much more confidence that your tests are good.

Best practice: Run mutation testing quarterly on critical code. Use AI to improve tests based on missed mutations. Track mutation score (should be >85% for critical code).

On This Page

Watch the Lecture
The Testing Truth
When to Use AI for Tests
The Test Generation Workflow
Test Quality Metrics
Smart Test Generation Practices
Coverage Without Quality
Continuous Test Improvement
What to Do Monday Morning
FAQ

Chapter Details

Part of