The Data Question: Where Does Your Data Go?

You're a CTO at a healthcare company. You use ChatGPT to help draft clinical notes summaries. You copy a patient's medical record (name, diagnosis, treatment) into ChatGPT and ask: "Summarize this in simpler language."

Question: Where is that data now? OpenAI's servers. Is it being used to train models? Probably not (enterprise accounts claim they don't use data for training). But it's accessible to OpenAI staff, potentially stored long-term, potentially breached.

For a healthcare company, patient data is protected by HIPAA. Sending it to external services without consent is illegal, even if the external service is secure.

This is the core privacy problem with AI APIs: your data leaves your control the moment you send it to an external service.

The Data Control Principle: Once data leaves your infrastructure, you've lost control over it. It can be retained, breached, subpoenaed, or repurposed. For sensitive data, this risk isn't acceptable. Either don't use AI APIs with sensitive data, or ensure the vendor has ironclad contractual guarantees (with penalties for breach).

Vendor Data Policies: Read the Fine Print

Different vendors have different policies. These are constantly evolving, so check current terms, but as of 2026:

OpenAI: Enterprise accounts don't use data for training. Free/paid consumers: unclear (terms say they might use data to improve the service). The distinction matters enormously.

Anthropic (Claude): Claude API conversations are not used for training. This is explicitly stated and is a competitive differentiator.

Google (Gemini): Varies by product. Gemini API has different policies than consumer Gemini. Business users get different terms than consumers.

Microsoft (Copilot): Enterprise Copilot keeps data on customer infrastructure, doesn't train on it. Consumer Copilot has different terms.

Open-source models (Llama, Mistral): You host and control them. No external data leakage, but you're responsible for infrastructure.

The pattern: enterprise/business accounts usually have better data policies. Consumer/free tiers usually allow training on your data.

Implication: If you're using AI for work, use business accounts, not personal accounts. The cost difference (business vs. personal) is usually small. The privacy difference is massive.

The Code Leakage Problem: Your IP Walking Out the Door

Engineers love GitHub Copilot and ChatGPT for generating code. But using them with proprietary code is a serious risk.

The IP Risk: Every time an engineer pastes code into ChatGPT, there's a risk it becomes training data (depending on account type). Your proprietary algorithms, security patterns, and domain-specific logic could end up in public models. For companies where the code IS the competitive advantage, this risk is unacceptable.

Scenario: Your senior engineer uses ChatGPT to generate boilerplate. They paste your company's codebase (or significant portions of it) into ChatGPT. They copy the generated output back.

What happened: Your proprietary code is now in OpenAI's training data (depending on which ChatGPT they used). Your code could end up in a public model. Competitors could reverse-engineer your approach. Attackers could discover security vulnerabilities in your code.

Real example: A company's proprietary API authentication scheme was sent to ChatGPT. Later, researchers discovered that same scheme in Copilot's outputs. The company's secret approach was no longer secret.

This is particularly serious for:

• Regulated industries: Finance, healthcare. Proprietary code might contain sensitive logic or patterns that shouldn't be visible externally.
  - Companies with secret sauce: Where competitive advantage is in the code itself (not the product).
  - Security-critical code: Authentication, encryption, access control. This code is high-value to attackers.

Mitigation:

• Use enterprise accounts: OpenAI, Anthropic, and others offer accounts where your data isn't used for training. Use these for work.
  - Redact before sending: Remove API keys, credentials, internal domain names, business logic before pasting into AI. Use AI only for standard patterns (CRUD operations, boilerplate).
  - Use local models: For sensitive code, self-host. Llama 2, Mistral, and others can run on your infrastructure. No external data leakage.
  - Policy and training: Make clear to engineers: "Don't paste proprietary code into free ChatGPT." Have audits to catch violations.
  - Code review: When an engineer uses Copilot to generate code, review it. Not just for correctness, but for whether it reveals something proprietary.

Customer Data and GDPR: Legal Obligations

You're a SaaS company with European customers. GDPR requires that customer data isn't transferred to countries without adequate data protection without explicit consent.

You want to use an AI service to analyze customer feedback. The process: extract feedback data, send to AI service for analysis, get insights back.

Problem: Customer data leaving your infrastructure. Sent to an external service (usually in the US). This might violate GDPR even if the service itself is compliant, because the customer didn't consent to their data being processed by a third party.

Legal perspective: You're the data controller. Your customers are data subjects. You have obligations to customers about what you do with their data. Sending it to external services requires either explicit consent or legitimate business justification with proper agreements.

Practical reality: Many companies are still figuring this out. Some DPAs (data protection authorities) are starting to enforce it. Safer to assume: if data is customer data or sensitive user data, don't send it to external APIs without legal review.

Mitigation:

• Anonymize: Before sending customer data to AI APIs, anonymize/pseudonymize it. Remove identifying information. The AI can still analyze patterns, but specific customers aren't identifiable.
  - Get consent: If you want to process customer data with AI, ask customers explicitly: "We use AI to analyze your feedback. Do you consent?" (Ideally, you've already hidden this in your privacy policy, but explicit is better.)
  - Data Processing Agreements: If using a vendor, have a DPA in place that specifies how data is handled, who can access it, how it's protected.
  - On-prem processing: For sensitive data, process it on your own infrastructure using self-hosted models. No external data leakage. Higher cost, but legally safer.
  - Legal review: Especially if you're in regulated industries or handle sensitive data, have your legal team review data usage practices before deploying.

Model Training and Data: Long-Term Data Retention

You want to fine-tune a model on proprietary data to improve its performance for your use case. You send training data to a vendor's fine-tuning service.

What happens: Your training data is used to fine-tune the model. The model is deployed. But the training data: where is it?

Different vendors handle this differently:

• OpenAI fine-tuning: Your data trains your model (not other customers' models). But OpenAI might keep the data for record-keeping.
  - Most vendors: Training data is retained for some period (days, weeks, months, indefinitely depending on terms).
  - Best-case scenario: Vendor deletes your data after training. Worst-case: data is kept indefinitely.

Risk: Your proprietary training data is stored by a third party, potentially vulnerable to breach. If the vendor's security is compromised, your data leaks.

For extreme sensitivity (nuclear codes, medical research, financial algorithms), you need on-prem solutions where data never leaves your control.

Mitigation:

• Understand vendor's data retention policies. Don't assume they delete it.
  - Ask for deletion guarantees in writing.
  - For highly sensitive data, self-host models.
  - Consider: is the performance gain from fine-tuning worth the security risk?

Licensing and Legal Risk: Copyright and IP

Models are trained on data. Some of that data is copyrighted. What are the legal implications?

Scenario: You train a code generation model on open-source code (lots of it is available, publicly licensed). The model learns from that code. Later, it generates new code that's similar to some training examples.

Question: Does the generated code inherit the license of the training data? If training data is GPL-licensed (copyleft), does generated code have to be GPL-licensed too?

Legal status: Unclear. Few cases have been tested. This is active litigation (GitHub Copilot is being sued over exactly this).

Conservative approach: Assume you might have liability. Be careful about training data sources. If you use GPL code in training and generate similar code, there might be licensing implications.

Mitigation:

• Understand what licenses are in your training data.
  - Avoid GPL-licensed code if you're generating code commercially.
  - Document training data sources.
  - Get legal advice for your specific situation.
  - Consider that even if legally uncertain, customers might object to code trained on copyleft licenses.

Practical Data Governance: Framework for Decision-Making

These are complex issues. How do you make decisions in practice?

Step 1: Classify Data Create tiers:

• Public: Information that's OK to be public. Safe to send to any API.
  - Confidential: Company-internal, not public, but not high-security. Can send to trusted vendors with contracts.
  - Highly Sensitive: Customer PII, payment data, health data, trade secrets. Only on-prem processing.

Step 2: Vendor Evaluation For each AI vendor, understand:

• What do they do with data? (training, retention, sharing)
  - What are their security controls?
  - Do they have DPAs and compliance certifications?
  - What's their track record? (have they had breaches?)

Step 3: Use-Case Matching For each use case:

• What data classification does it involve?
  - Which vendors are appropriate?
  - What additional controls (anonymization, encryption) are needed?

Step 4: Audit and Monitoring Continuously:

• Track what data goes where
  - Audit compliance with policies
  - Update policies as vendors change terms

What Comes Next

Privacy is one category of risk. The next lesson covers failure modes: ways AI systems fail that you need to plan for.

What to Do Monday Morning

• Audit current AI usage: what data is being sent to external services? Classify by sensitivity.
  - Check vendor policies for each service you use. Understand their data handling practices.
  - Create a data classification framework: public, confidential, highly sensitive.
  - Map use cases to classifications. What can be processed externally? What requires on-prem?
  - Review your privacy policy. Does it cover how customer data is processed by AI? Update if needed.

Key Insight

Data privacy with AI APIs is a serious responsibility. Proprietary code, customer data, and sensitive information can leak if sent to external services carelessly. Understand vendor policies, classify your data, use business accounts, and process sensitive data on-prem. The legal and business risks are real.

Frequently Asked Questions

Is it illegal to send customer data to ChatGPT?

Depends on jurisdiction and what "customer data" means. In GDPR regions, sending customer PII without consent is risky. In CCPA regions (California), similar. If it's company-internal data about your product, less risky. Bottom line: check with legal. Don't assume it's fine.

Do enterprise accounts actually promise not to use data for training?

Yes, major vendors explicitly promise this for enterprise accounts. But read the terms. They might retain data for other purposes (improving service, security, compliance). "Not used for training" doesn't mean "deleted immediately."

Is self-hosting a model expensive?

Depends on scale. A small model (7B parameters) might cost $100-500/month to host. A large model (70B) might cost $5k+/month. If you have significant usage, self-hosting can be cost-effective. If usage is low, external APIs are cheaper.

What if my vendor changes their data policy?

They can. All vendors reserve the right to change terms. If they do in a way that's unfavorable, your options: migrate to a different vendor, self-host, negotiate a new contract. Monitor for policy changes.

How sensitive is "proprietary code"?

Depends on your competitive model. If competitors can reverse-engineer your approach by seeing your code, it's sensitive. If your code is standard implementations of known patterns, it's less sensitive. Be honest about whether the code reveals your secret sauce.

Case Study: Financial Services Company Data Breach

A fintech company learned the hard way about data leakage. Here's what happened:

Timeline: Early 2024, they adopted ChatGPT across engineering teams. Engineers used free ChatGPT to write code, debug issues, and generate SQL queries. No governance. No contracts. No data classification.

The incident: An engineer working on transaction processing wrote a query to debug a suspicious transaction. They pasted the query into ChatGPT along with a sample transaction record. The sample contained: transaction ID, customer name, account number, payment amount, and the customer's email. All from their production database.

Immediate impact: That data was now in OpenAI's systems (unclear if used for training, but definitely logged). When they realized the mistake, they notified their legal team. Legal found multiple similar incidents across the engineering team.

Cascading problems: (1) Potential GDPR violation (customer data sent to US without consent). (2) Potential SOX violation (financial data handling). (3) Potential notification requirements (did they need to notify customers of the breach?). (4) Reputational risk if word got out.

Cost of incident: Legal review and risk assessment: $150K. Mandatory GDPR notifications to customers: 40 customers notified, 3 filed complaints with the DPA. Insurance deductible and investigation: $500K. Lost contracts due to reputational damage: 2 enterprise deals, $2.1M annual revenue. Total cost: ~$2.75M.

The fix: Post-incident, they implemented: (1) Enterprise ChatGPT accounts with data non-retention agreements. (2) Mandatory training for all engineers on data classification. (3) Code review policy: all code using external AI tools must be audited for sensitive data. (4) Internal tool deployment: self-hosted Llama 2 for safe AI use. (5) Data governance framework: what can leave the company? (Only anonymized, non-sensitive data.)

New costs: Tools and processes: $300K/year. Training and enforcement: $150K/year. Total: $450K/year ongoing.

ROI calculation: If similar incidents are avoided (probability: high given the controls), the investment pays for itself ~6x over in risk reduction alone.

Key lesson: Data governance isn't optional. It's especially critical in regulated industries. Invest in it upfront. The cost of mistakes is exponential.

Practical Safeguards: What to Implement Today

Policy Level: Create a written AI data usage policy. State clearly: "No customer data, PII, or proprietary algorithms in public AI tools." Require use of business accounts for work-related tasks. Make it clear and enforceable.

Technical Level: (1) Code review: require reviewers to flag if generated code involved sensitive data. (2) Audit logging: log what data goes where and when. (3) Secrets scanning: prevent API keys and credentials from entering prompts. (4) Egress filtering: if possible, prevent sensitive data from reaching external APIs at the network level.

Vendor Level: Get Data Processing Agreements (DPAs) in writing. Require vendors to: (a) explicitly state data isn't used for training, (b) specify retention periods, (c) commit to deletion on request, (d) maintain reasonable security standards. For regulated industries, require SOC 2 Type II or ISO 27001 certification.

Testing Level: Run penetration tests. Have someone try to exfiltrate data via AI tools. See what your actual controls catch. Fix gaps.

On This Page

The Data Question
Vendor Data Policies
The Code Leakage Problem
Customer Data and GDPR
Model Training and Data
Licensing and Legal Risk
Practical Data Governance
What Comes Next
Monday Morning Action

Chapter Details