The 9 Places Your AI System Leaks Data (and How to Seal Each One)

Why Encryption Doesn't Help

The default response to "our AI system handles sensitive data" is "we'll encrypt it." Encryption protects data at rest and in transit. It does nothing for data in use. The moment your application decrypts customer data to build a prompt, that data is exposed to the LLM provider, embedded in vector stores, written to logs, passed through tool calls, cached in memory, and stored in error reports.

AI systems leak data in ways traditional applications don't. The attack surface is not the network or the database. It's the pipeline: every step between receiving user input and delivering a response creates a potential leak point.

We mapped every leak point across the AI systems we've built and deployed. This article is the result. For the architectural solution (semantic tokenization, trust boundaries, policy-driven restore), see our GDPR compliance guide. This article focuses specifically on where leaks happen and how to detect them.

The 9 Leak Points

User Input (contains PII)
    │
    ├──▶ 1. PROMPTS ────────────▶ LLM Provider sees raw data
    │
    ├──▶ 2. EMBEDDINGS ─────────▶ Vector DB stores PII permanently
    │
    ├──▶ 3. LOGS ───────────────▶ Log aggregator indexes PII
    │
    ├──▶ 4. TOOL CALLS ─────────▶ External APIs receive PII in parameters
    │
    ├──▶ 5. AGENT MEMORY ───────▶ Conversation history stores PII
    │
    ├──▶ 6. ERROR MESSAGES ─────▶ Error tracking captures PII in stack traces
    │
    ├──▶ 7. CACHE ──────────────▶ Response cache stores PII in cached answers
    │
    ├──▶ 8. FINE-TUNING DATA ───▶ Training data contains PII permanently
    │
    └──▶ 9. AGENT HANDOFFS ─────▶ PII crosses trust boundaries between agents

1. Prompts

The most obvious leak point and the one most teams think they've solved. Every prompt sent to an external LLM provider contains whatever data you put in it. If you include a customer's name, email, order history, or complaint text in the prompt, the LLM provider receives it.

Where it leaks:

• The LLM provider's API receives the full prompt
• The provider may log prompts for abuse detection
• The provider may use prompts for model improvement (depends on terms of service and API vs consumer tier)

How to seal it: Replace raw PII with semantic tokens before the prompt reaches the LLM. The model receives {{person:p_001}} instead of the actual name. The token carries metadata (gender, formality, language) so the model can still produce correct output.

// Before: PII in prompt
const prompt = `Write a reply to Sara Mustermann about order #12345`;

// After: tokenized prompt
const transformed = await guardai.transform(prompt);
// transformed.safe_text = "Write a reply to {{person:p_001}} about order {{order:o_001}}"
const response = await llm.generate(transformed.safe_text);
const final = await guardai.rehydrate(response, transformed.session_state);

2. Embeddings

This is the leak point most teams miss entirely. When you embed documents for RAG, the embedding vectors encode semantic information about the content. If the content contains PII, the embeddings carry a representation of that PII in the vector store.

Where it leaks:

• Vector database stores embeddings derived from PII-containing documents
• Similar-document retrieval can surface PII from unrelated queries
• Vector database backups contain PII representations

How to seal it:

Option A: Tokenize documents before embedding. The embeddings are built on tokenized text. The vector store never contains PII-derived vectors.

Option B: Embed raw documents but tokenize at query time. The vector store is inside the trusted zone. Only the retrieved chunks cross to the LLM, and those are tokenized before crossing.

// Option A: embed tokenized content (safer, slightly lower retrieval quality)
const tokenized = await guardai.transform(documentText);
const embedding = await embedder.embed(tokenized.safe_text);
await vectorStore.upsert({ id: docId, embedding, metadata: { tokenized: true } });

// Option B: embed raw, tokenize at query time (better retrieval, more complex)
const embedding = await embedder.embed(documentText); // raw embedding
await vectorStore.upsert({ id: docId, embedding });
// At query time: retrieve chunks, then tokenize before sending to LLM

For more on RAG pipeline architecture, see our RAG reliability guide and vector search guide.

3. Logs

Your logging infrastructure becomes a PII store the moment you log raw prompts or responses. Your Datadog, CloudWatch, Elasticsearch, or Grafana Loki cluster now processes personal data, each requiring GDPR documentation, retention policies, and data subject access procedures.

Where it leaks:

• Structured logs containing prompt text
• Request/response logging middleware
• Debug logs during development that make it to production
• Third-party logging SDKs that capture request bodies

How to seal it: Log token IDs, never values. Log event types, entity counts, confidence scores, and policy names. Never log the raw text.

// Good: PII-free structured log
logger.info('transform_complete', {
    session_id: 'ses_abc',
    entities_detected: 3,
    entity_types: ['person', 'email', 'phone'],
    policy: 'german-support',
    duration_ms: 12,
});

// Bad: PII in logs
logger.info('Processing request', {
    input: 'Sara Mustermann wants to cancel order 12345...',
    // This is now PII in your log aggregator
});

4. Tool Calls

In agentic systems, the AI agent calls external tools (APIs, databases, services). The tool call parameters often contain PII extracted from the conversation. A CRM lookup tool receives a customer name. A billing tool receives an account number. An email tool receives a recipient address.

Where it leaks:

• Tool call parameters sent to external services
• Tool call logging (agent frameworks often log every tool call)
• Tool response data cached in agent memory

How to seal it: Intercept tool calls at the trust boundary. The agent passes tokenized parameters. The runtime resolves tokens to real values inside the trusted zone, calls the external service with real data, then tokenizes the response before returning it to the agent.

Agent: "Look up customer {{person:p_001}}"
  │
  ▼ (trust boundary)
Runtime: resolves p_001 to "Sara Mustermann"
Runtime: calls CRM API with real name
Runtime: gets response with real data
Runtime: tokenizes response
  │
  ▼
Agent receives: "Customer {{person:p_001}}, account {{customer_id:cid_001}}"

The agent never sees or stores the raw values. The external service operates inside the trusted zone. For how we implement this in agentic commerce, see our agentic commerce guide.

5. Agent Memory

Agentic AI systems maintain conversation memory across turns. This memory stores the entire conversation history, including any PII the user shared. If memory is persisted (Redis, database), PII accumulates over time.

Where it leaks:

• In-memory conversation history during the session
• Persisted memory in databases or key-value stores
• Long-term memory used for personalization across sessions
• Memory shared between agents in multi-agent systems

How to seal it: Scope memory by tenant, session, and thread. Tokenize PII in memory the same way you tokenize prompts. Set TTLs on persisted memory. Never share memory across tenant boundaries.

// Memory scoping: tenant + session + thread
const memoryKey = `${tenantId}:${sessionId}:${threadId}`;

// Memory content is tokenized
const memory = [
    { role: 'user', content: 'I am {{person:p_001}}, my order is {{order:o_001}}' },
    { role: 'assistant', content: 'Let me check order {{order:o_001}} for you, {{person:p_001}}.' },
];

// TTL: memory expires after session ends
await memoryStore.set(memoryKey, memory, { ttl: SESSION_TTL });

For multi-tenant memory isolation patterns, see our multi-tenant design guide.

6. Error Messages

When something fails, error messages and stack traces often contain the data that caused the failure. A JSON parse error includes the raw JSON. A validation error includes the invalid field value. An API timeout includes the request payload.

Where it leaks:

• Error tracking services (Sentry, Bugsnag, Datadog APM)
• Stack traces in server responses
• Error logs with request context
• Unhandled exception reporters

How to seal it: Sanitize error payloads before sending to error tracking. Strip request bodies from error context. Mask PII in error messages. Never return raw error details to clients.

// Error sanitizer middleware
function sanitizeError(error: Error, context: any): SanitizedError {
    return {
        message: error.message,
        code: error.code,
        // Strip PII from context
        context: {
            session_id: context.session_id,
            entity_types: context.entity_types,
            // Do NOT include: context.input, context.prompt, context.customerData
        },
    };
}

7. Cache

Response caching stores the full response, including any PII in the generated answer. If a cached response contains "Sara Mustermann's order was shipped yesterday," every subsequent user who triggers the same cache key sees Sara's data.

Where it leaks:

• Application-level response caches (Redis, Memcached)
• CDN caches (if AI responses are cached at the edge)
• Semantic caches (similar questions return cached answers with PII)

How to seal it: Cache tokenized responses, not rehydrated ones. The cache stores {{person:p_001}}'s order was shipped yesterday. Each user's request rehydrates with their own session mapping.

Alternatively, include the session or tenant ID in the cache key so personalized responses are never served to the wrong user.

8. Fine-Tuning Data

If you fine-tune a model on customer data, that data is permanently embedded in the model weights. You cannot extract specific records from a fine-tuned model. You cannot comply with a GDPR deletion request for data that's been used in training.

Where it leaks:

• Training datasets containing PII
• Model weights (data is encoded, not extractable but influences outputs)
• Training data stored by the model provider during fine-tuning

How to seal it: Tokenize training data before fine-tuning. The model learns patterns from tokenized text, not from real PII. If you must use real data for fine-tuning quality, document the legal basis, implement data retention policies, and be prepared to retrain the model if a deletion request requires it.

9. Agent Handoffs

In multi-agent systems, one agent might hand off a conversation to another agent. The handoff typically includes conversation context, which contains PII from the interaction. If agents operate across different trust boundaries (different services, different providers), PII crosses those boundaries during handoff.

Where it leaks:

• Handoff messages between agents
• Shared context stores used by multiple agents
• Agent-to-agent API calls carrying conversation state
• Orchestrator services that route between agents

How to seal it: Tokenize the handoff context. The receiving agent gets tokenized conversation history, not raw PII. Each agent operates within the same trust boundary model. Cross-boundary handoffs go through the same tokenization layer as any other trust boundary crossing.

For how we design multi-agent architectures with proper isolation, that guide covers the patterns.

The Output Guard: Catching What the Model Invents

Beyond the 9 input-side leak points, there's a 10th problem: the model can hallucinate PII that wasn't in the original input. If you ask the model to write a response to {{person:p_001}}, it might invent a phone number, an address, or a company name from its training data.

This hallucinated PII is not protected by your tokenization because it was never tokenized. The solution is a second-pass detection on the model's output:

async function outputGuard(response: string, sessionTokens: Set<string>): GuardResult {
    // Run PII detection on the model's response
    const detected = await detector.detect(response);

    // Check each detected entity against known session tokens
    const unknown = detected.filter(entity => !sessionTokens.has(entity.tokenId));

    if (unknown.length > 0) {
        return {
            safe: false,
            flagged: unknown,
            action: 'remove_or_flag', // Strip unknown PII or flag for review
        };
    }

    return { safe: true };
}

The output guard catches: invented phone numbers, hallucinated addresses, real company names from training data, and any other PII the model generates that wasn't in the original input.

For more on AI failure modes including hallucination, that guide covers the broader patterns.

Detection Priority

Not all leak points are equally dangerous. Prioritize based on exposure surface and data sensitivity:

Start with P0 (prompts and logs). These are the highest volume and the easiest to fix. Then address P1 (tool calls, memory, errors). P2 and P3 are important but less urgent.

Common Pitfalls

1. Thinking encryption solves the problem. Encryption protects transit and storage. It does nothing when data is decrypted for processing, which is exactly what an LLM does.

2. Only protecting the final LLM call. A 5-step agent workflow has 5 leak surfaces. Protecting only the generation step leaves 4 unprotected.

3. Logging prompts for debugging. Every prompt you log is PII in your log aggregator. Log token IDs, entity types, and metadata instead.

4. No output guard. The model hallucinates PII from training data. Without a second-pass detection on the output, hallucinated PII reaches the end user.

5. Caching rehydrated responses. If the cache stores "Sara Mustermann's order shipped," any user who hits the same cache key sees Sara's data.

6. Fine-tuning on raw customer data. Once PII is in model weights, it cannot be extracted or deleted. Tokenize training data.

7. Ignoring error tracking. Sentry and Bugsnag capture request context by default. That context contains PII if your requests contain PII.

8. Shared agent memory across tenants. Agent A in one tenant must not have access to agent B's conversation in another tenant.

Key Takeaways

• AI systems have 9 distinct leak points. Prompts, embeddings, logs, tool calls, agent memory, error messages, cache, fine-tuning data, and agent handoffs. Each needs its own protection strategy.

• Encryption doesn't help at processing time. Data must be decrypted before the LLM can process it. The protection must happen at the application layer, not the transport layer.

• Semantic tokenization is the architectural solution. Replace raw values with tokens that carry metadata. The LLM processes tokens. Real values stay inside the trusted zone.

• The output guard catches hallucinated PII. The model invents data from its training set. A second-pass detection on the output catches entities that weren't in the session's token mapping.

• Start with prompts and logs (P0). These are the highest volume and easiest to fix. Then address tool calls, memory, and errors (P1).

• Every leak point needs monitoring. Track how many PII entities are detected, how many are tokenized, how many slip through. Alert on anomalies.

We apply these patterns across all our AI systems through OGuardAI, our open-source semantic data protection runtime. If you're building AI systems that handle sensitive data, talk to our team or request a quote. See our AI services and trust page for more context on how we approach data protection.