Most teams treat the model’s final answer as the thing that must not leak. That framing is the vulnerability. LLM02:2026 defines sensitive information disclosure as exposure of confidential, regulated, privileged, or proprietary data through any channel the data subject, controller, or system owner did not authorise. The channel list is long: tool-call arguments, reasoning traces, retrieved chunks, multimodal output, logs, telemetry, embeddings, and observable inference properties such as timing, token length, log-probabilities, confidence, and cache-hit behaviour.
Each of those is an output. Each is subject to the same classification and redaction rules as the answer text. A pipeline that sanitises the completion while streaming raw retrieved chunks into an APM project has not implemented data protection; it has implemented output formatting.
The second framing the entry insists on: severity turns on what the recipient can learn, not on whether the leak looked like natural language. A membership-inference result that names a training set patient discloses nothing quotable and is still a reportable determination.
The four phases
Disclosure arises across four phases of the LLM lifecycle, and they need different controls. Mixing them up is why so many programmes buy an output filter and consider the problem addressed.
| Phase | What leaks | Why filtering the answer does not help |
|---|---|---|
| Training-time | Corpus content memorised into weights, a fine-tune, or a LoRA adapter, later reproduced verbatim or in recoverable form | The data is in the model; it survives deletion of the source |
| Inference-time | Live context: system prompt, RAG chunks, files, tool outputs, memory, or another session’s data | Summarisation, translation, and extraction surface more than was asked, including visually redacted spans |
| Pipeline-time | Fine-tuning, distillation, synthetic-data generation, gradients, SDKs, and observability move data into derived artefacts | The leak happens off the request path entirely |
| Observation-time | Facts inferred from externally measurable properties - token length under TLS, latency, log-probabilities, confidence, cache-hit signals | No content is ever returned to filter |
Memorisation scales log-linearly with capacity, duplication, and context length. Narrow adapters are a distinct problem from the base model: they memorise rare examples with high fidelity, which makes a LoRA published alongside a base model a targeted extraction surface in its own right.
Protected information here includes PII, PHI, financial data, credentials, API keys, trade secrets, model weights, privileged communications, classified or export-controlled material, and biometric and genomic identifiers.
Two structural failures
The source names two causes behind most incidents, and neither is a model problem.
Oversharing upstream. Unscoped drives, legacy permissions, and knowledge bases feed RAG with sensitive data that the model then retrieves exactly as designed. There is no misbehaviour to detect. The fix is the data surface, not the model. The OWASP data-security guidance owns this as DSGAI01.
Persistence. Once data has influenced weights, embeddings, or adapters, it stays extractable after the source record is deleted. That strains GDPR Article 17 and CCPA §1798.105 erasure obligations in a way that no retention policy on the source store resolves. Open-weights deployments make it worse: extraction, membership inference, and inversion all run offline at unbounded rates, so rate limiting is not available as a compensating control.
Applicable regimes named by the entry are the EU AI Act (Regulation (EU) 2024/1689, high-risk obligations from August 2026), GDPR, HIPAA, CCPA/CPRA, ISO/IEC 42001, and NIST AI 600-1.
Scope note: this entry covers the LLM as an application component. Where it acts as an autonomous actor (cross-session memory, tool choice, multi-step exfiltration), the amplified risk is owned by the Agentic Security Initiative entries, with deeper controls in DSGAI.
How it arises
Training-data memorisation and extraction. The November 2023 “poem” divergence attack drove gpt-3.5-turbo to emit more than 10,000 unique memorised examples for roughly USD 200 (Nasr et al., 2023). Vendor patches for that class have been bypassed repeatedly. Fine-tuned models and their LoRA adapters are more extractable than base models of the same scale. Litigation exhibits in NYT v. OpenAI and Getty v. Stability AI include examples of verbatim reproduction, with the Getty case centred on watermark reproduction.
Inference-time context and output disclosure. The March 2023 ChatGPT Redis bug exposed payment
PII for 1.2% of Plus subscribers. More than 4,500 shared conversations were indexed by Google in
2025 through missing noindex directives - a leak with no attacker at all. Clinical-embedding
vector stores sit in HIPAA audit-control scope that most teams have not operationalised, and
retrieval-layer authorisation is widely under-implemented. Reasoning traces and tool arguments are
outputs, not debugging leftovers; the trace channel additionally enables reasoning-trace-coercion
model extraction. Regex and blocklist filters fall to cross-lingual, base64, and hex encodings.
Aggregation counts too: budget plus hiring plus diligence data, each individually permitted,
synthesising a pending M&A target is a disclosure when policy prohibits the conclusion.
Embedding and representation disclosure. Modern inversion reconstructs plaintext from leaked or exported vectors, so an “embeddings-only” backup is a source-document breach. Cosine similarity does not respect ACLs. Authorise before retrieval, because post-generation filtering cannot undo a chunk already supplied to the model. The mechanisms belong to LLM09:2026 and DSGAI13; LLM02 owns the regulatory consequence.
Multimodal disclosure. Vision models OCR credentials and PII out of screenshots, notifications, and PDF metadata. Generators reproduce watermarks and identifiable faces. The Getty / Stable Diffusion watermark case is canonical. Cross-modal transformation, text rendered as an image or an image OCR’d back to text, bypasses single-modality DLP.
Inference-time side channels. This is the category most programmes have no control for at all:
| Result | Finding |
|---|---|
| SPV-MIA (Fu et al., 2024) | Membership-inference AUC of 0.9 against fine-tuned targets - enough for a regulatory breach determination about a named individual |
| Whisper Leak (McDonald & Bar Or, 2025) | Conversation topics classified at greater than 98% AUPRC across 28 production models, from encrypted traffic |
| Weiss et al. (2024) | 29% of response content reconstructed, topic inferred for 55%, via token length |
| Wu et al. (2025) | Prompt leakage through KV-cache sharing in multi-tenant serving |
| Dong et al. (2025) | A 4,112-token medical prompt inverted from a middle layer at F1 0.8688 (token matching) |
| Carlini et al. (2024) | A production model’s projection layer recovered through a logit-bias channel |
Training-pipeline disclosure. Gradient inversion by a malicious server (Boenisch et al., 2021), distillation, and synthetic-data carryover all move examples into derived models. Iterative query attacks against a DP-protected model refine LLM-generated queries and home in on confidence spikes to re-identify individuals. Fixed-epsilon differential privacy is therefore necessary but not sufficient without rate limiting, query-pattern detection, and per-user budgets.
Platform and ecosystem disclosure. Observability platforms (Langfuse, LangSmith, Datadog LLM Observability) log full prompts, completions, chunks, and traces by default. Two representative incidents: DeepSeek’s January 2025 ClickHouse exposure of more than one million rows of logs and API keys (Wiz, 2025), and Check Point’s 2026 disclosure of ChatGPT exfiltration through a hidden outbound channel in the code-execution runtime. One crafted prompt turned the runtime into a silent DNS channel while the visible answer stayed benign (Check Point Research, 2026).
Mitigations
The source arranges controls in the DSGAI tiered structure, which gives a graduated implementation path rather than a flat checklist. Tier 1 is not optional; Tiers 2 and 3 are scoped by threat model, because their cost is real.
Tier 1: foundational (every deployment)
- Govern corpora. Provenance, classification, and deduplication across near-duplicates, transliterations, and format variants. Scrub PII at ingest. Deduplication reduces but does not eliminate memorisation.
- Minimise context. Send only task-required fields to external providers. Disable auto-context
(
customer_360, full-record append) unless justified per template. - Authorise before retrieval. Enforce document- and chunk-level authorisation inside the index query, not at the application layer after retrieval. Isolate per-tenant indexes for high-sensitivity workloads.
- System-prompt hygiene. Never store secrets, credentials, or regulated data in system prompts.
- Sanitise with classifiers, not regex alone. Pattern matching plus NER plus trained classifiers, because regex fails on encoded and cross-lingual output.
- Budget queries per user and per session on sensitive endpoints, to disrupt enumeration and membership probing.
- Operational hygiene. Restrict and scrub logs and traces before APM ingestion, encrypt in transit and at rest, and enforce no-train/no-retain technically rather than as policy text.
Tier 2: hardening (regulated / high-sensitivity)
- DP-SGD calibrated to sensitivity and cardinality, with overfitting monitored as a memorisation proxy. Pair it with detection, because fixed budgets degrade under adaptive querying.
- Vector-store protection. Encryption, ACLs held separately from document ACLs, restricted export APIs, minimum-scope k-NN, and embedding-space probing detection.
- Gate log-probabilities, confidence, and explanations on production endpoints. These are the raw material for extraction and membership inference.
- Classify and redact reasoning traces as first-class output. Never log raw traces to unrestricted observability.
- Side-channel defences. Random padding and token batching for streaming, segregation of high-sensitivity tenants onto dedicated prefix caches, and partitioned KV caches under co-tenancy. The cost is latency and throughput; that is the trade being made.
- Format-preserving encryption for structured identifiers, with strict internal-versus-external routing separation and field allowlists on the external path.
- AI-aware audit logging into SIEM, continuous DLP and AI-SPM, and a documented domain inventory with an enforced join policy, so individually-permitted sources cannot combine into prohibited conclusions.
Tier 3: advanced (regulated, classified, high-target)
- Confidential computing (Intel TDX, AMD SEV-SNP, AWS Nitro Enclaves), or emerging privacy-preserving inference such as the AloePri covariant-obfuscation framework (Lin et al., 2026), where the threat model justifies the utility and latency cost. The source is explicit that this is a threat-model decision, not a default.
- Verifiable erasure across raw data, embeddings, checkpoints, and adapters, validated by post-unlearning extraction and membership-inference probes. Erasure you cannot measure is a claim, not a control.
- Disclosure red-teaming as a release gate: extraction, membership inference, embedding inversion, internal-state inversion, side channels, and LoRA extractability, measured quantitatively and aligned to MITRE ATLAS.
- Audit synthetic data against extractors. Resist distillation with probing detection, rate limits, and watermarking. Budget aggregate analytics.
- Exercise a disclosure incident-response playbook. Scope by data class and affected subject or session, then assess and meet applicable breach and serious-incident notification obligations (for example GDPR, HIPAA, EU AI Act Article 73). Follow with unlearning, retraining, or withdrawal; vector and cache cleanup; vendor notice; and a persistent-memory audit.
Attack scenarios
- Divergence prompts make a production model emit memorised PII, URLs, and live credentials at scale, triggering GDPR Article 33 notification.
- A shared-inference-state defect leaks one user’s medical-letter prompt into another user’s reasoning trace. HIPAA 60-day notification applies.
- Extended-thinking traces logged verbatim to a shared APM project expose retrieved PII to hundreds of engineers while the answer itself stays sanitised.
- Prompt injection makes a support bot print its system prompt and an embedded vendor API key.
- A shared legal RAG index crosses firm boundaries, synthesising one client’s privileged strategy into another client’s answer: an attorney-client waiver event.
- A leaked “embeddings-only” vector backup is reclassified as a source-document breach after inversion, restarting the 72-hour clock.
- Whisper Leak topic inference on encrypted streaming identifies users querying medical, legal, or political topics, without decryption.
- Membership inference against a clinical fine-tune identifies training-set patients at high AUC without extracting any record. This is a HIPAA-reportable determination.
- A model summarises PII hidden beneath a black-rectangle PDF redaction layer rendered over unmodified text.
- An injected “diagnostic check” makes a code runtime encode spreadsheet content into DNS queries while the visible statistical summary stays benign.
Scenarios 4 and 10 are worth noting together: both start as prompt injection and become disclosure because a channel existed that nobody classified as an output.
Related entries
- Prompt Injection (LLM01:2026) - the input-side compromise that turns into disclosure whenever context, traces, or tool arguments are reachable.
- Excessive Agency (LLM03:2026) - tool permissions and identity scope decide how much data a compromised model can reach in the first place.
- Supply Chain (LLM04:2026) - unclear supplier T&Cs and retention policies are a disclosure vector before any request is made.
- LLM09:2026 owns the mechanisms of embedding and vector weaknesses; this entry owns their regulatory consequence.
- More entries: AI Security.