01 - Introduction to RAG Security & Threat Model
Retrieval-Augmented Generation (RAG) bridges the gap between static model weights and dynamic enterprise data. By fetching relevant external knowledge chunks and appending them to the prompt context, RAG enables LLMs to answer questions on proprietary knowledge bases without requiring continuous, expensive model fine-tuning.
However, from an application security perspective, RAG transforms static prompts into dynamic, untrusted data processing pipelines. When untrusted documents are ingested, embedded, and injected directly into the LLM context window, attackers gain a direct vector to influence model execution, exfiltrate confidential data, or manipulate downstream application logic.
[!IMPORTANT] The Core Security Failure in RAG: Natural language LLMs do not enforce a hardware- or software-level boundary between system instructions (control plane) and retrieved context (data plane). In legacy computer architecture, this is equivalent to executing code straight out of user input buffers without stack separation or memory protection.
1. RAG Architecture & Technical Mechanicsβ
A enterprise RAG system consists of six distinct operational phases spanning two asynchronous workflows: The Ingestion Pipeline and The Retrieval & Generation Pipeline.
Deep Dive into Pipeline Phasesβ
Phase 1: Data Ingestion & Chunkingβ
Documents are pulled from internal repositories (SharePoint, Notion, S3, SQL DBs) or public web scrapers. They are parsed into discrete text chunks using strategies such as:
- Fixed-Size Chunking: Slicing text by strict character/token counts (e.g., 512 tokens with 50-token overlap).
- Recursive Character Chunking: Splitting hierarchically by paragraphs, sentences, and words.
- Semantic Chunking: Grouping text based on semantic embedding distance changes.
Phase 2: Embedding Generation & High-Dimensional Vector Spaceβ
Each chunk is passed to an embedding model (e.g., OpenAI text-embedding-3-small, HuggingFace all-MiniLM-L6-v2) which maps natural language into a dense vector space of dimension d (e.g., 1,536 dimensions).
Mathematically, the similarity between query vector Q and document chunk vector D is evaluated using metrics such as Cosine Similarity:
Cosine Similarity(Q, D) = (Q . D) / (||Q|| * ||D||)
Where Q . D is the dot product of the vectors, and ||Q|| represents the Euclidean norm of vector Q.
Phase 3: Vector Storage & Indexingβ
Vectors are stored in a specialized Vector Database (Pinecone, Qdrant, Milvus, Chroma, pgvector) alongside key-value metadata (e.g., tenant_id, document_id, author, creation_date). To enable sub-millisecond retrieval across millions of vectors, databases construct approximate nearest neighbor (ANN) indexes such as:
- HNSW (Hierarchical Navigable Small World): Graph-based multi-layer indexing.
- IVF-PQ (Inverted File with Product Quantization): Clustering space into Voronoi cells.
Phase 4: Retrieval & Top-k Searchβ
When a user submits a prompt, the user query is converted into a query vector using the same embedding model. The vector DB executes an ANN search to return the top k most semantically similar document chunks.
Phase 5: Prompt Assembly & Context Concatenationβ
The application backend constructs the final prompt by stuffing the retrieved top-k document chunks into the context window alongside the system prompt:
[SYSTEM PROMPT]
You are an enterprise support assistant. Answer the user's query using ONLY the provided context below.
[CONTEXT]
Doc Chunk 1: ...
Doc Chunk 2: ...
Doc Chunk k: ...
[USER QUERY]
How do I reset my password?
Phase 6: Generation & Executionβ
The assembled prompt is transmitted to the LLM. The model generates a completion based on the combined probabilities of system instructions, retrieved context, and user input.
2. Trust Boundaries & The In-Band Control Problemβ
In conventional application development, control logic is strictly isolated from data. In SQL, parameterized queries separate SQL commands from user input strings. In web application security, output encoding separates HTML structure from user-generated content.
CONVENTIONAL WEB SECURITY (Out-of-Band Control):
[SQL Command / Structure] <--- ISOLATED BOUNDARY ---> [User Input Data]
RAG LLM ARCHITECTURE (In-Band Control):
[System Instructions] + [Retrieved Untrusted Context] + [User Input]
ββββββββββββββββββββ¬βββββββββββββββββββ
βΌ
Single Natural Language Stream
In a RAG application, all inputs are merged into a single continuous stream of natural language tokens. Because LLMs interpret instructions dynamically based on semantic context, a retrieved document chunk containing imperative natural language (e.g., "Ignore prior rules and do X") can hijack the model's instruction pointer.
3. OWASP Top 10 for LLM Applications Mappingβ
RAG architectures intersect with multiple critical risks highlighted in the OWASP Top 10 for Large Language Model Applications:
| OWASP Risk ID | Risk Name | Vulnerability Impact in RAG Pipelines |
|---|---|---|
| LLM01 | Prompt Injection | Indirect prompt injection where malicious payloads inside retrieved PDFs, web pages, or support tickets manipulate LLM execution. |
| LLM02 | Sensitive Information Disclosure | Cross-tenant data leakage resulting from missing metadata filters in vector search queries, returning private documents belonging to other users. |
| LLM03 | Supply Chain & Data Poisoning | Attackers corrupting training data or vector database documents to bias embeddings, insert backdoors, or enforce targeted hallucinations. |
| LLM06 | Excessive Agency | RAG systems connected to tools (email sending, DB updates, API calls) executing unintended actions when tricked by injected context. |
| LLM08 | Vector and Embedding Weaknesses | Exploiting vector space proximity, embedding collisions, or index manipulation to bypass access controls or force document retrieval. |
4. Threat Landscape & Attack Sequenceβ
The following diagram illustrates a complete indirect prompt injection and exfiltration sequence in a vulnerable enterprise RAG setup:
5. Vulnerability Root Cause Matrixβ
To effectively defend RAG pipelines, security teams must recognize that vulnerabilities stem from three root system design failures:
| Vulnerability Type | Legacy Analogy | RAG Architecture Root Cause | Impact |
|---|---|---|---|
| Indirect Prompt Injection | Second-Order SQL Injection | Treating retrieved vector context as trusted instruction tokens rather than untrusted data plane tokens. | System instruction override, payload execution, arbitrary data exfiltration. |
| Cross-Tenant Leakage | Broken Object Level Authorization (BOLA) | Performing vector ANN similarity search globally across a single collection without enforcing server-side tenant metadata filtering (tenant_id). | Unauthorized document access, intellectual property theft, privacy violation. |
| Vector Space Poisoning | Cache Poisoning / Tampering | Allowing untrusted or unvalidated external documents into the embedding index without structural parsing, sanitization, or provenance verification. | Model hallucination, targeted retrieval manipulation, persistent system backdoors. |
[!TIP] AppSec Key Takeaway: Securing a RAG pipeline requires shifting from "prompt-level defense" to pipeline-level defense-in-depth. Every stageβfrom ingestion, vector database querying, context sanitization, to output renderingβmust enforce strict validation, authorization, and quarantine controls.