How RAG Chatbots Work: Document Chunking, Vector Embeddings, and Grounded Citations

By Muhammad Abbas • 4 min read

Large Language Models possess broad general knowledge, yet they know nothing about your proprietary policies, product SKUs, updated fee structures, or internal engineering SOPs. Worse, when asked a specific company question without matching context, standard models often guess convincingly. Retrieval-Augmented Generation (RAG) fixes this architectural limitation.

1. The Two Phases of Every RAG System

First comes offline indexing (ingestion): PDFs, Word documents, Markdown help centers, and database rows are parsed cleanly, segmented into logical semantic sections ('chunks'), converted into numerical vector embeddings, and persisted inside a vector index such as PostgreSQL with pgvector alongside metadata tags.

2. Why Pure Vector Search Isn't Enough (Hybrid Retrieval)

Vector embeddings excel at conceptual matching — understanding that 'annual leave carryover rules' relates to 'vacation days rollover policy'. However, dense vectors sometimes miss exact alphanumeric strings such as product SKU codes, invoice IDs, or specific HTTP error codes.

3. Security & Role-Based Access Before Retrieval

Never retrieve private company passages and trust a system prompt to 'avoid mentioning HR secrets'. Prompt instructions can be bypassed cleanly by adversarial users. Access control must execute directly inside the SQL vector WHERE clause using verified session permission claims so unauthorized chunks never enter the context window at all.

4. Evaluating Answer Grounding Before Launch

Before deploying an assistant to staff or external users, test it against both answerable questions and unanswerable trap questions. A well-tuned RAG system knows when to reply concisely: 'The uploaded documentation does not specify a policy for that topic — please contact support.'