RAG chatbot, embeddings, vector search and deployment in 2026

TL;DR

RAG (Retrieval-Augmented Generation) is a chatbot that searches relevant fragments in a knowledge base before generating an LLM answer. Key pieces: chunking, embeddings, vector store (e.g. pgvector in Branchly) and prompts with cited sources. Below: pipeline from PDF/FAQ to Next.js API, a technical angle complementing articles about knowledge base content, not about writing FAQ copy.

Who this is for

• Product teams deploying chatbots on corporate sites or in SaaS
• Developers looking beyond “dump the whole terms into the prompt”
• Companies with hundreds of documentation pages where fine-tuning makes no economic sense
• CTOs weighing inference cost vs answer quality

Keyword (SEO)

rag chatbot implementation, embeddings vector search, pgvector nextjs, ai chatbot knowledge base technical 2026

RAG vs prompt stuffing, why embeddings

An embedding is a vector (e.g. 1536 dimensions) representing semantics of a paragraph. The user question becomes a vector too, you find nearest cosine neighbors, not keyword matches.

Deployment pipeline, 6 steps

[Documents] → chunking → embedding API → [vector store]
                                              ↓
[User question] → embedding → top-k retrieval → prompt + LLM → answer + sources

1. Ingest, markdown from repo, offer PDFs, FAQ export from Branchly
2. Chunking, 400–800 tokens, overlap 50–100, heading in metadata
3. Embed, text-embedding-3-small (cheaper) or -large (more accurate)
4. Store, document_chunks table with embedding vector(1536) column
5. Query, embed question → ORDER BY embedding <=> query_vec LIMIT 5
6. Generate, GPT-4o-mini with instruction: “Answer only from context below”

Chunking, practical rules

Bad splits = bad answers, even with a good model.

• Split on Markdown headings (##), not mid-sentence
• Metadata: source_url, locale, updated_at, section_title
• Pricing tables, separate chunk with context “Pricing 2026”
• PL/EN duplicates, separate embeddings per language, locale filter in query

Example record in Branchly (branchly.cloud) with pgvector:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  content TEXT NOT NULL,
  embedding vector(1536),
  source_url TEXT,
  locale TEXT DEFAULT 'en',
  updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ON document_chunks
  USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

For <10k chunks ivfflat is enough; above that consider HNSW or a dedicated engine (Qdrant, Pinecone), still behind the same API.

Ingest code in Next.js

// scripts/ingest-knowledge.ts
import OpenAI from 'openai';
import { db } from '@/lib/db';

const openai = new OpenAI();

async function embedAndStore(chunks: { text: string; meta: ChunkMeta }[]) {
  for (const batch of chunk(chunks, 50)) {
    const res = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: batch.map((c) => c.text),
    });

    for (let i = 0; i < batch.length; i++) {
      await db.$executeRaw`
        INSERT INTO document_chunks (content, embedding, source_url, locale)
        VALUES (
          ${batch[i].text},
          ${JSON.stringify(res.data[i].embedding)}::vector,
          ${batch[i].meta.sourceUrl},
          ${batch[i].meta.locale}
        )
      `;
    }
  }
}

Run after content deploy or from cron on DevStudioIT Cloud, not on every user question.

RAG chat endpoint

// app/api/chat/route.ts
import OpenAI from 'openai';
import { streamText } from 'ai';

export async function POST(req: Request) {
  const { message, locale } = await req.json();

  const queryEmbedding = await embed(message);
  const chunks = await db.$queryRaw<Chunk[]>`
    SELECT content, source_url
    FROM document_chunks
    WHERE locale = ${locale}
    ORDER BY embedding <=> ${queryEmbedding}::vector
    LIMIT 5
  `;

  const context = chunks.map((c, i) => `[${i + 1}] ${c.content}`).join('\n\n');

  return streamText({
    model: openai('gpt-4o-mini'),
    system: `Answer in English. Use only the context below. If missing data, say you don't know.
    
Context:
${context}`,
    messages: [{ role: 'user', content: message }],
  });
}

Front-end: widget on the corporate site with links to source_url under the answer, builds trust and reduces hallucinations.

Retrieval quality, what improves relevance

• Hybrid search: BM25 (Postgres full-text) + vector, merge results, better for SKUs and product codes
• Re-ranking: after top-20 vector hits, cross-encoder or LLM picks top-5 (costlier, more precise)
• Query rewrite: LLM rewrites user question into a “search query” before embed
• Threshold: if max similarity < 0.75, reply “I don't have that information” instead of guessing

Log question, chunk_ids, similarity scores in Branchly, weekly review of 20 low-score conversations.

Cost and latency (rough 2026)

• Embedding 1M tokens text-embedding-3-small: order of a few USD
• GPT-4o-mini with 2k context tokens: cents per conversation
• pgvector query <50 ms with index, bottleneck is LLM (TTFT ~300–800 ms)
• Cache embeddings for identical questions (hash normalized question), ~30% savings on support FAQ

Hosting on DevStudioIT Cloud (devstudioit.cloud): Route Handler with streaming, rate limits per IP, OpenAI keys in env.

Security and GDPR

• Do not index customer PII or internal B2B pricing
• Rate limiting and CAPTCHA on public /api/chat
• Conversation log retention, 90-day policy, anonymization
• Privacy policy notice on processing via OpenAI (DPA, EU region if required)

Success metrics after go-live

After 4 weeks in production measure:

• Answer accuracy, manual review of 50 random chats (target: >85% correct)
• Fallback rate, how often low similarity triggers “don't know” (target: 10–20%)
• Average chunks in context vs latency
• Clicks on source_url, user trust proxy

Dashboard in Branchly: chat_logs table with question, chunk_ids, user_rating. Iterate chunking quarterly, answer stability beats daily tweaking.

FAQ

Does RAG replace the knowledge base article?

No, that article covers what to include (FAQ, policies). Here is how to search and generate technically.

pgvector vs Pinecone?

pgvector in Branchly = fewer moving parts for small/medium projects. Pinecone/Qdrant when >500k chunks or multi-tenant isolation.

How often to re-index?

On content change, CMS webhook → partial re-ingest of changed URLs, not the whole base.

Can we skip OpenAI?

Yes, local embedding models (Ollama, sentence-transformers) + Llama; trade-off: PL/EN quality and GPU ops.

Multilingual?

locale filter in SQL + separate chunks; do not mix PL and DE in one embed without language normalization.

CTA

Need a chatbot that cites your documentation instead of inventing answers?

• Get a RAG chatbot quote, architecture, Branchly pgvector, Next.js, DevStudioIT Cloud
• AI chatbot for business, business process and costs