📚 Production RAG Pipeline
Documents → Chunking → Embedding → Vector DB
↓
User Query → Embed → Retrieve → Rerank → LLM → Answer
✂️ Chunking Strategy
- Fixed size — 512 token, easy
- Recursive — paragraph → sentence fallback
- Semantic — embedding-based split
- Document-aware — markdown header, code block
- Overlap — 10-20% prevent context loss
🧬 Embedding Models
- OpenAI text-embedding-3-large (best general)
- Cohere embed-multilingual (Bengali support)
- BGE-M3, E5 (open source)
- Voyage-2 (code, finance specialized)
🗄️ Vector DB Options
- Pinecone — managed, fast
- Weaviate — hybrid search
- Qdrant — open source, Rust
- pgvector — Postgres extension
- Chroma — local dev
🎯 Reranking (Critical!)
from sentence_transformers import CrossEncoder
# Retrieve top 20
candidates = vector_db.search(query, k=20)
# Rerank to top 5
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
scores = reranker.predict([(query, c.text) for c in candidates])
top_5 = sorted(zip(candidates, scores), key=lambda x: -x[1])[:5]
🧪 RAG Evaluation
- Retrieval — precision@k, recall@k, MRR
- Generation — faithfulness, answer relevance
- Tools — Ragas, TruLens, DeepEval
🔧 Advanced Techniques
- Hybrid search (vector + BM25)
- HyDE (hypothetical doc embedding)
- Multi-query expansion
- Parent-child chunking
- Contextual retrieval (Anthropic)
🎓 মূল কথা ⭐
RAG = retrieval quality × generation quality। Bad retrieval → bad answer। Reranker সবসময় ব্যবহার করুন। Evaluation ছাড়া production-এ যাবেন না।