🏠 Local LLM কেন?
- Privacy — data কোথাও যায় না
- No API cost — unlimited use
- Offline — internet ছাড়া
- Customization — fine-tune freely
- No rate limit
- Compliance — HIPAA, GDPR friendly
🛠️ Tool Compare
| Tool | বৈশিষ্ট্য |
|---|
| Ollama ⭐ | Easiest, CLI + API |
| LM Studio | GUI, beginner-friendly |
| llama.cpp | Lowest level, fastest |
| vLLM | Production serving |
| Text Generation WebUI | Feature-rich UI |
| Jan | Open ChatGPT-like UI |
🚀 Ollama Quick Start
# Install (Mac/Linux/Windows)
curl https://ollama.com/install.sh | sh
# Pull model
ollama pull llama3.3
ollama pull qwen2.5-coder
ollama pull deepseek-r1
# Run
ollama run llama3.3
>>> "Hello!"
# API (OpenAI-compatible)
curl http://localhost:11434/v1/chat/completions \
-d '{ "model": "llama3.3", "messages": [...] }'
📊 Popular Open Model
| Model | Size | Use |
|---|
| Llama 3.3 70B | 43GB | General best |
| Llama 3.2 3B | 2GB | Fast, mobile |
| Qwen 2.5 Coder | Various | Coding |
| DeepSeek R1 | Various | Reasoning |
| Mistral 7B | 4GB | Efficient |
| Phi-4 | 9GB | Microsoft, small but smart |
| Gemma 2 | Various | Google |
💻 Hardware Requirement
- 3B-7B model — 8GB RAM, CPU OK
- 13B — 16GB RAM, GPU helpful
- 30B-70B — 32-64GB RAM + GPU (24GB+ VRAM)
- Mac M-series — unified memory advantage
- Quantization (4-bit, 5-bit) — half size, slight quality loss
🔧 Quantization
Original: FP16 (16-bit) — full quality, big
Q8_0: 8-bit — minor loss
Q5_K_M: 5-bit — sweet spot
Q4_K_M: 4-bit — small, slight quality drop
Q2_K: 2-bit — tiny, noticeable drop
llama3:70b-instruct-q4_K_M = 40GB
🎨 LM Studio (GUI)
- Browse Hugging Face models
- One-click download
- Chat UI
- Local API server
- Performance metric
- Beginner সবচেয়ে easy
🧪 vLLM (Production)
High-throughput serving — multiple concurrent request।
pip install vllm
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.3-70B-Instruct
# OpenAI-compatible API on port 8000
🔌 Local + Cloud Hybrid
- Sensitive data → local
- Complex query → cloud (GPT-4)
- Cost-sensitive bulk → local
- Latency critical → local
- Best quality needed → cloud
🛠️ IDE Integration
- Continue.dev (VS Code) → Ollama
- Cursor → custom local endpoint
- Aider → any OpenAI-compatible
- Open WebUI → ChatGPT-like for local
⚠️ Limitation
- Quality gap vs frontier model
- Slower than cloud (without GPU)
- Hardware investment
- No native vision/audio (some)
- Maintenance overhead
- Power consumption
💡 Use Case
- Code completion (privacy)
- Document summary (sensitive)
- Internal RAG (compliance)
- Bulk classification
- Offline travel/remote
- Learning experiment
- Hobby project (no API bill)