📊 কেন Observability?
LLM app non-deterministic — same input ভিন্ন output। Production-এ কী হচ্ছে দেখা ছাড়া debug অসম্ভব।
🔍 কী Track করবেন
- Request/response (full)
- Latency (p50, p95, p99)
- Token count (input/output)
- Cost per request
- Error rate, type
- User feedback (👍/👎)
- Model version
- Tool call sequence (agent)
- Cache hit rate
- Retry count
🛠️ Observability Tool
| Tool | বৈশিষ্ট্য |
|---|---|
| LangSmith | LangChain native, full trace |
| Helicone ⭐ | Drop-in proxy, dashboard |
| PromptLayer | Version + analytics |
| Langfuse | Open source |
| Arize Phoenix | OSS, eval focus |
| Datadog LLM | Enterprise APM |
| OpenTelemetry | Standard tracing |
🚀 Helicone Quick Setup
// Just change base URL!
const openai = new OpenAI({
baseURL: "https://oai.helicone.ai/v1",
defaultHeaders: {
"Helicone-Auth": `Bearer ${process.env.HELICONE_KEY}`
}
});
// Now all requests logged + dashboard
🏷️ Custom Property
headers: {
"Helicone-Property-User": userId,
"Helicone-Property-Feature": "code-review",
"Helicone-Property-Env": "production"
}
// Slice/dice in dashboard
🔬 Trace Example (Agent)
Request 12345 (3.2s, $0.04): ├─ Plan generation (0.8s, GPT-4o) │ └─ Output: ["search_web", "summarize"] ├─ Tool: search_web (1.5s) │ └─ Returned 5 results ├─ Tool: read_url (0.5s) └─ Final answer (0.4s, GPT-4o-mini)
📈 KPI Dashboard
- Daily active users
- Average cost per user
- p95 latency trend
- Error rate by feature
- User satisfaction score
- Token usage growth
- Top expensive queries
- Hallucination report rate
🚨 Alert Setup
- Cost spike (>2x normal)
- Error rate >5%
- Latency p95 >10s
- Cache hit rate drop
- Specific user abuse
- Model deprecation warning
🐛 Debugging Workflow
User reports: "Bot gave wrong answer" 1. Find request_id in trace tool 2. Inspect full prompt + response 3. Check retrieved context (RAG) 4. Verify tool call sequence (agent) 5. Compare to similar successful requests 6. Replay with adjusted prompt 7. Add to eval set as regression test
🧪 Eval-driven Development
- Test set maintain (real failure case)
- Every prompt change → eval run
- CI/CD-এ auto eval
- Threshold: regression block deploy
- Continuous expansion
💡 Pro Tips
- Day 1 থেকে log করুন
- PII redact log-এ
- Sample log (volume control)
- User_id-এ correlate
- Annotate failure cause
- Weekly review session
- Cost dashboard public to team