🔁 Prompt Engineering = Iterative কাজ
প্রথম prompt খুব কমই সেরা। Test করুন, measure করুন, refine করুন।
🎯 Optimization Loop
1. Success metric define 2. Baseline prompt 3. Diverse input-এ test 4. Failure mode identify 5. Hypothesis → variation 6. A/B compare 7. Winner adopt 8. Repeat
📏 Success Metric
- Accuracy (ground truth-এর সাথে match)
- Format compliance (valid JSON %)
- Latency, token cost
- User satisfaction (👍/👎)
- Task completion rate
- Hallucination rate (sample audit)
📋 Test Set তৈরি
২০-১০০টা representative example — easy, hard, edge case।
[
{ "input": "...", "expected": "...", "category": "easy" },
{ "input": "...", "expected": "...", "category": "hard" },
{ "input": "...", "expected": "...", "category": "edge" }
]
⚖️ LLM-as-Judge ⭐
Stronger model দিয়ে weaker model-এর output evaluate।
"Rate this response 1-5 on: - Accuracy (factual correctness) - Relevance (question answer করে?) - Completeness - Tone (professional) Justify each score."
🚨 Common Failure Mode
- Reasonable request refuse
- Hallucinated fact
- Wrong format (markdown, যখন JSON চেয়েছিলেন)
- Truncation (max_tokens hit)
- Off-topic drift
- Tone mismatch
- Repetition
- Over-hedging ("It depends...")
🔧 Fix Technique
| Problem | Fix |
|---|---|
| Hallucination | RAG, "say I don't know" |
| Wrong format | Schema, example |
| Drift | Constraint, max_tokens |
| Hedging | "Be decisive" |
| Refusal | Reframe, context |
| Verbose | "Max 100 words" |
| Slow | Smaller model, streaming |
🛠️ A/B Testing Tool
- PromptLayer — version + analytics
- LangSmith (LangChain) — full tracing
- Helicone — cost + observability
- Weights & Biases Prompts
- OpenAI Evals (open source)
- Promptfoo — CLI eval
📝 Promptfoo Example
# promptfooconfig.yaml
prompts:
- "Summarize: {{text}}"
- "TL;DR: {{text}}"
providers: [openai:gpt-4o-mini]
tests:
- vars: { text: "..." }
assert:
- type: contains
value: "key point"
- type: llm-rubric
value: "Is this concise and accurate?"
📂 Version Control ⭐
prompts/ ├── system/ │ ├── code_review_v1.md │ ├── code_review_v2.md (current) │ └── changelog.md ├── user/ └── eval_data.json # Git-এ track, PR-এ review prompt change
✅ Production Checklist
- ☐ Objective clear stated
- ☐ Test set created (২০+ case)
- ☐ Metric defined
- ☐ Baseline measured
- ☐ কমপক্ষে ৩টা variant tested
- ☐ Edge case covered
- ☐ Format validation
- ☐ Cost analyzed
- ☐ Documented in version control
- ☐ Rollback plan
📊 Live Monitoring
- Input, output, latency, cost log
- User feedback signal
- Drift detection (output distribution change)
- Anomaly alert (refusal spike)
🧠 Meta-prompting
LLM-কেই prompt improve করতে দিন।
"You are a prompt engineering expert. Analyze this prompt and suggest improvements for: - Clarity - Specificity - Hallucination reduction - Output consistency Original prompt: [my prompt] Return: 1. Issues identified (numbered list) 2. Refined prompt 3. Expected improvement"
💡 Golden Rule
Don't guess, measure। A/B test ছাড়া prompt change production-এ push করবেন না।