Manuel Gil

Series 3: Prompt Engineering & MLflow

🎯 Focus: Giving the Bot a “Brain”

Now we’re going to give our bot intelligence! We’ll integrate an LLM API and set up MLflow to track and optimize the bot’s responses through prompt engineering.

📚 Topics Covered

Integrating the LLM API

  • Choosing an LLM provider (OpenAI, Anthropic, Cohere, etc.)
  • Setting up API authentication and credentials
  • Understanding token limits and costs
  • Error handling and rate limiting

Replacing Dummy Replies

  • Replace the simple rule-based replier with LLM calls
  • Design prompts that work well for YouTube comments
  • Handle different types of comments intelligently
  • Ensure responses are appropriate and on-brand

Setting up MLflow

  • Install and configure MLflow
  • Understanding experiments and runs
  • Logging prompts, parameters, and metrics
  • Comparing different approaches

Logging Prompt Experiments

  • Structure your experiments for reproducibility
  • Log prompt variations
  • Track response quality metrics
  • Monitor performance over time

Versioning the Best-Performing Prompts

  • Register models in MLflow
  • Create model stages (Staging, Production)
  • Transition prompts as you improve them
  • Rollback to previous versions if needed

🚀 What You’ll Build

By the end of this series, you’ll have:

  • ✅ An AI-powered bot that generates intelligent replies
  • ✅ MLflow tracking your experiments
  • ✅ A versioning system for prompt management
  • ✅ The ability to compare and optimize prompts data-driven way

🧪 Experiment Loop

Design Prompt
    ↓
Run in MLflow Experiment
    ↓
Evaluate Responses
    ↓
Log Metrics & Results
    ↓
Compare Against Baseline
    ↓
Promote Best to Production

📝 Prerequisites

  • Completion of Series 2 (Bot Skeleton)
  • Understanding of LLM APIs and prompt design
  • Familiarity with experiment tracking concepts

💡 Key Concepts

  • Prompt Engineering: Crafting prompts to get better responses from the LLM
  • Experiment Tracking: Recording all trials and their outcomes
  • Model Registry: Maintaining versions of your best prompts/settings
  • Reproducibility: Being able to recreate any previous result

🎬 Watch & Follow Along

Follow the video as we integrate GPT/LLM API and set up MLflow tracking. Use this guide for:

  • Code snippets and API examples
  • MLflow configuration
  • Prompt templates to start with

Next Step: Once your AI-powered bot is working well and tracked in MLflow, Series 4 will automate its execution using GitHub Actions.