Tutorial

Table of Contents Setting Up LLaVA/BakLLaVA with vLLM: Backend and API Integration Why vLLM for Multimodal Inference The Challenges of Serving Image + Text Prompts at Scale Why Vanilla Approaches Fall Short How vLLM Solves Real-World Production Workloads Configuring Your…

Table of Contents The Rise of Multimodal LLMs and Efficient Serving with vLLM Introduction to Multimodal LLMs What Are Multimodal LLMs? Milestones in Multimodal LLM Evolution Flamingo (DeepMind, 2022) GPT-4V (OpenAI, 2023) LLaVA (Large Language and Vision Assistant, 2023) BakLLaVA…

Table of Contents Post Training Qwen3 for Math Reasoning Using GRPO Group Relative Policy Optimization (GRPO) Challenges with Proximal Policy Optimization (PPO)? Computational Overhead and Memory Requirements Value Function Instability and Representation Collapse Hyperparameter Sensitivity and Training Instability Bias in…

Table of Contents Preparing the BLIP Backend for Deployment with Redis Caching and FastAPI Introduction What We’re Building in This Lesson Why Redis Caching Matters for Inference What Is Caching? What Is Redis? Configuring Your Development Environment Running a Local…

Table of Contents Meet BLIP: The Vision-Language Model Powering Image Captioning What Is Image Captioning and Why Is It Challenging? Why It’s Challenging Why Traditional Vision Tasks Aren’t Enough Configuring Your Development Environment A Brief History of Image Captioning Models…

Table of Contents Synthetic Data Generation Using the VLM-as-Judge Method Configuring Your Development Environment Set Up and Imports Downloading Images Locally Using Qwen as VLM-as-Judge Converting the JSON File to the Hugging Face Dataset Format Inspecting the Dataset Pushing the…

Table of Contents Synthetic Data Generation Using the BLIP and PaliGemma Models Why VLM-as-Judge and Synthetic VQA Configuring Your Development Environment Set Up and Imports Download Images Locally Inference with the Salesforce BLIP Model Convert JSON File to the Hugging…

Table of Contents Fine Tuning SmolVLM for Human Alignment Using Direct Preference Optimization What Is Preference Optimization? Types of Techniques Reinforcement Learning from Human Feedback (RLHF) Reinforcement Learning from AI Feedback (RLAIF) Direct Preference Optimization (DPO) Identity Preference Optimization (IPO)…

Table of Contents Run YOLO Model in the Browser with ONNX, WebAssembly, and Next.js What Is Browser-Based Inference and Why Does It Matter? Why Run YOLO in the Browser? No Server Required Instant Demos and Prototypes Low Latency and High…