High-Throughput GPU Inference Batching System Design

High-Throughput GPU Inference Batching System Abstract: How do you build a system that supports high-concurrency requests against an API you cannot change? This article walks through a complete infrastructure design for a GPU inference batching system — one that optimizes GPU utilization via a server-side batching mechanism that intelligently balances latency and throughput. From clarifying questions to deep trade-off analysis, this is a FAANG-level deep dive into one of the hardest infrastructure problems in applied ML. Table of Contents Clarifying Questions Crash Strategy & Key Points Elite Bonus Points (FAANG Rubrics) Functional Requirements Non-Functional Requirements Back-of-Envelope Estimation High-Level Design Low-Level Design Trade-offs, Alternatives & Optimizations 1. Clarifying Questions Before designing anything, you need to nail down assumptions. Here are the key questions — and the assumptions we'll carry forward: Question Assumption What is the peak QPS and the ta