0:01When an LLM processes requests in a static batch, the fast requests finish early, but they can't leave.
0:07The GPU just sits there, waiting for the slowest one, while new requests pile up.
0:12This is a serious problem with static batching, one that continuous batching aims to address.
0:17So, in this video, we'll look at "continuous batching", and how it fixes the problems with static batching. But first, a quick recap.
0:27In the previous video, we saw three problems with static batching.
0:31First, batch lock-in. Once a batch forms, it's sealed. No requests can join or leave until every request finishes.
0:40Second. Wasted GPU cycles, Short requests finish early but their slots sit idle, The GPU can't reassign them.
0:48And third, queue starvation. New arrivals wait for the slowest request in the current batch before they get a turn.
0:56Continuous batching solves all three. Let's see how it works.
1:01But first, a few assumptions for our demo.
1:04We're using "hybrid batching" with "chunked prefill". When a new request arrives, the GPU processes its prefill tokens alongside the existing batch's decode tokens in the same forward pass.
1:15We cover this hybrid batching technique in a separate video in case you are interested in learning about it.
1:21For timing, a pure decode step takes about 10 milliseconds. A hybrid step, where decode and a prefill chunk happen together, takes about 25 milliseconds.
1:31And the requests have different input lengths and different output lengths, so they won't all finish at the same time. That's the whole point.
1:39With that in mind, let's step into the demo.
1:43We're going to step through a timeline of how continuous batching handles requests in real time.
1:49Let me walk you through the layout first.
1:51On the left is the timeline. It will log every event as it happens, arrivals, decode steps, hybrid steps, and completions.
1:60Up top on the right, we have the current time indicator, it tracks the simulation clock as events unfold.
2:06Below that is the request pipeline, On the left, the waiting area, where new requests queue up when they first arrive.
2:13And on the right, the prefill section, where the GPU processes their input tokens before they can join decode.
2:20Below the pipeline is the decode batch, where active requests generate tokens together.
2:25At the bottom left, the completed area, where finished requests will land with a checkmark.
2:30And next to it, these counters track our key metrics, steps completed and tokens generated.
2:36Now let's start the clock.
2:39Three requests, A, B, and C, have already arrived and been prefilled. At T equals zero, they're in the decode batch, ready to generate.
2:48The GPU status switches to "Decoding", meaning it's generating one token per request.
2:54Step 1 completes after 10 milliseconds. Notice the counter at the bottom tracking our progress.
3:00Three tokens generated, one for each active request.
3:05Step 2 begins. Another 10 milliseconds on the clock.
3:09And it completes. Two steps done so far.
3:12Six tokens total, three per step, since we have three requests in the batch.
3:17Step 3 starts. So far, this looks like standard batch processing. Three requests moving in lockstep, nothing new has joined or left.
3:27Now watch what happens.
3:29At T equals 25, request D arrives.
3:32See D appear in the waiting area? The batch is mid-step, so D has to wait until the current iteration finishes.
3:39In static batching, D would be stuck waiting for the "entire" batch. Continuous batching handles this differently.
3:47Step 3 finishes. That's 30 milliseconds on the clock. Now here's where continuous batching kicks in.
3:54The GPU forms a "hybrid" batch. D moves from the waiting area into active prefill, but look at the GPU status. It says "Hybrid", not "Prefilling."
4:04That's because A, B, and C are still decoding in the same forward pass. The GPU processes D's prefill chunk right alongside their decode tokens, all in one iteration.
4:15This hybrid step takes about 25 milliseconds instead of 10. It's a bit longer, but nobody is stalled.
4:22Hybrid step 4 completes. And look at the token counter, it went up. Three more tokens for A, B, and C, even while D was being prefilled.
4:31That's the key difference from alternating approaches. The existing batch never stops generating.
4:37But D has a long input context, one chunk wasn't enough. D still needs a second chunk. So
4:43Hybrid step 5 starts. Another combined iteration, A, B, and C decode while D's second prefill chunk processes alongside them.
4:53It completes at T equals 80. Fifteen tokens total. And D's prefill is now done, it's ready to join the decode batch.
5:02Coincidentally, request B has just finished and so it slides right over to the completed area.
5:07No waiting for A and C to finish. B is done and gone.
5:11With static batching, B would have had to sit there and wait until both A and C finished, even though it's already done. Here, it leaves immediately.
5:20And now D joins the decode batch alongside A and C. The batch went from three to two when B left, and now it's back to three.
5:28No wasted slots. The batch resizes dynamically.
5:32Let's pause the timeline for a moment.
5:34Look at what just happened. D arrived mid-batch, got prefilled in hybrid iterations alongside the existing batch, and joined as soon as it was ready.
5:43Meanwhile, B finished and left immediately, it didn't have to wait for A or C. And critically, A and C kept generating tokens the entire time.
5:52That's the core of continuous batching.
5:54You've probably got the idea by now, but let's run through a few more steps to see how the pattern reinforces itself.
6:01Request E arrives at T equals 88. It enters the waiting area, same pattern as D before.
6:08Step 6 completes. 18 tokens total. Now the scheduler can include E in the next iteration.
6:16E moves into prefill. The GPU forms another hybrid batch, A, C, and D keep decoding alongside E's prefill.
6:24E's input context happens to be shorter than D's.
6:28So its prefill finishes in a single hybrid step. And A, C, and D still generated tokens alongside it.
6:35E joins the batch right away. We now have "four" requests decoding together, A, C, D, and E.
6:43Think about what's happened. This started as a batch of three, requests have arrived and departed throughout, and the existing batch never stopped generating.
6:54Step 8 completes. We're up to 25 tokens generated across the simulation.
7:00And now D finishes and joins B in the completed area. That's two down. A, C, and E continue decoding without interruption.
7:09That's continuous batching in action. Now let's recap the key points.
7:15First, dynamic membership. Requests join and leave the batch freely. No lock-in and no idle slots.
7:22Second, hybrid iterations. Prefill and decode share the same forward pass, existing requests keep generating while new ones get processed.
7:32And third, full GPU utilization. The GPU stays busy through the entire process. Every cycle does useful work.
7:40These three properties are why continuous batching has become the standard approach for production LLM serving.
7:47Thanks for watching. Goodbye!

Continuous Batching: Dynamic Request Scheduling for LLM Inference

Created by Abhyuday DesaiUpdated 2 Oct, v2Runs 7 min 55 sec

0 views

About this video

A visual walkthrough of continuous batching for LLM inference. Covers hybrid iterations with chunked prefill, dynamic batch membership, and how the GPU stays fully utilized as requests arrive and complete at different times. Companion to the Static Batching video.

What changed

v202 Oct 2026abhydesai7 min 55 sec

Re-rendered in the studio with chapters and a transcript.

v1 has no transcript to compare. 10 sec shorter.

0 views0 likes on this version

v104 Apr 2026abhydesai8 min 5 sec

First published cut.

0 views0 likes on this version

Watch v1