A card at 30% is not a card you need to replace. It is a card waiting.
In short
Generation is memory-bandwidth bound, not compute bound. One request at a time leaves most of the card idle, and batching fixes that without buying hardware.
Generation is memory-bandwidth bound, not compute bound. One request at a time leaves most of the card idle because the arithmetic units finish long before the weights arrive from memory.
That is why a 30% reading is not a hardware problem. The card has capacity; nothing is asking it to use that capacity. Batching converts headroom into throughput — the same weight matrix serves several requests in one pass instead of being read several times.
Static batching waits for a full batch, runs it, waits again. Throughput improves, latency worsens, and the worst case is a short request stuck behind a long one for the whole batch duration.
Continuous batching admits new requests as slots free up at each decode step. That is the mechanism behind most of the throughput gains people attribute to "a faster inference server" — the scheduler is doing the work, not the kernel.
The cost is that latency becomes a distribution instead of a number. Measure the mean and batching looks free; measure p99 or you will ship a regression nobody reports until users say "sometimes it hangs".
| Symptom | Likely cause | First thing to try |
|---|---|---|
| GPU 25–35%, latency fine | No batching | Enable continuous batching |
| GPU high, latency high | Batch too large for the cache | Cap max batch, shorten context |
| GPU low, CPU high | Tokenisation on the hot path | Batch the tokeniser |
| Spiky utilisation | No admission control | Limit queue depth, shed load |
Slightly, yes — while total throughput rises a lot. That is the trade. If one user's latency is a hard requirement, cap batch by size rather than by waiting for a full batch.
No — it lowers utilisation further. If the card is idle because nothing is scheduled on it, a bigger card is idle more expensively.
Only together with a KV cache calculation. Batch size multiplies cache usage, and the OOM that follows arrives under load — which makes it look like a stability problem instead of a sizing one.