Inference Batch Planner
Build synthetic GPU inference batches and inspect the throughput–waiting-time trade-off.
TryAt 140 requests/s, compare Even and Poisson. Reseed Poisson; then compare 0 vs 80 ms batch wait at 140 and 20 req/s.
Watch forAt 140 req/s, fuller synthetic batches may improve throughput. Compare p95 wait; this is not a hardware benchmark.
The linked story describes related work; the scenarios and results here are synthetic.
Interactive model
Skill practicedBalance inference throughput against worker queue wait
Connection to my workNeuralEye documents batched inference; this model explores queueing trade-offs, not GPU performance.
Understand the result
Choose evenly spaced arrivals or a seeded Poisson stream at the same average rate, then feed that synthetic load to a small inference service. The model groups requests up to a batch-size limit, may hold a partial batch briefly for more arrivals, and schedules batches across workers using an illustrative service-time curve. The batch-formation wait cap is separate from total queue wait: a busy worker can add more time before a request starts. The Poisson model does not create observed bursts. This is not a hardware benchmark, a Triton replica, or NeuralEye production data.
Controls
- Arrival pattern
- Compare an evenly spaced stream with Poisson-distributed inter-arrival times at the selected average rate. Poisson arrivals vary around that rate; they are not a replay of clustered production bursts.
- Arrival rate (requests/s)
- Sets the average rate of the seeded synthetic request arrivals. Higher offered load can fill batches faster and can also outpace the worker pool.
- Maximum batch size
- Caps how many requests the scheduler groups before dispatching a batch.
- Time to form a batch (ms)
- Sets how long a partial batch may wait for more arrivals before dispatch. This caps batch formation only; a busy worker can add queue wait after dispatch. Zero sends each request on its own.
- GPU workers
- Sets the number of parallel synthetic workers that can run batches.
- Reseed
- Creates another deterministic arrival sequence while keeping the selected service settings.
What to watch for
- At the same average rate, compare Even with Poisson. Even arrivals keep the offered count fixed in the 12-second window; Poisson arrival count and timing depend on the seed.
- At zero batch-formation wait, each batch should contain one request and completed throughput should match the single-request baseline. A busy worker can still leave requests queued.
- At high arrival rates, add workers and compare completed requests with the backlog. The batch-formation cap does not cap total queue wait: p95 counts each offered request from arrival until a worker starts, including worker-availability wait.
- Reseeding keeps the service settings but changes the arrival sequence. Compare seeds to see how timing shifts these finite-window results.
Why it happens
The synthetic service curve includes a fixed launch overhead plus additional work that grows more slowly than batch size. Grouping requests can amortize that overhead, but a request may wait both while a partial batch gathers arrivals and after dispatch while workers are busy. The configured cap applies only to the first wait; p95 measures arrival through worker start. Real systems have model-, hardware-, shape-, memory-, and scheduler-specific behavior; this model is illustrative.
How this idea relates to the linked work
NeuralEye’s project story describes batching and parallel inference in distributed GPU workflows. This synthetic experiment isolates the serving trade-off between grouping requests, worker capacity, and total wait; it is not a NeuralEye benchmark or production-serving trace.
See the broader engineering brief