The Brownout Dial

Controls

Arrival rate λ

How many requests per second arrive. Push it past capacity (servers × 3 req/s each) to trigger overload.

Servers c

How many workers serve requests in parallel. Capacity = servers × 3 req/s.

Timeout

How long a request waits before its client gives up. Tighter timeouts turn overload into wasted work faster.

Shedding (429 fast-reject)

Admission control — once too much work is already in flight, new arrivals are instantly rejected instead of queued.

Retries

Rejected or timed-out requests come back after a short delay, adding to the arrival load.

Retry budget

Caps how many retries can be in flight at once, so a retry storm can't snowball unbounded.

Run rescue sequence

Scripts a 20-second tour — overload with nothing on, then shedding, then a retry storm, then a retry budget — so the whole story plays hands-free.

What to watch for

• With shedding OFF and load high, watch Goodput climb, peak, then fall — even as Utilisation stays pinned near 100%.
• Turn Shedding ON at the same load: Reject rate jumps but Goodput recovers and holds flat.
• With Retries ON and no shedding, the collapse gets worse — rejected or timed-out work retries, adding MORE load to an already-overloaded system.

Why it happens

Servers process requests strictly in arrival order. Once the queue backs up past what timeouts allow, a server can spend its entire service time completing a request whose client already gave up — that's wasted work, not goodput. Under heavy overload almost every completion is wasted, so goodput collapses while the servers stay fully busy doing nothing useful. Shedding fixes this by capping how much work is ever admitted, so nothing waits long enough to become wasted work.