Skip to content
Tony Nguyen

Case study · LLM serving

Helix: Building a Failure-Aware Control Plane for LLM Serving

A Go control plane in front of vLLM that decides what happens to a request when capacity runs out or a worker fails, and a workbench that turns those decisions into repeatable experiments.
Role
Solo personal projectDesign, build and evaluation
Year
2026
Stack
Go · gRPC + mTLS · HTTP/SSE · SQLite · Python · vLLM · React/TypeScript
Status
Validated once on a real GPU (Oct 2026); experiments run locally on simulated workers
All work

01Summary

A model server such as vLLM turns prompts into tokens. Something in front of it has to decide what happens when every worker is busy, when a worker dies halfway through a response, and when a retry might repeat text the user has already read. Helix is that layer. A single Go process makes those decisions, and separate Python workers run vLLM on the GPUs.

Each decision is an explicit rule with its own tests, and a workbench runs repeatable experiments to check that the rules hold. On one rented RTX 4090, Helix passed all five failure scenarios I tested, in a single run. Everything else runs on a laptop with simulated workers, where the experiments confirmed one fix and showed that one of my scheduling ideas fell short of its target. §7 has the evaluation and §8 what didn’t work.

  • 5 / 5

    reliability scenarios passed on the real GPU

  • 768 / 768

    capacity-comparison requests succeeded on the first attempt (384 through Helix, 384 direct to vLLM)

  • 2.39 s → 0.13 s

    longest wait behind a failed request, before and after the queue-stall fix (worst of 3 repeats)

The first two come from one session on a rented RTX 4090 with vLLM 0.11.0, recorded once; the third from simulated workers on a laptop.

02The problem

An LLM request is slow, holds scarce GPU capacity while it runs, and streams: the client reads the answer before it is finished. That turns ordinary request routing into six problems, each of which a user eventually sees as a timeout, an error, a stall or duplicated text.

Admission when workers are saturated
When every slot is busy, a request can wait or be refused. An unbounded wait hides overload until memory or deadlines run out; an early refusal tells the client to back off.
Scheduling
Workers differ in which model they hold, how busy they are and which prompt prefixes they have cached. Loading a model takes seconds, so the choice of worker can matter a lot, or not at all. That has to be measured.
Worker crashes
A crashed, hung or disconnected worker looks the same from outside. The control plane needs a rule for when a silent worker counts as gone, and for the requests it held.
Deadlines and cancellation
A request that is cancelled or past its deadline should free its GPU slot, but generation runs in another process, so stopping it is asynchronous.
Safe retries
A second attempt is a new sample from the model. Once the client has part of the answer, a retry would show it a repeated or different opening, so there must be an exact point after which a request is never retried.
Control-plane and data-plane coordination
The process that decides is not the process that generates. Messages between them can be late, or come from a worker process that has since been replaced; such output must never reach a client or free capacity.

Helix makes each of these an explicit rule in code, and each rule something an experiment can measure.

03Requirements and constraints

What Helix must guarantee

  • No duplicated output. Once any part of the answer may have reached the client, the request is never replayed.
  • No stale effects. Output from a replaced process, connection or attempt never changes a request or frees capacity.
  • Bounded resources. The queue and every buffer have a fixed size; overload produces a refusal, not growth.
  • One explicit outcome. Every request ends exactly once (succeeded, failed, timed out or cancelled) with a reason, and every decision is recorded so an experiment can show what a rule costs.

What it had to work within

  • Helix cannot see when bytes reach the client’s socket, so the retry boundary must be an event it controls: a fragment entering the request’s output buffer.
  • Neither Helix nor vLLM can resume a stream from an offset, so a failure after output ends the request; only attempts with no output are retried.
  • A worker declared lost may still be computing, so there is no exactly-once guarantee, only a guarantee that such output is ignored.
  • A rented GPU host may accept no inbound connections, so workers dial out to the control plane over mutual TLS and open no port.
  • There was one GPU session on a personal budget, so real-hardware evidence is one 31-minute session on a rented RTX 4090 (about $0.46, estimated from the hourly price), and everything else runs on one laptop.
  • Simulated workers measure the control plane, not the GPU, so every run record says where it ran, and their latencies are read as queueing and scheduling behaviour only.

04Architecture

Helix has two kinds of process. helixd is the control plane: one Go process on one machine that owns every decision. Workers are separate Python processes, usually one per GPU host, each running vLLM beside it on loopback. Clients only ever talk to helixd.

Helix processes (stacked)Clients talk HTTP to helixd, one Go process that makes every decision. Each GPU host runs a Python worker beside vLLM and dials out to helixd over gRPC with mutual TLS. The workbench starts helixd and the workers for each run.ClientsHTTP POST in, SSE outhelixdcontrol plane: one Go processHTTP APIPOST in · SSE outControl decisionsadmission · queueplacement · leasesretries · fencingSQLite historygRPC + mTLS, dialed by workerGPU HOST · DATA PLANEPython workerloopback HTTPvLLMWorkbench (test harness)Starts a fresh helixd and workers per run,sends HTTP traffic, checks 14 invariants.
Fig. 1The processes and where they run. The control plane is one process; workers are separate processes, usually on other hosts, and each opens its own connection to it. The workbench is a test harness, not part of serving.
Read Fig. 1 as text
  1. Clients send HTTP POST requests to helixd and receive the answer as server-sent events (SSE).
  2. helixd is one Go process on one machine (the control plane). It holds the HTTP API, the decisions (admission, queue, placement, leases, retries, fencing) and a SQLite history of finished requests.
  3. Each GPU host runs a Python worker and vLLM (the data plane). The worker calls vLLM over HTTP on loopback and dials out to helixd over two gRPC streams with mutual TLS. There can be several such hosts.
  4. The workbench starts a fresh helixd and fresh workers for every experiment run, sends HTTP traffic and records the result.

So Helix is a single-node control plane, not a distributed one: there is no replication or consensus, and if helixd stops, live requests are lost. The distributed-systems problem sits at its edge, in workers on other machines that can pause, disconnect or be replaced while they hold a request. In the real-GPU run, helixd ran on a Mac and the worker in a rented GPU Pod in the EU.

Four terms carry the rest of this article.

Lease
A worker’s right to receive work, renewed by heartbeats and timed on helixd’s clock. When it expires, the worker is treated as gone.
Incarnation
One run of a worker process. Each process picks a random ID at start, so a restarted worker is a new incarnation and is never mistaken for the old one.
Connection epoch
A random 128-bit value assigned to each worker connection and carried on every frame, so a frame with an old epoch belongs to a replaced connection.
Fencing
Rejecting a message from an incarnation, connection or attempt that is no longer current: it is counted and dropped before it can change a request or free capacity.

Where a request goes

Fig. 2 follows one request. The client posts JSON (1). Admission inserts it into the bounded FIFO queue in the same step, or answers HTTP 429 (2). On the next placement pass, the policy picks an eligible worker and reserves a slot (3). The assignment goes out over the control stream the worker opened (4), and the worker calls vLLM on loopback (5). Fragments return on a separate output stream, pass the fencing checks, and enter the request’s 32-fragment output buffer, which streams to the client as server-sent events (6).

Where a request goes (stacked)In order: the client posts; admission queues the request or returns 429; placement reserves a worker slot; the hub sends the assignment to the Python worker, which calls vLLM on loopback; fragments return to the hub, which fences stale frames, pass the retry boundary into the output buffer, and stream to the client as SSE.ClientPOST /v1/requestsAdmissionHTTP 429 when the queue is fullBounded FIFO queuePlacementpolicy picks a worker,reserves a slotWorker hub (mTLS)helixdassignment (control stream)Python workerloopback HTTPvLLMGPU HOSTfragments (output stream)Worker hubfences stale framesRetry boundary→ output bufferhelixdSSEClientaccepted · token · done
Fig. 2One request through helixd and back. Numbers follow the text; accent marks the request’s path. In the wide layout, the dotted box is the Manager’s lock: everything inside it changes under one mutex.
Read Fig. 2 as text
  1. The client sends HTTP POST /v1/requests to the HTTP API inside helixd.
  2. Admission (inside the Manager lock) inserts the request into the bounded FIFO queue, or returns HTTP 429 with Retry-After when the queue is full.
  3. Placement takes the head of the queue, asks the policy (round_robin, residency_first or prefix_affinity) for an eligible worker from the worker registry, and reserves a slot.
  4. The worker hub sends the assignment over the control stream that the worker dialed, using gRPC over mutual TLS.
  5. The Python worker on the GPU host calls vLLM over HTTP on 127.0.0.1.
  6. Fragments return over the worker’s output stream to the hub, which fences stale frames, then pass the retry boundary into the request’s output buffer and stream to the client as SSE.
  7. Also inside the Manager lock: the retry decision (with deadlines) and the worker registry (leases, incarnations, slot reservations).

All live state (the queue, request records, the worker registry and slot reservations) sits behind one mutex in a single Manager. Choosing a worker, reserving its slot and dequeuing the request happen inside that lock; network calls and generation happen outside it.

The worker opens both streams because rented GPU hosts often accept no inbound connections. They are separate so that slow output cannot delay the heartbeats that keep the lease alive, and every buffer on the way is bounded: if one assignment’s 32-frame buffer overflows, only that assignment fails.

Who owns each decision

The split between deciding and generating is strict: every decision belongs to helixd, and the worker and vLLM only execute.

Table 1: Decisions and their owners
DecisionOwnerRule
Admit or refusehelixd: admissionBounded FIFO; HTTP 429 when full
Which workerhelixd: placement policyEligible workers only; a slot reserved under the lock
Is a worker alivehelixd: worker registry, on its own clockLease renewed by heartbeats; expiry or an UNAVAILABLE error ends it
Retry or failhelixd: retry decisionRetry only with no output handed off, on another incarnation, within budget
Is this output currenthelixd: worker hub, then the ManagerFencing checks on every frame
Generate tokensvLLM, through the workerModel, batching, KV cache; Helix never generates
Stop generatingThe worker, on a cancel from helixdBest effort, and asynchronous
Table 1Decisions and their owners.

05Key decisions

Four decisions shape everything else. Each one starts from a constraint above, and each has a cost I chose to pay.

1.Retry only before the first output handoff

A retry is safe only if the client has seen nothing, and Helix cannot observe the client’s socket. So the boundary is the last event Helix controls: a fragment being accepted into the request’s output buffer. Before it, a failed attempt may be retried once, on a different incarnation; after it, a failure ends the request FAILED with whatever was sent.

Two independent checks enforce it: the retry decision requires that no fragment was handed off, and the request’s state machine refuses RETRY_PENDING once one was, so a bug in either cannot cause a replay.

Request lifecycle and retry boundary (stacked)QUEUED, ASSIGNED and RUNNING in a column. RUNNING is split by the retry boundary: above it no fragment has been handed off and a failure moves to RETRY_PENDING, which is re-placed on another incarnation; below it a failure ends FAILED. RETRY_PENDING ends FAILED with no_alternate_worker or retry_queue_full. Terminal states: SUCCEEDED, FAILED, TIMED_OUT, CANCELLED.QUEUEDplacedASSIGNEDRUNNING0 fragmentshanded off≥ 1 fragmenthanded offretry boundary(Manager.Emit)RETRY_PENDINGTERMINALSUCCEEDEDFAILEDTIMED_OUTCANCELLEDafter outputno_alternate_workerAny nonterminal state may also endTIMED_OUT (deadline) or CANCELLED(DELETE, client disconnect).
Fig. 3Request lifecycle and the retry boundary. State names as in the code. There is no separate streaming state: what matters is whether any fragment has been handed off, so RUNNING is drawn split. Accent marks the retry path and the boundary that closes it. Edges from every nonterminal state to FAILED are omitted.
Read Fig. 3 as text
  1. States: QUEUED, ASSIGNED, RUNNING and RETRY_PENDING are nonterminal; SUCCEEDED, FAILED, TIMED_OUT and CANCELLED are terminal and have no outgoing transitions.
  2. QUEUED → ASSIGNED when placement reserves a slot; ASSIGNED → RUNNING when the attempt is dispatched; RUNNING → SUCCEEDED on a valid completion.
  3. Retry path: ASSIGNED or RUNNING → RETRY_PENDING, only while no fragment has been handed to the output buffer and the retry budget allows. RETRY_PENDING → ASSIGNED on a different worker incarnation.
  4. The retry boundary: once the first fragment is handed off, RUNNING can no longer move to RETRY_PENDING; a failure ends the request FAILED with partial output.
  5. RETRY_PENDING → FAILED with no_alternate_worker when only the failed, still-healthy incarnation could serve it, or retry_queue_full when the queue is full. Any nonterminal state may also end TIMED_OUT (deadline) or CANCELLED (DELETE or client disconnect).
Rejected:
Replaying partial output (repeated or divergent text); resuming from an offset (needs deterministic decoding and engine support); a boundary at socket-write completion (a finished write is still not proof of receipt).
Cost:
Conservative: a fragment buffered but never delivered still disables retry, so some recoverable failures surface as errors. The reverse never happens.
Revisit when:
A client acknowledgement or replay protocol exists and is tested end to end.

2.Fence by incarnation, connection and assignment

Worker names and addresses are reused across restarts, so neither can tell an old process from a new one. Helix checks three identities instead: the incarnation, the connection epoch, and an assignment ID unique to each attempt, with a contiguous sequence number on every frame. A frame is applied only if all three are current. Slot reservations belong to the incarnation that made them, so a late completion from an old process cannot free a slot its replacement is using.

Rejected:
Trusting the worker name or endpoint (both are reused across restarts); a per-worker generation counter (needs durable state).
Cost:
A brief network break costs every request in flight on that worker, because nothing resumes. Old incarnation IDs (up to 65,536) are kept in memory until the control plane restarts.
Revisit when:
Workers must survive a control-plane restart, which needs durable membership.

3.Bound the queue, and keep it strictly FIFO

Admission and queue insertion are one atomic step into a single FIFO with a fixed bound (128 by default). When it is full the client gets HTTP 429 with Retry-After: 1, and the refused request is never stored or run. Retries count against the same bound and rejoin at the tail.

If the head has no eligible worker, everything behind it waits. That head-of-line blocking keeps the baseline easy to reason about, and it is measured rather than hidden: in one local experiment a blocked head held two eligible followers for about 475 ms, against 2.6 ms without it. It is also why the defect in Failure handling could stall a whole queue.

Rejected:
Unbounded buffering (hides overload until memory or deadlines run out); per-model queues.
Cost:
A head with no eligible worker blocks followers that could run.
Revisit when:
A measured workload justifies a fairness-aware queue, with starvation tests.

4.One process, one lock, live state in memory

Every change to live state happens under one mutex in one Go process, which makes the important operations atomic without a protocol: two requests can never take the same slot. Live state is deliberately not persisted; SQLite stores only finished requests and scheduler decisions.

Rejected:
A Python-only control process; several cooperating services; a durable SQL work queue, Redis or a workflow engine.
Cost:
If helixd restarts, live requests are lost and workers must be restarted. History writes still happen under the lock, so slow storage can delay scheduling and heartbeats.
Revisit when:
Lock contention is measured, or recovering in-flight work becomes a requirement.

06Failure handling

Two scenarios show how the rules combine: a worker that disappears while holding a request, and a defect in which one failed request stalled a queue.

A worker goes silent mid-request

Suppose a worker pauses or loses its network while it holds a request (Fig. 4). When its lease expires, helixd marks that incarnation expired and closes its connection; an UNAVAILABLE error from the worker does the same at once. The attempt fails, and the retry rule decides what happens next.

  1. Client → helixd

    The client posts a request; helixd accepts it.

  2. helixd → worker A

    helixd sends assignment 1 over worker A’s connection.

  3. Worker A

    A pauses or loses its network. Its heartbeats stop.

  4. helixd

    A’s lease expires on helixd’s clock: A is marked expired and its connection closed. No fragment was handed off, so the request is queued for a retry with A excluded.

  5. helixd → worker B

    Attempt 2 goes to B: another worker, or the same worker restarted as a new process.

  6. Worker B → client

    B streams fragments; helixd forwards them to the client as SSE, then done.

  7. Worker A → helixd

    A frame from A that was still in flight carries A’s old epoch and attempt. It is fenced: counted as stale and dropped.

Fig. 4What happens when a worker goes silent. One request, with no output handed off when its worker goes silent. Had a fragment already been handed off, step 4 would end the request FAILED instead.
Read Fig. 4 as text
  1. The client posts a request; helixd accepts it.
  2. helixd sends assignment 1 to worker incarnation A over A’s connection.
  3. A goes silent (a pause or a network partition): its heartbeats stop.
  4. A’s lease expires on helixd’s clock. helixd marks A expired and closes its connection. Attempt 1 handed off no fragments, so the request becomes RETRY_PENDING with A excluded.
  5. helixd places attempt 2 on incarnation B: another worker, or the same worker restarted as a new process.
  6. B streams fragments; helixd hands them to the client as SSE, ending with done.
  7. A frame from A that was still in flight arrives with A’s old epoch and attempt. It fails the fencing checks and is counted and dropped.

With no output handed off, the request rejoins the queue with that incarnation excluded and runs on another: a second worker, or the same worker restarted as a new process. If none appears before the deadline, it ends TIMED_OUT. With output handed off, it ends FAILED and the client keeps what it has. Anything the old incarnation still sends is fenced.

On the real GPU, the harness killed its worker mid-stream: the request ended FAILED on its first attempt with no replay, and the worker was marked expired. With two simulated workers, a failure injected before any output moved to the second worker and succeeded in 3 of 3 repeats; a failure after two fragments ended FAILED, unretried, in 3 of 3.

A queue stall: one failed request blocked everything behind it

With a single worker, a request that failed before any output was queued for a retry that excluded that worker. The worker stayed healthy, so no replacement would ever register and the retry could never be placed. Under strict FIFO it sat at the head until its deadline (30 s by default, up to an hour), and every request behind it waited too while the only worker sat idle. Some of those requests timed out after 7 of their 8 fragments, so their clients got truncated answers.

A design review of the retry path found it, and a deterministic regression test with a fake clock reproduced it: after 20 ticks of 100 ms the follower was still queued, with the only worker healthy and idle.

The fix adds one check to placement. If the head can only ever run on incarnations it already failed on, and one of those is still healthy, it ends FAILED with no_alternate_worker and placement moves on (Fig. 5). A retry still waits when another worker could serve it later, when its excluded worker is already lost (a replacement may register), and on a first attempt.

Before the fix

Before the fixThe head is a retry that excludes w1, the only worker, so it can never be placed. w1 is HEALTHY and idle. The followers wait behind the head until its deadline; the head ends TIMED_OUT and some followers time out too.bounded FIFO · strict orderhead (retry)RETRY_PENDINGnextQUEUEDnextQUEUED…w1 · HEALTHYidle0 slots in useexcluded: its attemptfailed on w1followers waitbehind the headuntil its deadlineHead TIMED_OUT; 2, 1, 2 followerstimed out too.

After the fix

After the fixPlacement sees that only the excluded, still-healthy w1 could serve the head, ends it FAILED no_alternate_worker, and places the followers on w1 in FIFO order.bounded FIFO · strict orderhead (retry)RETRY_PENDINGnextQUEUEDnextQUEUED…w1 · HEALTHYidle0 slots in useFAILEDno_alternate_workerat onceplacedAll 39 other requests succeeded,3/3 repeats.
Fig. 5The stall, and the check that ends it. One worker, w1, whose first request fails before output. Before the fix the head can never be placed; after it, the head ends at once and the queue drains.
Read Fig. 5 as text
  1. Before the fix: one worker, w1, healthy and idle. The head of the FIFO is a retry whose attempt on w1 failed before output; w1 is excluded for it, so it can never be placed. The requests behind it wait until the head’s deadline, and some time out. After the fix: placement sees that only the excluded, still-healthy w1 could serve the head, ends it FAILED with no_alternate_worker, and places the followers on w1.

The regression test now passes. A liveness invariant, written after the review, checks every workbench record for this stall. The invariant did not discover the defect; it makes the stall visible in any record and would flag a regression.

Then a before-and-after experiment changed only the helixd binary (the “before” build disables the check with a one-line edit), with the same checkout, configuration and seeds: one simulated worker with 2 slots that fails its first request before output, 40 requests from 4 concurrent clients, a 2.5 s deadline, 3 repeats per binary.

Before the fix, in every repeat, 3 requests waited more than a second behind the failed head and 1 or 2 of them timed out. After it, no request waited longer than 129 ms and the other 39 succeeded every time (Fig. 6). I report the longest wait rather than the p95: with 4 concurrent clients at most 3 requests can queue behind a stalled head, so the p95 of 40 requests lands on one of them by accident.

Fig. 6Longest wait behind the failed request, per repeat. Simulated workers on one laptop: control-plane timings with a configured service time, not GPU latency. Before the fix, the waiting requests were released only when the head reached its 2.5 s deadline.

The fix has a price. With one worker, a transient error on a healthy worker now fails the request at once, and a worker that would have registered before the deadline no longer gets the retry. No test covers that second case yet.

Three smaller defects the tests and experiments surfaced
  • A worker completion race

    In an early scheduler experiment, one request was rejected by its worker before output because the worker reported completion before releasing its local capacity, so Helix could reassign while the old stream was still closing. The retry rule did its job and the request succeeded on its second attempt. The worker now releases local state first, with a test that pauses exactly at that point.

  • Cancel cleanup could fail the next request

    Freeing a slot wakes placement at once, so the next assignment could reach a one-slot worker still closing the cancelled attempt’s vLLM stream. A 1 ms cleanup was enough for the follower to be rejected. The worker now waits up to 1 s, never past the new deadline, for pending cancellations: 8 of 8 followers failed before and 8 of 8 succeeded after, through the real vLLM adapter with a 5 ms stream close.

  • A dispatch in the last millisecond killed the worker

    An assignment sent inside a request’s final millisecond carried a timeout of 0, which the worker treated as a protocol violation and exited, losing every attempt it held. The hub now sends nothing with under 1 ms left, and the worker refuses such an assignment on its own and keeps serving.

07Evaluation

Read this before any latency number

Two sources, two questions. One real-GPU session shows the rules hold with vLLM on real hardware; experiments on simulated workers show how the control plane behaves under controlled load and injected faults.

Simulated-worker latencies are not GPU latencies. The real helixd drives workers whose time to first fragment and per fragment are configured, so their numbers measure queueing, scheduling and retries only.

The real-GPU run was recorded once. It is not a benchmark, and it does not show that Helix is faster than vLLM.

How the workbench measures

An experiment defines arms that differ in one factor (a setting, a worker topology or an injected fault) over the same workload and seed. For every arm and repeat, the workbench starts a fresh helixd and workers, sends requests through the public HTTP API, and records the client’s view, every attempt and state change, and every placement decision. A checker then evaluates 14 invariants on the saved record, such as “every accepted request ends exactly once” and “no attempt that produced output is retried”. Runs whose workload, seed, platform or code differ are reported as not comparable.

Helix Lab request inspector for one request in a recorded run: status Failed after 1 attempt; a panel explaining that attempt 1 on w1 failed before output, a retry was scheduled but could not be placed, so the request ended FAILED no_alternate_worker; then client-clock and server-clock timelines.
Fig. 7One request in a run record. The workbench’s browser front end, opened on the failed request from the queue-stall experiment after the fix, with client and server events on their own clocks. Simulated workers.

On a real GPU

The validation ran on 2026-10-15 in one 31-minute session on a single NVIDIA GeForce RTX 4090, rented as an on-demand cloud Pod in the EU. It served Qwen2.5-0.5B-Instruct (FP16, 2,048-token context) with vLLM 0.11.0 in eager mode at a GPU memory fraction of 0.5, with three separate limits: 8 slots on the worker and a queue of 8 in Helix, and at most 4 concurrent sequences in vLLM. helixd ran on a Mac; the worker ran in the Pod beside vLLM and dialed out through a reverse SSH tunnel, so no inference port was exposed.

Each reliability scenario had an invariant declared in advance, checked against Helix’s records and vLLM’s metrics counters. All five passed.

Table 2: Five reliability scenarios on the real GPU
ScenarioHow it was inducedObservedLimit of the claim
Token accountingOne request, output capped at 64 tokensvLLM’s counters moved by 7 prompt and 64 output tokens; Helix recorded 7 and 64Counters compared after the request, not atomically
Cancellation after outputClient cancelled after the first fragmentCANCELLED on attempt 1, no replay; Helix and vLLM back to 0 activeAsynchronous: vLLM generated 20 tokens, 4 were delivered
Deadline500 ms request deadlineTIMED_OUT after 27 fragments; 500.22 ms on the server, 501.23 ms at the client; no retryAsynchronous: 43 tokens generated, 27 delivered
Worker death after outputHarness killed its worker mid-streamFAILED on attempt 1, no replay; worker marked expiredProcess death, not an isolated engine fault
Bounded admissionNo healthy worker; 8 requests, then a 9th8 queued; the 9th got HTTP 429 with Retry-After: 1A refused request has no ID, by design
Table 2Five reliability scenarios on the real GPU. One run each, on 2026-10-15.

A capacity comparison then ran a closed loop of 1, 2, 4 and 8 concurrent clients down two routes, through Helix and directly to vLLM, each request asking for exactly 128 tokens at temperature 0, 96 requests per cell. All 768 succeeded on the first attempt, with no retries or partial streams.

Through Helix, the median time to first token was 229–239 ms at 1–4 clients; on the direct route it was 457–546 ms. At 8 clients both were about 1,570 ms. The full table, with p95, is in the reference section.

That gap is not Helix’s doing. Through Helix, requests rode one persistent gRPC stream to the worker; the direct route opened a new HTTP connection per request through an SSH-forwarded port. The time between fragments was about the same on both routes (median 11.8–12.3 ms), so the difference is all in time to first token. Per-request connection setup over the long Mac-to-EU link is the most plausible cause; that is an inference, not a measurement.

At 8 clients vLLM’s limit of 4 sequences became the bottleneck on both routes, and they converged. GPU utilization averaged 12–16%, so none of this says anything about the RTX 4090’s capacity.

Inside Helix, queue time from acceptance to assignment had a median of 4.7–5.7 ms (p95 10.4–12.7 ms) at every concurrency, and the scheduling decision itself took a median of at most 0.002 ms.

On simulated workers

An experiment on simulated workers, with configured service times, isolates one control-plane choice.

What a queue bound costs. One worker with 4 slots, a seeded Poisson stream of 40 requests per second, 120 requests per arm. The arms set the load to about 0.5, 1 and 2 times capacity and, at 2, the queue bound to 8 or 64. One repeat, so no spread estimate.

Table 3: Queue bound against offered load
Load (measured)Queue boundAcceptedRefused (429)Queue wait p95Met latency target
0.498120027 ms100%
1.04811010203 ms92%
2.0786456401 ms13%
2.126411913,100 ms9%
Table 3Queue bound against offered load. Simulated workers, one run. “Met latency target” is the share of sent requests that met both latency targets.

At twice capacity, the short queue refuses almost half the requests and keeps the wait near 400 ms; the deep queue accepts almost everything and makes everyone wait about 3.1 s, so even fewer meet the target. Neither is free; the workbench shows the trade instead of choosing.

What this does not show

  • That Helix adds negligible overhead: the route comparison cannot isolate it.
  • Behaviour on other GPUs, larger models or a tuned engine; per-token latency distributions; p99 (fewer than 100 samples per cell).
  • A synchronous engine abort: cancellation and deadline cleanup were asynchronous.

08What didn’t work

Prefix affinity failed its pre-registered criterion

prefix_affinity sends follow-up turns of a chat to the worker most likely to still have their prompt prefix cached. I tested it on a synthetic fixture runtime, not vLLM and not a GPU, against criteria written in advance. The mechanism worked: follow-up turns stayed on their home worker every time (52–74% for round robin), and the cache hit rate rose from about 0.70 to 0.82.

The primary criterion still failed. It asked for a median time-to-first-token ratio against round robin of at most 0.80, with every repetition below 1.0, at 4 users with an ample cache and at 6 with a constrained one. The ratios were 0.67 and 0.98 at 4 users, 0.77 and 0.92 at 6; most of the spread came from round robin’s luck in where turns landed. The criterion was revised once, after early dry runs and before the final one these figures come from, and it now stays as registered: the result is published as a failure, not re-tuned into a pass. A companion criterion, that a queue bound derived from the latency target keeps goodput under overload, passed.

An earlier dry run had caught a design flaw before any GPU time was spent: a shared system prompt pinned every new conversation to one worker. The fix, borrowed from SGLang’s cache-aware router, counts a match only if it covers at least half the prompt.

The real-GPU comparison was confounded

I wanted the capacity comparison to show what Helix costs on top of vLLM. It cannot: the two routes used different transports, so any difference mixes the transport with Helix. The next GPU session needs both routes on the same transport, and a measured round-trip time, before any overhead claim.

No universal winner for residency-aware routing

The mechanism works: in a two-worker experiment with a configured 200 ms model-load penalty, round robin loaded a model 14–16 times per run and residency-first never did. But a 288-run matrix on simulated workers, against a criterion declared in advance (every paired repetition improves p95 by at least 10%), found a benefit only with one client at a time and a load cost of at least 100 ms. At 2, 4 or 8 clients no cell met it: when the worker holding the model is busy, a locality preference has little to choose from.

What I took from these

  • Write the invariant for the failure you just found, and keep it. It turns the next occurrence into a failed check in any record.
  • Fix the criterion before the data, and publish it when it fails. A criterion that moves after the data arrives measures nothing.

09What I’d improve next

  • Run the pre-registered GPU session. The prefix-affinity and goodput plan for real vLLM is written with fixed thresholds (two Qwen2.5-1.5B-Instruct instances on one RTX 4090, about 70 minutes). No GPU result exists yet.
  • Accept bring-your-own-GPU on real hardware. The path where a developer’s own GPU host dials in has only run end to end with a CPU stand-in for vLLM.
  • Take history writes out from under the lock. Synchronous SQLite writes under the Manager’s mutex keep ordering simple but let slow storage delay scheduling and heartbeats.
  • Measure head-of-line blocking before changing FIFO. The workbench records how long a head blocks eligible followers, but no experiment exercises it yet; any change to strict FIFO needs its own starvation tests.

10Reference

Detail for readers who want the exact rules. Everything above stands without it.

The fencing checks on every token and result frame
Table 4: A frame is applied only if every check holds
CheckChecked by
The request is not finished; the frame names its current attempt and assigned incarnation; the request is ASSIGNED or RUNNINGManager
The incarnation is current for its worker, HEALTHY or DRAINING, with a live leaseManager
The connection epoch matches the live connectionWorker hub
The assignment ID is known, unfinished, and bound to this request and attemptWorker hub
The sequence number is the previous one plus one; phases arrive in order (acknowledged, running, fragments, completed or failed)Worker hub
The assignment’s 32-frame buffer has room; an overflow fails that assignment instead of growing the bufferWorker hub
Table 4A frame is applied only if every check holds.
What the control plane does in each failure
Table 5: Outcome by failure situation
SituationBehaviour
Queue fullHTTP 429 with Retry-After: 1; never stored or run
No eligible workerStays queued until capacity changes or the deadline passes
Backend error before any output handoffRetry on another incarnation if budget remains; FAILED no_alternate_worker if only the failed, still-healthy incarnation could serve it
Failure after any output handoffFAILED with partial output and a reason; never retried
Lease expiry or replacementOld results fenced; outstanding attempts failed or retried by the same rules
Deadline passesTIMED_OUT; best-effort cancellation; no retry
DELETE or client disconnectCANCELLED; best-effort cancellation; no retry
Client reads too slowly (32-fragment buffer full)FAILED slow_consumer; attempt cancelled
Control-plane crashLive state lost; finished history remains; workers must restart
Table 5Outcome by failure situation.
Placement policies and eligibility
Table 6: The three placement policies
PolicyRuleStatus
round_robinAmong eligible workers, the one after the last selected. Ignores residency and load.Baseline
residency_firstPrefer a worker that already holds the requested model, least busy first; otherwise the least busy eligible worker.Default
prefix_affinityPrefer the worker whose cached prompt prefixes cover at least half the prompt, unless it is much busier than the least busy; otherwise the least busy.Tested only on a synthetic fixture
Table 6The three placement policies.

Every policy chooses only among workers that are healthy with a live lease, advertise the requested model, have a free slot, were not used by an earlier failed attempt of this request, and are idle or already serving the same model.

Full real-GPU capacity table, with p95
Table 7: Capacity comparison by concurrency and route
ClientsRouteLatency p50 / p95, msTTFT p50 / p95, ms
1Helix1,738 / 1,855239 / 259
1Direct to vLLM1,966 / 2,111546 / 598
2Helix1,768 / 1,932231 / 275
2Direct to vLLM1,976 / 2,175457 / 590
4Helix1,782 / 2,097229 / 346
4Direct to vLLM2,067 / 2,255513 / 570
8Helix3,142 / 3,3351,569 / 1,829
8Direct to vLLM3,104 / 3,6151,570 / 2,119
Table 7Capacity comparison by concurrency and route. Client-observed, successful requests only, nearest-rank percentiles over 96 requests per cell. Different transports per route.
Components and technology choices
Table 8: Components by layer
LayerChoice
Public APIHTTP with JSON in and server-sent events out (accepted, token, done)
Control planeGo; one Manager and one mutex for all live state
Worker transportProtocol Buffers over gRPC; two worker-dialed streams per worker; mutual TLS with a pinned certificate per worker identity
HistorySQLite for finished requests and scheduler decisions; live state stays in memory
WorkersPython asyncio agent with a vLLM backend and a simulated backend for tests
Model runtimevLLM 0.11.0, unchanged, on loopback beside the worker
Token accountingFragments counted separately from the token usage vLLM reports, which is authoritative
WorkbenchPython runner, 14 invariants, run comparison and reports, with a React/TypeScript front end
Table 8Components by layer.
Known limits
  • Not built: multi-node KV-cache-aware routing, disaggregated prefill and decode, autoscaling, Kubernetes integration, exactly-once execution.
  • Leases are not consensus. Pauses and network problems can make a healthy worker look dead; Helix prefers fencing its output over recovering it.
  • The operator HTTP API has no authentication and listens on loopback only.
  • Token usage is not reported for failed or cancelled requests.

About this project

Helix is a solo personal project: I designed and built it, with AI coding assistants (Claude Code and Codex) as part of my development workflow. The code is private; this article draws on its design records, tests and run records.