The LMCache Coordinator: a control plane for fleet-wide KV cache

LMCache's MP mode gives every node its own KV cache server, but each server sees only itself. The coordinator is the control plane that makes them act as one cache: it builds a shared, fleet-wide view from cache events and lets pluggable policies act on it. Our goal is one KV cache for the whole fleet, whatever the inference workload, from online serving and agents to RL rollouts, run by policies that adapt to each workload and a control plane that scales from one cluster to many.

Contents

1. Motivation

2. Architecture

3. Pluggable design

4. Fault tolerance

5. Observability

6. Future directions

7. References

Motivation: from per-node KV Cache to a fleet-wide control plane

Background: KV cache, offloading and MP mode

Every LLM inference request starts with prefill: the model reads the whole prompt and builds up the state that every later token depends on. In full-attention layers that state is the keys and values of each token; in hybrid models, linear-attention and Mamba layers keep a fixed-size recurrent state instead. LMCache calls all of it the KV cache. Prefill is the expensive part, so inference engines like vLLM and SGLang keep this cache in GPU memory and reuse it whenever a new request shares a prefix, skipping prefill for that prefix and cutting Time To First Token.

That reuse only works while the cache is still on the GPU, and contexts keep getting longer. Multi-turn chats, agents and agentic RL rollouts with long tool traces, and RAG prompts full of documents, all produce more KV than GPU memory can hold for long, so it is evicted before it can be reused. KV cache offloading keeps it alive by moving it to cheaper tiers, such as CPU DRAM, disk or remote storage, and loading it back on reuse, which costs far less than recomputing it.

Offloading alone leaves one weakness: when the cache lives inside the engine process, it shares the engine's fate, and a restart or crash can wipe it out. LMCache's multi-process (MP) mode removes that tie by running the cache as a standalone server on each node, decoupled from the engine. The engine focuses on inference and the cache survives worker restarts. The cache also cannot crash the engine, its CPU work stays off the inference path, the engines on a node share one L1 cache in CPU DRAM, and an L2 tier such as local disk or S3 sits behind it. For a gentle introduction to MP mode and how KV moves between the engine and the cache server, see Understanding LMCache MP Mode Transfer Paths.

But each server is still an island

MP mode solves caching within a node. A real deployment, though, is a fleet: tens of nodes, each with its own MP server, whether they serve users or generate rollouts for RL training. Each server is independent by design and knows only about itself.

Here is where that hurts first. Suppose an RL job samples sixteen responses from each prompt, spread over four nodes with only CPU offloading. Node A prefills a long prompt and keeps its KV in L1. Moments later, nodes B, C and D get samples of the same prompt, and each prefills it again from scratch, because none of them knows node A already holds it. The KV they need sits in a peer's memory, one network hop away.

The same blind spot shows up all over the fleet. A few examples:

1. Lookup. Where in the fleet does a prefix live? And with non-prefix caching, which cached chunks does a request contain, at any offset, on any server, as when a RAG request reuses documents? Both questions need a fleet-wide index, not a per-node one.

2. Online scheduling and policies. While requests run, something has to decide where work goes and how its KV moves: which node still has room in L1, whether to load KV from a peer or recompute it, where to store new KV, and what to evict. Today each server makes these calls alone, seeing only itself.

3. Offline, proactive fleet-wide cache control. Outside the request path, an operator or an RL scheduler wants to promote a prompt's KV into node C's memory before a traffic shift or the next rollout batch, demote cold KV to cheaper storage, move KV from a busy node to an idle one, pin a hot system prompt so it is never evicted, or delete one tenant's data. Today that means talking to every server one by one with manual scheduling.

Why a coordinator

Networking solved this kind of problem long ago by splitting the data plane from the control plane. Routers move packets fast; a separate control plane learns the whole topology and tells them where to send things. KV cache offloading has so far been almost all data plane: moving bytes between GPU, CPU, disk and network as fast as possible. What is missing is the part that sees the whole fleet and decides.

That is the coordinator: one standalone process that builds a fleet-wide view of every MP server's cache and uses it to enforce quotas, evict, pin, prefetch and look things up across machines.

A process that sees the whole fleet could easily become its bottleneck, so we currently keep the coordinator out of the inference path. A request that hits an MP server's local cache is served without ever talking to it. The coordinator only advises: its view of the fleet can lag slightly behind, so a server checks with the owner before using KV held somewhere else. If the coordinator goes down, inference keeps running on every node, and only the fleet-wide features wait for it to come back.

The vision

Our vision is one KV cache for the whole fleet. Every node's GPU memory, CPU memory and storage should behave like a single pool, so KV computed anywhere can be found and reused everywhere, whether the fleet is serving users or generating rollouts for RL. The MP servers stay the data plane that moves the bytes; the coordinator becomes the programmable control plane that decides what to keep, where to put it, and when to move it. Those decisions should be policies anyone can write and swap, so new workloads, new hardware and new ideas can change how the cache behaves without touching the engines.

Architecture: one stream in, plain HTTP out

The coordinator is currently a FastAPI process with a simple shape. Facts about the fleet's caches flow in as one event stream. Decisions flow back out as ordinary HTTP calls to endpoints the MP servers already have. Everything in between is either a view (what is true) or a controller (what to do about it).

Every cache fact enters through the event gate, while fleet membership arrives separately through register and heartbeat calls. Every action leaves as an HTTP call to a server's existing endpoint. The controllers are the pluggable part.

Follow one KV chunk through the system:

1. A server reports it. When an MP server stores, reads or deletes a chunk, it records a cache event and sends a batch of them to the coordinator about once a second.

2. The events travel directly over HTTP by default, or through Kafka in production.

3. The event gate admits them. It is the single entry point for every fact about the fleet's caches.

4. The broadcaster fans them out to every component that consumes events.

5. Views turn events into a picture of the fleet: the registry (who is alive), the key directory (where each chunk lives) and usage (bytes per tenant and server). Views are shared by all controllers; state specific to one policy, like an LRU order, stays private to its controller.

6. Controllers act on that picture. Built-in ones handle eviction and prefetch; controllers are pluggable, so you can add or replace them without changing LMCache.

7. Decisions go out as plain HTTP to endpoints the MP servers already expose, such as DELETE /cache/objects or POST /cache/prefetches.

The split between views and controllers matters later: views are the shared picture every controller reads, and controllers hold the policy, which is the part we made pluggable.

Pluggable design: controllers as plugins

The built-in eviction policy is a reasonable default, not the right answer for every deployment. One team wants an L1 budget as well as an L2 one. Another wants deletes sent to the exact server that holds a copy, because its L2 is local disk rather than a shared bucket. A third wants a placement policy nobody upstream has thought of yet. So we made the policy layer, the controllers, something you can add and replace without forking LMCache.

A controller is one class. You subclass the Controller marker, and the coordinator finds it by scanning a package. There is no list to edit in create_app and no field to add to a config class. Beyond that, a controller opts into extra roles just by having the right method:

It can live outside the LMCache tree. Put your controllers in an ordinary Python package and name it at startup. Here a package called acme_controllers adds a controller that serves /acme/reaper, with its own setting:

lmcache coordinator --extra-config '{
  "controller_packages": ["acme_controllers"],
  "acme.reaper_interval": 10
}'

It can replace a built-in. Loading a new eviction controller is not enough, because the built-in one would still run and the two would fight over the same state and the same endpoints. So you switch the built-in off by name:

lmcache coordinator --extra-config '{
  "controller_packages": ["acme_controllers"],
  "disabled_controllers": ["FleetEvictionController"]
}'

Three small rules make swaps safe:

• Endpoints belong to their controller. /quota and /cache/pins come from the eviction controller's own get_routers(). Disable it and they disappear, so a replacement can serve the same paths and existing clients keep working.

• State belongs to the component, not the class. Durable sections are named by what they hold (pins, quotas), so a replacement inherits the pins and quotas the old controller saved.

• Typos fail loudly. A package that does not import, or a disabled name that matches nothing, stops the coordinator at boot. An operator who thinks they disabled something is worse off than one who sees an error.

Controllers may only depend on views. A controller asks for the views it needs (registry, key directory, usage) and gets the same shared instances everyone else reads. It cannot reach another controller. Shared state is what views are for; if two controllers need each other's policy, they were really one controller. This keeps an out-of-tree controller from breaking when a peer ships somewhere else. For the same reason, views are not pluggable: which views exist is the coordinator's contract with every controller.

Fault tolerance: being wrong safely

A control plane fails in two ways. It can go down, or it can stay up and quietly believe something false. The first is easy to survive here, because the coordinator is off the hot path: servers keep serving, and fleet features wait. The second is the dangerous one, so most of the design is about keeping the picture correct through restarts, crashes and lost messages.

When a server restarts or dies

An MP server restart wipes its DRAM, so every L1 entry it reported is now a lie. The coordinator catches this with the incarnation number. When the first batch from a new incarnation arrives, the gate fences the old one: every consumer drops the L1 entries that server reported. L2 entries are deliberately kept, because those bytes sit on disk or in S3 and survive the restart. They leave only when a DELETE event says so.

A server that crashes and never comes back is caught by heartbeats instead. If it is silent for 30 seconds (checked every 10), the health loop removes it and fences its L1 entries the same way.

When events are duplicated or lost

Every server's stream is ordered by seq, and that per-server order is the only ordering the design needs. Each server is the only writer of its own facts, so there is no global clock and no cross-server tie-breaking. A batch with an old seq is a retry or a redelivery, and the gate drops it. Applying an event is also idempotent: storing twice is an upsert, and deleting something absent is a no-op.

A skipped seq means events were lost. The gate flags that server's slice as possibly stale and keeps going. With Kafka, loss is rare: the server's producer buffers up to 64 MB and retries in order for up to five minutes, so a broker restart delays events instead of dropping them.

When the coordinator restarts

The coordinator writes two files, because its state comes in two kinds:

Derived state is large and changes constantly, so it is written periodically; operator intent is small and irreplaceable, so it is saved on every change. The checkpoint also stores the Kafka offset that produced its views, and that offset is the only read cursor, so after a restart the restored state and the resume point always agree. Taking a checkpoint briefly pauses ingest, so the saved views are consistent with each other.

Every persistence failure is survivable. A missing or corrupt checkpoint means a cold start, not a crash, and a failed write is retried on the next tick. In the same spirit, if one view fails on a batch, only that view misses it.

High availability today

Today, high availability means a fast, correct restart of a single instance, not a hot standby. A restart loses no events when the deployment has:

1. Kafka retention longer than any expected outage.

2. --checkpoint-path and --metadata-path on a persistent volume.

3. A liveness probe on GET /healthz.

4. replicas: 1 with the Recreate update strategy. The default RollingUpdate briefly runs two coordinators, and two consumers in one group would each see only part of the fleet.

Observability

Like any service, the coordinator tracks request rates and latencies, for both the calls it serves and the calls it makes to MP servers. But those numbers alone miss its real risk. Everything it knows comes from the servers' events, so if events are lost, late or half-applied, it keeps answering and quietly answers wrong. Its observability therefore starts from the event stream and the work built on top of it, and asks four questions:

1. Is the view complete? Did any batches go missing, did a server admit to dropping events, and did any view fail to apply a batch? A failed apply means two views now disagree, and that is worth waking someone up for.

2. How old is it? How long does a stored chunk take to show up in the directory, and has a server stopped sending events while still heartbeating?

3. Are the loops alive? When did the health, checkpoint and eviction loops last finish a pass?

4. What is each controller doing? Did it start, how much work is it doing, and what is it failing at?

A few rules keep this cheap and honest. Each component keeps one stats() snapshot, and both metrics and the status page read it, so they never disagree. Labels come only from fixed lists; per-server values are gauges keyed by instance_id that disappear when the server leaves, and keys or hashes are never labels. And a scrape must never walk a whole index under a lock, because a slow scrape would stall ingest. Counts are kept up to date as events are applied instead.

Two things are deliberately not done. /healthz never fails just because the coordinator is degraded, because Kubernetes would restart it and throw away its state. And there is no readiness check that waits for warm-up, because it would block the very events needed to warm up.

Every controller reports on itself. Each controller has an up/down gauge, so one that fails to start is visible at once, and a controller with its own loop reports when that loop last finished a pass. Each role then adds its own metrics. Eviction counts sweeps, keys selected, keys skipped and why (pinned, or no server can delete them), and delete requests sent or failed. Quota reports each tenant's bytes used against its limit, pins report how many keys are protected, and prefetch tracks requests submitted, in flight, time to complete and how many requested keys were actually found.

These metrics are named by role, not by class: whichever controller enforces eviction emits the eviction.*, quota.* and pins.* metrics. A replacement may add labels and metrics but never rename them, so swapping in your own eviction controller keeps every dashboard and alert working.

Everything is exported as OpenTelemetry metrics under the lmcache_coordinator. prefix, scraped by Prometheus from GET /metrics or pushed to a collector with --otlp-endpoint. For people, GET /status shows the same snapshots on one page: per-server stream state, Kafka position, controller and loop health, and the last checkpoint.

Future directions: from knowing to deciding, from one coordinator to many

Today the coordinator mostly knows: where every chunk lives, how much each tenant uses, how full each server is. Its decisions are still simple, such as LRU eviction under a quota, or a prefetch someone explicitly asked for. A full control plane needs six capabilities: state, lookup, policy, action, scale and reliability. The diagram above and the table below show where the coordinator stands on each today and where it goes next, and the rest of this section walks through the biggest shifts.

Smarter policy. Inference workloads reuse KV in very different ways, and the biggest gains ahead come from policies built around them. Multi-turn chat and coding agents keep returning to long, growing contexts. Multi-agent workflows hand work between roles in patterns we can learn. RL rollouts sample the same prompt many times across many nodes, and RAG requests share documents at arbitrary offsets. Each pattern deserves its own placement, prefetch and eviction strategy, and each can arrive as a controller reading the shared views, with no new data path. The plugin model and record-and-replay make this practical: a new policy ships as an out-of-tree controller, and it can be compared, tuned or even learned against a real fleet's replayed traffic before it touches production.

Taken one step further, people need not write these policies by hand. A self-evolving agent could own the policy layer: watch the fleet through the shared views and metrics, propose a new controller, evaluate it against replayed traffic, and promote it only when it beats the current one, then repeat as workloads shift. The coordinator already has the pieces such a loop needs: controllers as the unit of change, replay as a safe test bench, and observability as the score.

Richer actions. Smarter policies need more ways to act than delete, pin and prefetch. Promoting KV into a node's memory, demoting it to cheaper storage, and moving or duplicating it across nodes would let a policy reshape where the cache lives, not only trim it.

Cheaper lookups. As policies lean on the coordinator, lookups move closer to the request path and event volume grows with the fleet, so the cost of each call starts to matter. Today the coordinator speaks plain HTTP with JSON, which is easy to read and debug. Alternatives such as gRPC offer typed contracts, streaming for events and cheaper calls, and the MP servers already offer gRPC to engines, so the right protocol is worth revisiting as the fleet grows.

Growing with the fleet, and beyond it. As fleets reach thousands of nodes and inference spans both serving and RL training, one process will not hold every view or make every decision. The coordinator itself may also become distributed, or even hierarchical: a coordinator per rack or cluster making fast local decisions, a parent above them setting fleet-wide policy, and warm standbys taking over without a cold start. The same structure stretches across datacenters and clouds, where cooperating coordinators place and move KV with bandwidth and price as inputs. The event-driven design makes this possible: every view is derived from ordered per-server event streams, so state can be split, rebuilt or moved by replaying streams instead of copying it.

Every step keeps the same shape as today: events in, shared views, pluggable policy, plain actions out. That shape is what lets the coordinator grow from a fleet-wide directory into the control plane for one KV cache across the whole fleet.

References

Papers

• Yihua Cheng et al. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv:2510.09665, 2025.

• Jiayi Yao et al. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. EuroSys 2025.

• Rui Zhang, Chaeeun Kim, Liting Hu. A Policy-Driven Runtime Layer for Agentic LLM Serving. arXiv:2605.27744, 2026.

Blog posts

• Jiayue Chen, Tony Lin and the LMCache Team. Understanding LMCache MP Mode Transfer Paths: A Beginner's Guide. LMCache Blog, June 2026.

Design docs (in the LMCache repository, dev branch)

• Coordinator overview: REST surface, layout, lifecycle

• Cache-event ingest: event gate, sources, Kafka cursor

• Cache-event emission: the MP-server side and its sinks

• Key directory and blend index

• Usage tracking and eviction, warm prefetch, fleet memory usage

• Controllers and plugins

• Durable state

• Observability

• User guide: running the coordinator

Issues and pull requests (LMCache/LMCache)

• Control-plane RFC, #4226

• Backbone and membership: #3488, #3516

• Quota, usage and eviction: #3630, #4310

• Warm prefetch: #3827

• Key directory from cache events: #4275, #4292

• Fleet CacheBlend lookup: #4438

• Ingest layer and controllers: #4526

• Durable state: #4653, #4678

• Pluggable controllers: #4723, #4760, #4895

• Kafka transport: #5285, #5286, #5354, #5464

• Record and replay: #5312, #5345

• Observability: #4389, #4800, #5533