TL;DR
LMCache’s Raw Block backend turns an unmounted block device (or a pre-sized file for development) into a server-owned KV cache tier. It stores KV objects in fixed-size slots, bypasses the filesystem and page cache when configured with O_DIRECT, keeps recovery metadata on the same device, and supports the LMCache multiprocess (MP) L2 path.
The backend began as a deliberately small Rust pread/pwrite implementation. The community then added aligned direct-buffer I/O, io_uring with registered buffers and batching, NVMe io_uring_cmd passthrough, MP integration through a shared RawBlockCore, CI-safe testing, and NVMe Flexible Data Placement (FDP). Later work hardened alignment, recovery, lifecycle, and accounting behavior on real devices.
Today, Raw Block is useful when a deployment wants a local, high-capacity, restart-recoverable KV tier with explicit control over the I/O path. It is not a filesystem, a general-purpose block allocator, or a promise that raw I/O will beat every buffered-storage configuration. Its value is a controlled storage path that LMCache can optimize for large KV objects.
1. Why a Raw Block KV Cache Tier?
Long-context, multi-turn, RAG, and agentic workloads can generate far more reusable KV state than GPU HBM or host DRAM can retain. Local NVMe offers much more capacity, but a conventional file backend also brings filesystem metadata, page-cache policy, and file lifecycle overhead that do not always match KV cache behavior.
KV cache objects have several useful properties for a block-oriented design:
1. They are large relative to ordinary metadata operations.
2. LMCache already knows their identity and lifecycle.
3. They can be placed in fixed-size, aligned slots.
4. A server-owned backend can coordinate allocation, eviction, and recovery.
The original Raw Block RFC therefore asked a narrow question: can LMCache own the on-device layout and issue direct I/O without building a full filesystem?
The answer became a Rust-backed storage path with three layers:
RawBlockCore owns cache semantics, while the Rust layer owns low-level I/O and buffer safety. The MP adapter can evolve with LMCache’s control plane without duplicating the device layout.
2. Starting Small: Rust, Fixed Slots, and O_DIRECT
The first merged implementation, PR #2482, intentionally used synchronous pread and pwrite. It introduced:
• a Rust RawBlockDevice exposed to Python;
• raw-device and regular-file size discovery;
• fixed-size slot allocation;
• O_DIRECT support to bypass the page cache;
• explicit handling of short reads and writes; and
• Python error propagation for low-level I/O failures.
Starting with a small path made the invariants visible: device offsets and total I/O lengths must be aligned, payloads must fit their slots, and a slot must not be recycled while a write is still in flight.
The next step, PR #2573, made the direct-I/O path more useful. When a caller provides an aligned CPU buffer, the Rust layer can issue I/O directly from that memory. If only the tail is unaligned, LMCache can use a bounce buffer for the tail instead of copying the entire object. Alignment was also plumbed into LMCache’s CPU allocator so that zero-copy was an end-to-end property rather than a local optimization hidden inside Rust.
The resulting POSIX path is conceptually simple: aligned LMCache CPU buffer → Python buffer view → Rust pointer validation → pread/pwrite with O_DIRECT → NVMe.
When the buffer, offset, or transfer length cannot satisfy the selected path’s constraints, LMCache either uses an aligned bounce buffer where supported or rejects the operation instead of silently issuing unsafe I/O.
3. Recovery Belongs on the Device
A raw device has no filenames or directory tree. LMCache therefore needs its own durable mapping from cache keys to physical slots.
PR #2614 moved recovery metadata onto the raw device. A reserved metadata region stores versioned checkpoints describing the committed key index and slot layout. At startup, RawBlockCore validates the checkpoint before rebuilding its in-memory state.
The important rule is that recovery follows a durable committed state. Subsequent fixes made that rule stronger:
• PR #3700 rejects checkpoints whose block_align or header_bytes do not match the active layout.
• PR #4078 rebuilds reusable slots from committed entries and next_slot, avoiding leaks caused by crash-time snapshots of the free list.
• PR #4283 reports recovered slot sizes to the MP accounting path so quota and eviction state include recovered data.
• PR #3169 added load_checkpoint_on_init=false for operators who deliberately want an empty logical cache without changing the on-device layout.
This makes Raw Block different from a volatile runtime tier: after a clean or crash restart, LMCache can reconstruct the key index from the device, subject to matching layout and metadata versions.
4. From POSIX to io_uring and NVMe Passthrough
The initial synchronous path established correctness, but small I/O and concurrent submissions need lower per-operation overhead.
4.1 io_uring with batching and registered buffers
Ankit Kumar added the first io_uring engine in PR #2635. The implementation introduced a dedicated submission/completion worker, implicit batching to reduce io_uring_enter calls, batch completion tracking, and fixed-buffer registration for aligned paged CPU memory.
The PR’s microbenchmark reported:
| Write size | POSIX | io_uring + fixed buffers | Observation |
| 64 KiB | 3,144.8 ops/s | 11,245.8 ops/s | About 3.6× in that setup |
| 4 MiB | 742.0 ops/s | 748.5 ops/s | Essentially equal |
Note that io_uring primarily reduces submission and syscall overhead. As object size grows and device bandwidth dominates, the relative benefit can shrink.
4.2 io_uring_cmd for direct NVMe commands
After the MP refactor, PR #3274 restored the io_uring path and added NVMe io_uring_cmd passthrough. Instead of opening /dev/nvme0n1 as a block device, this mode opens a namespace character device such as /dev/ng0n1 and submits NVMe commands through IORING_OP_URING_CMD.
This removes the filesystem and most of the block path, but it also exposes device constraints that the kernel normally hides:
• transfers must respect namespace LBA geometry;
• large I/O must be split by queue byte and segment limits;
• multi-page PRP/SGL mappings require aligned userspace buffers; and
• an SQE build failure must complete the logical operation rather than leave a waiter blocked forever.
These are not theoretical details. Nayeon Kim fixed unaligned multi-page read buffers in PR #3891 and made SQE-build errors fail cleanly in PR #4294. Sangyoon Kwon aligned temporary passthrough buffers in PR #3841 and bounded automatically selected transfer sizes by both max_hw_sectors_kb and max_segments * page_size in PR #3882.
Bypassing the block layer moves alignment, splitting, completion, and error-handling responsibilities into the application rather than removing them.
5. Raw Block in LMCache MP Mode
LMCache MP mode runs storage in a standalone server process. Inference workers communicate with that server, while L1 and L2 lifecycle decisions stay inside LMCache.
PR #3119 introduced the MP Raw Block L2 adapter and extracted the shared RawBlockCore. The adapter implements the MP L2 contract for asynchronous store, lookup-and-lock, load, unlock, delete, usage reporting, and status reporting. Prefetch loads directly into caller-provided L1 buffers; the adapter does not create a second hidden CPU cache.
Slot reclamation is integrated with LMCache’s global L2 eviction controller. Recovery bootstraps existing keys and their slot-based usage into the MP accounting path.
The PR’s functional end-to-end validation used Qwen2.5-14B-Instruct with TP=2 and a raw NVMe partition. It completed all six warmup and six query requests, reported 78/78 query-round prefix hits from L2, and recovered 508 indexed keys. The measured query TTFT was not yet better than the warmup round, so the result was presented as correctness evidence rather than a tuned performance claim.
There is one important topology distinction:
• The older in-process backend can map distinct partitions to TP ranks.
• The MP Raw Block adapter is server-owned and currently uses one configured device path; it does not expose per-TP device-path mappings.
6. Making Placement KV-Aware with NVMe FDP
Raw access controls where LMCache writes. NVMe Flexible Data Placement (FDP) adds control over how the SSD groups writes with different lifetimes.
Daegyu Han, with low-level NVMe contributions from Ankit Kumar, added FDP discovery and placement plumbing in PR #4016. The path can:
• query controller-reported reclaim-unit handle status;
• validate and pass non-zero placement identifiers through Rust writes;
• keep metadata checkpoints separate from KV data placement;
• map cache_salt prefixes to placement identifiers;
• optionally separate local-rank streams within each cache-salt bucket; and
• prefer reusing free slots that previously carried the same placement identifier.
With the default cache_salt_prefix policy, values such as rag:app1 and rag:app2 share the rag placement bucket. With cache_salt_rank, local rank is added to that placement decision. The full cache_salt remains part of the logical LMCache key; FDP controls writes and is not used to locate objects on reads.
In the PR’s Samsung PM9D3a experiment, nine concurrent synthetic LMCache storage traces wrote four times a 960 GiB configured capacity at 89% device utilization. Separating workload classes with FDP reduced reported write amplification from 2.600 to 1.425, a 45.2% reduction. The same experiment reported 55.6% lower average write latency, 29.4% lower write p90, 35.7% lower average read latency, and 22.0% lower read p90. These results are specific to that device, fill level, reclaim-unit configuration, and trace.
FDP separates write lifetimes rather than accelerating I/O, so the benefit depends on device support and on whether LMCache’s logical workload classes really have different write lifetimes.
7. Reliability Work Is Part of the Feature
Fast-path work tends to get the headline, but the backend became usable through repeated hardening on real devices.
Sangyoon Kwon strengthened configuration and recovery invariants across PR #3260, PR #3700, PR #4078, and PR #4283. This work covered power-of-two alignment, aligned offsets and lengths, checkpoint compatibility, reconstruction of free slots, and correct recovered-capacity reporting. Sangyoon also hardened temporary-buffer alignment and transfer-size selection for io_uring_cmd.
Nayeon Kim focused on lifecycle and failure paths. PR #3698 rolls back references and reserved keys if async dispatch fails during event-loop shutdown. PR #3891 fixed unaligned NVMe passthrough reads, and PR #4294 turned SQE construction failures from hangs into explicit errors.
Dongjoo Seo added CI-safe coverage in PR #3203. The tests use truncated regular files to exercise the Rust binding, RawBlockCore, recovery, and MP adapter without requiring root access or destructive NVMe setup. Hardware-gated tests remain available for the behavior that a regular file cannot emulate.
The layered test strategy is:
1. regular-file tests validate allocation, metadata, lifecycle, and most error paths in normal CI;
2. opt-in Linux tests cover O_DIRECT and io_uring when the environment supports them; and
3. hardware-gated tests validate NVMe passthrough and FDP on real devices.
8. What the Performance Data Measures
The Raw Block history includes microbenchmarks, early end-to-end comparisons, and device-stress experiments. They measure different questions.
The early RFC prototype reported up to 1.595× higher TP6 populate throughput and 2.662× higher aggregate retrieval throughput for six TP1 instances than LocalDiskBackend in that test. Those numbers came from a development branch assembled from several in-progress changes.
Later TP4 validation in PR #2948 was more mixed:
| Backend | Throughput | Mean TTFT |
| Local CPU | 10.82 req/s | 723.06 ms |
| Buffered LocalDisk | 8.55 req/s | 930.88 ms |
| Rust Raw Block | 7.17 req/s | 1,110.23 ms |
| LocalDisk O_DIRECT | 1.95 req/s | 4,095.91 ms |
Raw Block substantially outperformed that O_DIRECT file configuration, but it did not beat buffered LocalDisk or DRAM in this workload. This is expected: the page cache can be effective when the host has enough memory and the working set is favorable.
The practical conclusions are:
• use the io_uring microbenchmark to study submission overhead;
• use end-to-end TTFT and throughput to determine whether the storage path is exposed to users;
• test past cache capacity so results do not accidentally measure only DRAM or the page cache;
• report device model, fill level, I/O size, queue depth, and buffer alignment; and
• treat FDP results as device-lifetime experiments, not generic I/O-engine results.
9. How to Try the MP Adapter
The current MP adapter is documented in the Raw Block storage guide. A basic io_uring configuration looks like this:
BASH
lmcache server \
–l1-size-gb 80 \
–eviction-policy LRU \
–l1-align-bytes 4096 \
–l2-adapter ‘{
“type”: “raw_block”,
“device_path”: “/dev/nvme0n1”,
“slot_bytes”: 1048576,
“block_align”: 4096,
“header_bytes”: 4096,
“meta_total_bytes”: 268435456,
“use_odirect”: true,
“io_engine”: “io_uring”,
“iouring_queue_depth”: 256,
“num_store_workers”: 2,
“num_lookup_workers”: 1,
“num_load_workers”: 4
}’
For NVMe passthrough, use the namespace character device and disable the POSIX O_DIRECT flag:
JSON
{
“type”: “raw_block”,
“device_path”: “/dev/ng0n1”,
“slot_bytes”: 1048576,
“io_engine”: “io_uring”,
“use_uring_cmd”: true,
“use_odirect”: false,
“iouring_queue_depth”: 256
}
For FDP-capable hardware, add:
JSON
{
“fdp_enabled”: true,
“fdp_data_placement_policy”: “cache_salt_rank”
}
Choose slot_bytes for the largest logical object the adapter will store, including the per-slot header. The device must be dedicated to LMCache and must not be mounted or concurrently used by another filesystem or application. Raw Block owns its configured region and may overwrite existing data.
10. Current Limits and Future Work
The current backend has several explicit limits:
• Linux-only I/O paths and Linux alignment semantics.
• Fixed-size slots, so internal fragmentation depends on the gap between payload size and slot_bytes.
• A single server-owned device path per MP adapter; no MP per-TP path map.
• The I/O path is CPU-buffer-to-storage, not NVMe-to-GPU GPUDirect Storage.
• io_uring_cmd requires a namespace character device and a kernel/device combination that supports the command path.
• FDP requires compatible NVMe hardware and enough placement identifiers for the selected policy.
• Correctness can be tested on regular files, but throughput, DMA alignment, NVMe passthrough, and FDP require real hardware.
Ankit Kumar and Preetham Jain are also developing an SPDK I/O engine in PR #4661. The proposal adds local PCIe and NVMe-over-Fabrics transports, hugepage-backed external memory registration, DMA buffer pools, polling workers, and lockless submission rings. As of this draft, that PR is under review, supports polling rather than interrupt mode, and does not yet provide MP integration. It should be treated as future work, not a released Raw Block capability.
Other likely follow-ups include multi-device striping or policy, tighter NUMA/core placement, large-I/O tuning with hugepages, more complete MP zero-copy buffer registration, and end-to-end benchmarks that isolate device bandwidth from L1 and GPU-transfer effects.
11. A Community-Built Storage Path
Raw Block is the result of a sequence of focused contributions:
• Dongjoo Seo (DongDongJu) initiated the RFC, built the initial Rust and O_DIRECT path, added aligned-buffer integration and on-device recovery, expanded TP support, extracted RawBlockCore, integrated the MP L2 adapter, and added CI-safe coverage.
• Ankit Kumar (ankit-sam) added the io_uring engine, fixed-buffer registration, batching, and NVMe io_uring_cmd; he also contributed to FDP plumbing and is leading the SPDK proposal.
• Sangyoon Kwon hardened alignment, checkpoint validation and recovery, slot accounting, passthrough temporary buffers, and automatic transfer-size selection.
• Daegyu Han (daegyu94) added NVMe FDP discovery, placement directives, cache-salt/rank policies, metadata separation, slot affinity, and hardware evaluation.
• Nayeon Kim (nayeonikim) improved core maintainability and fixed async reference leaks, unaligned passthrough reads, and SQE-error hangs.
Daejun Park and Dongjin Kim contributed to TP>1 device partitioning and isolation. LMCache maintainers and reviewers, including Samuel Shen and the MP/storage maintainers, helped shape concurrency, eviction, recovery, and integration semantics across the series.
The backend was not designed as a single large patch. It grew from a minimal correct path into a shared, recoverable MP tier, then added faster engines and media-aware placement while keeping each step reviewable.
12. Summary
LMCache Raw Block began with a simple idea: reusable KV cache already has its own keys and lifecycle, so it should be possible to store it directly on a block device without forcing every object through a filesystem abstraction.
The current implementation combines:
• a Rust raw-device binding;
• aligned O_DIRECT I/O;
• io_uring batching and optional fixed buffers;
• NVMe io_uring_cmd passthrough;
• fixed-slot allocation and on-device checkpoint recovery;
• a shared RawBlockCore used by both the original backend and MP L2 adapter;
• global L2 eviction and recovered-capacity accounting;
• FDP-based write-lifetime placement; and
• CI-safe software tests plus opt-in hardware validation.
Raw Block will not be the best tier for every workload. Buffered files can win when the page cache fits, DRAM remains faster, and direct NVMe paths demand careful alignment and device management. But when deployments want explicit storage ownership, predictable capacity, restart recovery, and a path for device-specific optimization, Raw Block gives LMCache a strong foundation.
Resources
Primary implementation, documentation, and ongoing-work references used in this draft.
• Raw Block RFC and early benchmark history
• LMCache Raw Block MP documentation
