Deploying LLMs in Clusters #1: running “vLLM production-stack” on a cloud VM
TL;DR [Github Link] | [More Tutorials] | [Interest Form] Tutorial Video (click below) The Context vLLM has taken the open-source community by storm, with unparalleled hardware and model support plus an active ecosystem of top-notch contributors. But until now, vLLM has mostly focused on single-node deployments. vLLM Production-stack is an open-source reference implementation of an […]
High Performance and Easy Deployment of vLLM in K8S with “vLLM production-stack”
TL;DR The Context In the AI arms race, it’s no longer just about who has the best model—it’s about who has the best LLM serving system. vLLM has taken the open-source community by storm, with unparalleled hardware and model support plus an active ecosystem of top-notch contributors. But until now, vLLM has mostly focused on […]
You focus on KV cache research, we make it compatible with vLLM
🚀 Building your system on KV cache? Try building it on LMCache! By using LMCache for your research, you can focus on KV cache management, while we handle all the vLLM integration and compatibility for you. Here’s why LMCache works as your research testbed: Check our codebase and documentations for more information! Update: we are […]
Are you a vLLM user? Change just ONE line of code to unlock 100x more KV cache storage power!
Are you a vLLM user? Unlock 100x more KV cache storage space for your multi-round conversation and document QA applications using LMCache! Just ONE line change to your code! Offline inference For offline inference, you can use LMCache within two steps: First run And then change to and now you are good to go! Like […]
LMCache: Turboboosting vLLM with 7x faster access to 100x more KV caches
TL;DR: LMCache turboboosts vLLM with 7× faster access to 100x more KV caches, for both multi-turn conversation and RAG . [💻 Source code] [📚 Paper1] [📚 Paper2] [🎬 3-minute introduction video] LLMs are ubiquitous across industries, but when using them with long documents, it takes forever for the model even to spit […]
