This is the Linux app named LMCache whose latest release can be downloaded as lmcache-0.3.7.tar.gz. It can be run online in the free hosting provider OnWorks for workstations.
Download and run online this app named LMCache with OnWorks for free.
请按照以下说明运行此应用程序：
- 1. 在您的 PC 中下载此应用程序。
- 2. 在我们的文件管理器 https://www.onworks.net/myfiles.php?username=XXXXX 中输入您想要的用户名。
- 3. 在这样的文件管理器中上传这个应用程序。
- 4. 从此网站启动OnWorks Linux online 或Windows online emulator 或MACOS online emulator。
- 5. 从您刚刚启动的 OnWorks Linux 操作系统，使用您想要的用户名转到我们的文件管理器 https://www.onworks.net/myfiles.php?username=XXXXX。
- 6. 下载应用程序，安装并运行。
SCREENSHOTS
Ad
LM缓存
商品描述
LMCache is an extension layer for LLM serving engines that accelerates inference, especially with long contexts, by storing and reusing key-value (KV) attention caches across requests. Instead of rebuilding KV states for repeated or shared text segments, LMCache persists and retrieves them from multiple tiers—GPU memory, CPU DRAM, and local disk—then injects them into subsequent requests to reduce TTFT and increase throughput. Its design supports reuse beyond strict prefix matching and enables sharing across serving instances, improving efficiency under real multi-tenant traffic. The broader project includes examples, tests, a server component, and public posts describing cross-engine sharing and inter-GPU KV transfers. These capabilities aim to lower latency, cut GPU cycles, and stabilize performance for production workloads with overlapping prompts or retrieval-augmented contexts. The end result is a cache fabric for LLMs that complements engines rather than replacing them.
功能
- KV cache reuse across queries to cut TTFT and boost throughput
- Multi-tier storage across GPU, CPU, and local disk
- Cross-engine sharing for interoperability between serving stacks
- Non-prefix reuse to exploit any overlapping text segments
- Inter-GPU KV transfer for advanced pipeline/disaggregated setups
- Example suites, tests, and a server to ease integration
程式语言
Python
分类
This is an application that can also be fetched from https://sourceforge.net/projects/lmcache.mirror/. It has been hosted in OnWorks in order to be run online in an easiest way from one of our free Operative Systems.
