This is the Linux app named KVCache-Factory whose latest release can be downloaded as KVCache-Factorysourcecode.tar.gz. It can be run online in the free hosting provider OnWorks for workstations.
Download and run online this app named KVCache-Factory with OnWorks for free.
Follow these instructions in order to run this app:
- 1. Downloaded this application in your PC.
- 2. Enter in our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.
- 3. Upload this application in such filemanager.
- 4. Start the OnWorks Linux online or Windows online emulator or MACOS online emulator from this website.
- 5. From the OnWorks Linux OS you have just started, goto our file manager https://www.onworks.net/myfiles.php?username=XXXXX with the username that you want.
- 6. Download the application, install it and run it.
SCREENSHOTS:
KVCache-Factory
DESCRIPTION:
KVCache-Factory is an open-source research framework designed to explore and implement unified key-value cache compression techniques for autoregressive transformer models. In large language models, the key-value cache stores intermediate attention states that enable efficient token generation during inference, but these caches can consume large amounts of GPU memory when handling long contexts. KVCache-Factory provides a platform for implementing and evaluating multiple compression strategies that reduce memory usage while preserving model performance. The framework integrates several state-of-the-art methods such as PyramidKV, SnapKV, H2O, and StreamingLLM, allowing researchers to compare and experiment with different approaches within the same environment. It also supports advanced inference configurations such as Flash Attention v2 and multi-GPU inference setups for very large models.
Features
- Unified framework for experimenting with multiple KV-cache compression methods
- Support for algorithms such as PyramidKV, SnapKV, H2O, and StreamingLLM
- Integration with modern attention implementations including Flash Attention v2
- Multi-GPU inference support for large transformer models
- Benchmarking tools for evaluating long-context performance and memory usage
- Visualization utilities for analyzing attention patterns and cache behavior
Programming Language
Python
Categories
This is an application that can also be fetched from https://sourceforge.net/projects/kvcache-factory.mirror/. It has been hosted in OnWorks in order to be run online in an easiest way from one of our free Operative Systems.