Mirrors
/
hf_text-generation-inference
mirror of https://github.com/huggingface/text-generation-inference.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
hf_text-generation-inference/server/Makefile-flash-att

16 lines
743 B
Plaintext
Raw Normal View History

feat(server): use latest flash attention commit (#543) @njhill FYI
2023-07-04 12:23:55 -06:00
flash_att_commit := 3a9bfd076f98746c73362328958dbc68d145fbec
feat(docker): improve flash_attention caching (#160)
2023-04-09 11:59:16 -06:00
fea(dockerfile): better layer caching (#159)
2023-04-14 02:12:21 -06:00
flash-attention:
# Clone flash attention
Add RoCm support (#1243) This PR adds support for AMD Instinct MI210 & MI250 GPUs, with paged attention and FAv2 support. Remaining items to discuss, on top of possible others: * Should we have a `ghcr.io/huggingface/text-generation-inference:1.1.0+rocm` hosted image, or is it too early? * Should we set up a CI on MI210/MI250? I don't have access to the runners of TGI though. * Are we comfortable with those changes being directly in TGI, or do we need a fork? --------- Co-authored-by: Felix Marty <felix@hf.co> Co-authored-by: OlivierDehaene <olivier@huggingface.co> Co-authored-by: Your Name <you@example.com>
2023-11-27 06:08:12 -07:00
pip install -U packaging ninja --no-cache-dir
feat(server): use latest flash attention commit (#543) @njhill FYI
2023-07-04 12:23:55 -06:00
git clone https://github.com/HazyResearch/flash-attention.git
fea(dockerfile): better layer caching (#159)
2023-04-14 02:12:21 -06:00
build-flash-attention: flash-attention
cd flash-attention && git fetch && git checkout $(flash_att_commit)
cd flash-attention && python setup.py build
cd flash-attention/csrc/rotary && python setup.py build
cd flash-attention/csrc/layer_norm && python setup.py build
install-flash-attention: build-flash-attention
pip uninstall flash_attn rotary_emb dropout_layer_norm -y || true
feat(docker): improve flash_attention caching (#160)
2023-04-09 11:59:16 -06:00
cd flash-attention && python setup.py install && cd csrc/layer_norm && python setup.py install && cd ../rotary && python setup.py install