Mirrors/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-11-22 16:17:57 +01:00
Code Issues Packages Projects Releases Wiki Activity

Add AutoGPTQ wheels to requirements.txt

Browse Source
This commit is contained in:
oobabooga 2023-06-02 00:47:11 -03:00
parent f344ccdddb
commit d0aca83b53
2 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/GPTQ-models-(4-bit-mode).md
View File

@ -18,7 +18,7 @@ There are two ways of loading GPTQ models in the web UI at the moment:
* supports more models * supports more models
* standardized (no need to guess any parameter) * standardized (no need to guess any parameter)
* is a proper Python library * is a proper Python library
* no wheels are presently available so it requires manual compilation * ~no wheels are presently available so it requires manual compilation~
* supports loading both triton and cuda models * supports loading both triton and cuda models
For creating new quantizations, I recommend using AutoGPTQ: https://github.com/PanQiWei/AutoGPTQ For creating new quantizations, I recommend using AutoGPTQ: https://github.com/PanQiWei/AutoGPTQ
@ -175,7 +175,7 @@ python server.py --model llama-7b-4bit-128g --listen --lora tloen_alpaca-lora-7b
### Installation ### Installation
To load a model quantized with AutoGPTQ in the web UI, you need to first manually install the AutoGPTQ library: No additional steps are necessary as AutoGPTQ is already in the `requirements.txt` for the webui. If you still want or need to install it manually for whatever reason, these are the commands:
``` ```
conda activate textgen conda activate textgen

2
requirements.txt
View File

@ -21,3 +21,5 @@ bitsandbytes==0.39.0; platform_system != "Windows"
https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.39.0-py3-none-any.whl; platform_system == "Windows" https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.39.0-py3-none-any.whl; platform_system == "Windows"
llama-cpp-python==0.1.56; platform_system != "Windows" llama-cpp-python==0.1.56; platform_system != "Windows"
https://github.com/abetlen/llama-cpp-python/releases/download/v0.1.56/llama_cpp_python-0.1.56-cp310-cp310-win_amd64.whl; platform_system == "Windows" https://github.com/abetlen/llama-cpp-python/releases/download/v0.1.56/llama_cpp_python-0.1.56-cp310-cp310-win_amd64.whl; platform_system == "Windows"
https://github.com/PanQiWei/AutoGPTQ/releases/download/v0.2.0/auto_gptq-0.2.0+cu118-cp310-cp310-win_amd64.whl; platform_system == "Windows"
https://github.com/PanQiWei/AutoGPTQ/releases/download/v0.2.0/auto_gptq-0.2.0+cu118-cp310-cp310-linux_x86_64.whl; platform_system != "Windows"