Mirrors/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-11-22 16:17:57 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update GPTQ-models-(4-bit-mode).md

Browse Source
This commit is contained in:
oobabooga 2023-05-29 15:07:59 -03:00 committed by GitHub
parent 962d05ca7e
commit 166a0d9893
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 15 additions and 21 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
docs/GPTQ-models-(4-bit-mode).md
View File

@ -1,13 +1,10 @@
In 4-bit mode, models are loaded with just 25% of their regular VRAM usage. So LLaMA-7B fits into a 6GB GPU, and LLaMA-30B fits into a 24GB GPU.
GPTQ is a clever quantization algorithm that lightly reoptimizes the weights during quantization so that the accuracy loss is compensated relative to a round-to-nearest quantization. See the paper for more details: https://arxiv.org/abs/2210.17323 GPTQ is a clever quantization algorithm that lightly reoptimizes the weights during quantization so that the accuracy loss is compensated relative to a round-to-nearest quantization. See the paper for more details: https://arxiv.org/abs/2210.17323
## AutoGPTQ ## AutoGPTQ
AutoGPTQ is the recommended way to create new quantized models: https://github.com/PanQiWei/AutoGPTQ AutoGPTQ is the recommended way to create new quantized models: https://github.com/PanQiWei/AutoGPTQ
#### Installation ### Installation
To load a model quantized with AutoGPTQ in the web UI, manual installation is currently necessary: To load a model quantized with AutoGPTQ in the web UI, manual installation is currently necessary:
@ -19,7 +16,7 @@ pip install .
You are going to need to have `nvcc` installed (see the [instructions below](https://github.com/oobabooga/text-generation-webui/blob/main/docs/GPTQ-models-(4-bit-mode).md#step-0-install-nvcc)). You are going to need to have `nvcc` installed (see the [instructions below](https://github.com/oobabooga/text-generation-webui/blob/main/docs/GPTQ-models-(4-bit-mode).md#step-0-install-nvcc)).
#### Usage ### Usage
Place the output folder generated by AutoGPTQ in your `models/` folder and load it with the `--autogptq` flag: Place the output folder generated by AutoGPTQ in your `models/` folder and load it with the `--autogptq` flag:
@ -29,9 +26,9 @@ python server.py --autogptq --model model_name
Alternatively, check the `autogptq` box in the "Model" tab of the UI before loading the model. Alternatively, check the `autogptq` box in the "Model" tab of the UI before loading the model.
#### Offloading ### Offloading
In order to do CPU offloading or multi-cpu inference with AutoGPTQ, use the `--gpu-memory` flag. It is currently somewhat slower than offloading with the `--pre_layer` option in GPTQ-for-LLaMA. In order to do CPU offloading or multi-gpu inference with AutoGPTQ, use the `--gpu-memory` flag. It is currently somewhat slower than offloading with the `--pre_layer` option in GPTQ-for-LLaMA (more on that below).
For CPU offloading: For CPU offloading:
@ -45,7 +42,7 @@ For multi-GPU:
python server.py --autogptq --gpu-memory 3000MiB 6000MiB --model model_name python server.py --autogptq --gpu-memory 3000MiB 6000MiB --model model_name
``` ```
#### Applying LoRAs ### Using LoRAs with AutoGPTQ
Not supported yet. Not supported yet.
@ -63,7 +60,11 @@ Different branches of GPTQ-for-LLaMa are currently available, including:
Overall, I recommend using the old CUDA branch. It is included by default in the one-click-installer for this web UI. Overall, I recommend using the old CUDA branch. It is included by default in the one-click-installer for this web UI.
### Installation ### Installation using precompiled wheels
https://github.com/jllllll/GPTQ-for-LLaMa-Wheels
### Manual installation
#### Step 0: install nvcc #### Step 0: install nvcc
@ -158,18 +159,11 @@ You can also use multiple GPUs with `pre_layer` if using the oobabooga fork of G
### Using LoRAs with GPTQ-for-LLaMa ### Using LoRAs with GPTQ-for-LLaMa
At the moment, this feature is not officially supported by the relevant libraries, but a patch exists and is supported by this web UI: https://github.com/johnsmith0031/alpaca_lora_4bit This requires using a monkey patch that is supported by this web UI: https://github.com/johnsmith0031/alpaca_lora_4bit
In order to use it: To use it:
1. Make sure that your requirements are up to date: 1. Clone `johnsmith0031/alpaca_lora_4bit` into the repositories folder:
```
cd text-generation-webui
pip install -r requirements.txt --upgrade
```
2. Clone `johnsmith0031/alpaca_lora_4bit` into the repositories folder:
``` ```
cd text-generation-webui/repositories cd text-generation-webui/repositories
@ -178,13 +172,13 @@ git clone https://github.com/johnsmith0031/alpaca_lora_4bit
⚠️ I have tested it with the following commit specifically: `2f704b93c961bf202937b10aac9322b092afdce0` ⚠️ I have tested it with the following commit specifically: `2f704b93c961bf202937b10aac9322b092afdce0`
3. Install https://github.com/sterlind/GPTQ-for-LLaMa with this command: 2. Install https://github.com/sterlind/GPTQ-for-LLaMa with this command:
``` ```
pip install git+https://github.com/sterlind/GPTQ-for-LLaMa.git@lora_4bit pip install git+https://github.com/sterlind/GPTQ-for-LLaMa.git@lora_4bit
``` ```
4. Start the UI with the `--monkey-patch` flag: 3. Start the UI with the `--monkey-patch` flag:
``` ```
python server.py --model llama-7b-4bit-128g --listen --lora tloen_alpaca-lora-7b --monkey-patch python server.py --model llama-7b-4bit-128g --listen --lora tloen_alpaca-lora-7b --monkey-patch