Mirrors/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-12-23 21:18:00 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update GPTQ-models-(4-bit-mode).md

Browse Source
This commit is contained in:
oobabooga 2023-04-22 12:48:20 -03:00 committed by GitHub
parent 7438f4f6ba
commit 2c6d43e60f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 15 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
docs/GPTQ-models-(4-bit-mode).md
View File

@ -4,6 +4,18 @@ This is possible thanks to [@qwopqwop200](https://github.com/qwopqwop200/GPTQ-fo
GPTQ is a clever quantization algorithm that lightly reoptimizes the weights during quantization so that the accuracy loss is compensated relative to a round-to-nearest quantization. See the paper for more details: https://arxiv.org/abs/2210.17323
## GPTQ-for-LLaMa branches
Different branches of GPTQ-for-LLaMa are available:
| Branch | Comment |
|----|----|
| [Old CUDA branch (recommended)](https://github.com/oobabooga/GPTQ-for-LLaMa/) | The fastest branch, works on Windows and Linux. |
| [Up-to-date triton branch](https://github.com/qwopqwop200/GPTQ-for-LLaMa) | Slightly more precise than the old CUDA branch, 2x slower for small context size, only works on Linux. |
| [Up-to-date CUDA branch](https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/cuda) | As precise as the up-to-date triton branch, 10x slower than the old cuda branch for small context size. |
Overall, I recommend using the old CUDA branch. It is included by default in the one-click-installer for this web UI.
## Installation
### Step 0: install nvcc
@ -19,7 +31,7 @@ See this issue for more details: https://github.com/oobabooga/text-generation-we
### Step 1: install GPTQ-for-LLaMa
Clone the GPTQ-for-LLaMa repository into the `text-generation-webui/repositories` subfolder and install it:
* Clone the GPTQ-for-LLaMa repository into the `text-generation-webui/repositories` subfolder and install it:
```
mkdir repositories
@ -31,7 +43,7 @@ python setup_cuda.py install
You are going to need to have a C++ compiler installed into your system for the last command. On Linux, `sudo apt install build-essential` or equivalent is enough.
https://github.com/oobabooga/GPTQ-for-LLaMa corresponds to commit `a6f363e3f93b9fb5c26064b5ac7ed58d22e3f773` in the `cuda` branch of the original repository and is recommended by default for stability. Some models might require you to use the up-to-date CUDA or triton branches:
If you want to you to use the up-to-date CUDA or triton branches instead of the old CUDA branch, use these commands:
```
cd repositories
@ -57,7 +69,7 @@ https://github.com/qwopqwop200/GPTQ-for-LLaMa
* Converted without `group-size` (better for the 7b model): https://github.com/oobabooga/text-generation-webui/pull/530#issuecomment-1483891617
* Converted with `group-size` (better from 13b upwards): https://github.com/oobabooga/text-generation-webui/pull/530#issuecomment-1483941105
Note: the tokenizer files in those torrents are not up to date.
⚠️ The tokenizer files in the sources above may be outdated. Make sure to obtain the universal LLaMA tokenizer as described [here](https://github.com/oobabooga/text-generation-webui/blob/main/docs/LLaMA-model.md#option-1-pre-converted-weights).
### Step 3: Start the web UI: