Mirrors/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-12-04 21:10:30 +01:00
Code Issues Packages Projects Releases Wiki Activity

Created llama.cpp models (markdown)

oobabooga 2023-03-31 15:15:31 -03:00
parent dc4b59d0ff
commit c4238755a5
1 changed files with 21 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
llama.cpp-models.md Normal file

@ -0,0 +1,21 @@
Available after [#447](https://github.com/oobabooga/text-generation-webui/pull/447) thanks to [@thomasantony](https://github.com/thomasantony) and his [llamacpp-python](https://github.com/thomasantony/llamacpp-python) library.
## Using llama.cpp in the web UI
1. Re-install the requirements to get `llamacpp` intalled:
```
pip install -r requirements.txt --upgrade
```
2. Follow the instructions in the llama.cpp README to generate the `ggml-model-q4_0.bin` file: https://github.com/ggerganov/llama.cpp#usage
3. Create a folder with name starting in `llamacpp` inside `models/`. For instance, `models/llamacpp-7b`
4. Place `ggml-model-q4_0.bin` inside that folder.
5. Start the web UI normally:
```
python server.py --model llamacpp-7b
```