From 74bf2f05b1e71941600a773d6f97ddc5497f6d25 Mon Sep 17 00:00:00 2001 From: oobabooga <112222186+oobabooga@users.noreply.github.com> Date: Thu, 1 Jun 2023 11:58:33 -0300 Subject: [PATCH] Update llama.cpp-models.md --- docs/llama.cpp-models.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/llama.cpp-models.md b/docs/llama.cpp-models.md index be368fd3..ae6b1b9e 100644 --- a/docs/llama.cpp-models.md +++ b/docs/llama.cpp-models.md @@ -12,7 +12,10 @@ Follow the instructions in the llama.cpp README to generate the `ggml-model.bin` ## GPU offloading -Enabled with the `--n-gpu-layers` parameter. If you have enough VRAM, use a high number like `--n-gpu-layers 200000` to offload all layers to the GPU. Otherwise, start with a low value like `--n-gpu-layers 10` and gradually increase it until you run out of memory. +Enabled with the `--n-gpu-layers` parameter. + +* If you have enough VRAM, use a high number like `--n-gpu-layers 200000` to offload all layers to the GPU. +* Otherwise, start with a low number like `--n-gpu-layers 10` and then gradually increase it until you run out of memory. To use this feature, you need to manually compile and install `llama-cpp-python` with GPU support.