mirror of
https://github.com/ggerganov/llama.cpp.git
synced 2024-12-24 13:28:50 +01:00
ec893798b7
* tests : verify that RoPE is "additive" * llama : replace ggml_diag_mask_inf with ggml_add (custom -inf mask) * ggml : ggml_rope now takes a vector with positions instead of n_past * metal : add rope_f16 kernel + optimize cpy kernels * llama : unified KV cache + batch inference API * llama : add new llama_decode() API that works with llama_batch * llama : add cell_max heuristic for more efficient kv_cache * llama : extend llama_kv_cache API * llama : more robust cell_max heuristic + wip shift * metal : disable concurrency optimization * llama : add llama_kv_cache_shift_seq + no more context swaps * llama : apply K-cache roping for Falcon and Baichuan * speculative : fix KV cache management * parallel : example for serving multiple users in parallel * parallel : disable hot-plug to avoid cache fragmentation * fixes : speculative KV cache + llama worst-case graph * llama : extend batch API to select which logits to output * llama : fix worst case graph build * ggml-cuda : update rope implementation for parallel decoding (#3254) * ggml-cuda : update rope implementation for parallel decoding * better solution for p0 computation * fix rope * simpler rope implementation --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * make : add parallel to build + fix static functions in llama.cpp * simple : fix token counting * parallel : various improvements * llama : fix cell_max logic + rename functions * parallel : try smaller batches when the KV cache is fragmented * parallel : fix sequence termination criteria * llama : silence errors KV cache errors * parallel : remove new line from prompt * parallel : process system prompt once + configurable paramters + llama API * parallel : remove question with short answers * parallel : count cache misses * parallel : print misses on each request * parallel : minor * llama : fix n_kv to never become 0 * parallel : rename hot-plug to continuous-batching * llama : improve llama_batch API + simplify parallel example * simple : add parallel decoding support * simple : improve comments + free batch * ggml-cuda : add rope f16, restore performance with parallel decoding (#3272) * ggml-cuda : add rope f16, restore performance * offload KQ_mask with all models * fix rope shift --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * llama : disable MPI for now ggml-ci * train : make KQ_pos memory buffer permanent via dummy scale op * ggml : revert change to ggml_cpy, add ggml_cont_Nd instead (#3275) ggml-ci * parallel : fix bug (extra BOS) + smaller token_prev array * parallel : fix cases where the input prompts can overflow the batch * parallel : add disabled experimental batch chunking in powers of two * llama : llama.h formatting + comments * simple : add README.md * llama : fix kv cache heuristic when context is less than 32 * parallel : fix crash when `-n -1` * llama : simplify returns if/else branches * metal : use mm kernels for batch size > 2 * examples : utilize new llama_get_logits_ith() * examples : add example for batched decoding * examples : do not eval prompt 2 times (close #3348) * server : clear the KV cache beyond n_past before llama_decode * server : avoid context swaps by shifting the KV cache --------- Co-authored-by: slaren <slarengh@gmail.com> |
||
|---|---|---|
| .. | ||
| .gitignore | ||
| CMakeLists.txt | ||
| embd_input.py | ||
| embd-input-lib.cpp | ||
| embd-input-test.cpp | ||
| embd-input.h | ||
| llava.py | ||
| minigpt4.py | ||
| panda_gpt.py | ||
| README.md | ||
Examples for input embedding directly
Requirement
build libembdinput.so
run the following comman in main dir (../../).
make
LLaVA example (llava.py)
- Obtian LLaVA model (following https://github.com/haotian-liu/LLaVA/ , use https://huggingface.co/liuhaotian/LLaVA-13b-delta-v1-1/).
- Convert it to ggml format.
llava_projection.pthis pytorch_model-00003-of-00003.bin.
import torch
bin_path = "../LLaVA-13b-delta-v1-1/pytorch_model-00003-of-00003.bin"
pth_path = "./examples/embd-input/llava_projection.pth"
dic = torch.load(bin_path)
used_key = ["model.mm_projector.weight","model.mm_projector.bias"]
torch.save({k: dic[k] for k in used_key}, pth_path)
- Check the path of LLaVA model and
llava_projection.pthinllava.py.
PandaGPT example (panda_gpt.py)
- Obtian PandaGPT lora model from https://github.com/yxuansu/PandaGPT. Rename the file to
adapter_model.bin. Use convert-lora-to-ggml.py to convert it to ggml format. Theadapter_config.jsonis
{
"peft_type": "LORA",
"fan_in_fan_out": false,
"bias": null,
"modules_to_save": null,
"r": 32,
"lora_alpha": 32,
"lora_dropout": 0.1,
"target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"]
}
- Papare the
vicunav0 model. - Obtain the ImageBind model.
- Clone the PandaGPT source.
git clone https://github.com/yxuansu/PandaGPT
- Install the requirement of PandaGPT.
- Check the path of PandaGPT source, ImageBind model, lora model and vicuna model in panda_gpt.py.
MiniGPT-4 example (minigpt4.py)
- Obtain MiniGPT-4 model from https://github.com/Vision-CAIR/MiniGPT-4/ and put it in
embd-input. - Clone the MiniGPT-4 source.
git clone https://github.com/Vision-CAIR/MiniGPT-4/
- Install the requirement of PandaGPT.
- Papare the
vicunav0 model. - Check the path of MiniGPT-4 source, MiniGPT-4 model and vicuna model in
minigpt4.py.