1
0
mirror of https://github.com/ggerganov/llama.cpp.git synced 2025-01-19 08:20:10 +01:00
Commit Graph

1 Commits

Author SHA1 Message Date
Ren Xuancheng
229ffff872
llama : add BPE pre-tokenization for Qwen2 ()
* Add BPE pre-tokenization for Qwen2.

* minor : fixes

---------

Co-authored-by: Ren Xuancheng <17811943+jklj077@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2024-05-08 15:06:43 +03:00