jaime-m-p 37bef89433 tokenizer : BPE fixes (#7530) ... * Random test: add_bos_token, add_eos_token * Random test: add BPE models for testing * Custom regex split fails with codepoint 0 * Fix falcon punctuation regex * Refactor llm_tokenizer_bpe: move code to constructor * Move 'add_special_bos/eos' logic to llm_tokenizer_bpe * Move tokenizer flags to vocab structure. * Default values for special_add_bos/eos * Build vocab.special_tokens_cache using vocab token types * Generalize 'jina-v2' per token attributes * Fix unicode whitespaces (deepseek-coder, deepseek-llm) * Skip missing byte tokens (falcon) * Better unicode data generation * Replace char32_t with uint32_t		2024-06-18 18:40:52 +02:00
..
build-info.cmake	cmake : fix issue with version info not getting baked into LlamaConfig.cmake (#3970)	2023-11-27 21:25:42 +02:00
build-info.sh	build : link against build info instead of compiling against it (#3879)	2023-11-02 08:50:16 +02:00
check-requirements.sh	Move convert.py to examples/convert-legacy-llama.py (#7430)	2024-05-30 21:40:00 +10:00
ci-run.sh	ci : add model tests + script wrapper (#4586)	2024-01-26 14:18:00 +02:00
compare-commits.sh	llama-bench : allow using a different printer for stderr with -oe (#7722)	2024-06-04 14:32:42 +02:00
compare-llama-bench.py	ggml : remove OpenCL (#7735)	2024-06-04 21:23:20 +03:00
convert-gg.sh	Move convert.py to examples/convert-legacy-llama.py (#7430)	2024-05-30 21:40:00 +10:00
debug-test.sh	Added a single test function script and fix debug-test.sh to be more robust (#7279)	2024-05-17 22:40:14 +10:00
gen-authors.sh	license : update copyright notice + add AUTHORS (#6405)	2024-04-09 09:23:19 +03:00
gen-build-info-cpp.cmake	cmake : fix issue with version info not getting baked into LlamaConfig.cmake (#3970)	2023-11-27 21:25:42 +02:00
gen-unicode-data.py	tokenizer : BPE fixes (#7530)	2024-06-18 18:40:52 +02:00
get-flags.mk	build : pass all warning flags to nvcc via -Xcompiler (#5570)	2024-02-18 16:21:52 -05:00
get-hellaswag.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
get-pg.sh	scripts : improve get-pg.sh (#4838)	2024-01-09 19:21:13 +02:00
get-wikitext-2.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
get-wikitext-103.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
get-winogrande.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
hf.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
install-oneapi.bat	support SYCL backend windows build (#5208)	2024-01-31 08:08:07 +05:30
LlamaConfig.cmake.in	ggml : remove OpenCL (#7735)	2024-06-04 21:23:20 +03:00
pod-llama.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
qnt-all.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
run-all-perf.sh	scripts : add pipefail	2023-08-29 10:50:30 +03:00
run-all-ppl.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
run-with-preset.py	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
server-llm.sh	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
sync-ggml-am.sh	ggml : remove OpenCL (#7735)	2024-06-04 21:23:20 +03:00
sync-ggml.last	ggml : sync	2024-06-18 09:50:45 +03:00
sync-ggml.sh	ggml : remove OpenCL (#7735)	2024-06-04 21:23:20 +03:00
verify-checksum-models.py	convert.py : add python logging instead of print() (#6511)	2024-05-03 22:36:41 +03:00
xxd.cmake	build: generate hex dump of server assets during build (#6661)	2024-04-21 18:48:53 +01:00