staviq 1a159553f9 tokenizer : special token handling (#3538) ... * Rewrite special token handling from #1931 * shorten param name, add st verification by type * use offsets instead of copy by substr * formatting, remove copying iterator on delete * llama : normalize code-style * swift fix * print pfx/sfx if verb, main: split pfx input sfx * dont add space when using special tokens * minor : comment + spacing --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>		2023-10-17 18:11:01 +03:00
..
main.swift	tokenizer : special token handling (#3538)	2023-10-17 18:11:01 +03:00