Mathijs Henquet 78203641fe server : Add option to return token pieces in /tokenize endpoint (#9108) ... * server : added with_pieces functionality to /tokenize endpoint * server : Add tokenize with pieces tests to server.feature * Handle case if tokenizer splits along utf8 continuation bytes * Add example of token splitting * Remove trailing ws * Fix trailing ws * Maybe fix ci * maybe this fix windows ci? --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>		2024-09-12 22:30:11 +02:00
..
ISSUE_TEMPLATE	Removes multiple newlines at the end of files that is breaking the editorconfig step of CI. (#8258)	2024-07-02 12:18:10 -04:00
workflows	server : Add option to return token pieces in /tokenize endpoint (#9108)	2024-09-12 22:30:11 +02:00
labeler.yml	labeler : updated sycl to match docs and code refactor (#8373)	2024-07-08 22:35:17 +02:00
pull_request_template.md	github : update pr template	2024-06-16 10:46:51 +03:00