Leng Yue 35f73049af
speculative : add heuristic algorithm (#3006)
* Add heuristic algo for speculative

* Constrain minimum n_draft to 2

* speculative : improve heuristic impl

* speculative : be more rewarding upon guessing max drafted tokens

* speculative : fix typos

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-09-14 19:14:44 +03:00
..
2023-07-19 10:01:11 +03:00
2023-03-29 20:21:09 +03:00
2023-08-30 09:29:32 +03:00
2023-08-08 14:44:48 +03:00