4 Commits

Author	SHA1	Message	Date
Georgi Gerganov	fa0e677820	llama : extend batch API to select which logits to output	2023-09-19 00:24:13 +03:00
Georgi Gerganov	897caccdf4	fixes : speculative KV cache + llama worst-case graph	2023-09-18 22:32:28 +03:00
Georgi Gerganov	466b513851	parallel : disable hot-plug to avoid cache fragmentation	2023-09-18 21:34:20 +03:00
Georgi Gerganov	0161372b9a	parallel : example for serving multiple users in parallel	2023-09-18 20:37:28 +03:00