agray3 13dca2a54a

Vectorize load instructions in dmmv f16 CUDA kernel (#9816) ...

* Vectorize load instructions in dmmv f16 CUDA kernel

Replaces scalar with vector load instructions, which substantially
improves performance on NVIDIA HBM GPUs, e.g. gives a 1.27X overall
speedup for Meta-Llama-3-8B-Instruct-F16 BS1 inference evaluation on
H100 SXM 80GB HBM3. On GDDR GPUs, there is a slight (1.01X) speedup.

* addressed comment

* Update ggml/src/ggml-cuda/dmmv.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

2024-10-14 02:49:08 +02:00

..

cmake

llama : reorganize source code + improve CMake (#8006)

2024-06-26 18:33:02 +03:00

include

rpc : add backend registry / device interfaces (#9812)

2024-10-10 20:14:55 +02:00

src

Vectorize load instructions in dmmv f16 CUDA kernel (#9816)

2024-10-14 02:49:08 +02:00

.gitignore

vulkan : cmake integration (#8119)

2024-07-13 18:12:39 +02:00

CMakeLists.txt

cmake : do not hide GGML options + rename option (#9465)

2024-09-16 10:27:50 +03:00