b1385e9aa9
CUDA: correct the lowest Maxwell supported by CUDA 12 (llama/11984)
...
* CUDA: correct the lowest Maxwell supported by CUDA 12
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2025-02-27 08:55:36 +02:00
48f5e893f5
MUSA: support ARM64 and enable dp4a .etc (llama/11843)
...
* MUSA: support ARM64 and enable __dp4a .etc
* fix cross entropy loss op for musa
* update
* add cc info log for musa
* add comment for the MUSA .cc calculation block
---------
Co-authored-by: Bodhi Hu <huaishun.hu@mthreads.com >
2025-02-27 08:55:36 +02:00
51a3580c79
CUDA: use async data loading for FlashAttention (llama/11894)
...
* CUDA: use async data loading for FlashAttention
---------
Co-authored-by: Diego Devesa <slarengh@gmail.com >
2025-02-27 08:55:36 +02:00
47cc043e69
cuda : add ampere to the list of default architectures (llama/11870)
2025-02-27 08:55:36 +02:00
4e07957bf9
musa: bump MUSA SDK version to rc3.1.1 (llama/11822)
...
* musa: Update MUSA SDK version to rc3.1.1
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* musa: Remove workaround in PR #10042
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
---------
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2025-02-27 08:55:36 +02:00
3be9670f17
HIP: Remove GCN from list of devices that avoid MMQ (llama/11831)
2025-02-27 08:55:36 +02:00
86729fcd6d
HIP: Switch to std::vector in rocblas version check (llama/11820)
2025-02-27 08:55:36 +02:00
556f773d53
CUDA: fix CUDART_VERSION checks (llama/11821)
2025-02-27 08:55:36 +02:00
1b67d72f87
CUDA: use arch list for compatibility check (llama/11775)
...
* CUDA: use arch list for feature availability check
---------
Co-authored-by: Diego Devesa <slarengh@gmail.com >
2025-02-27 08:55:36 +02:00
7561da244e
CUDA: fix min. version for movmatrix (llama/11751)
2025-02-27 08:55:36 +02:00
01c9aafbfd
CUDA: support for mat. mul. with ne03 != ne13 (llama/11656)
2025-02-27 08:55:36 +02:00
bae6bbf487
CUDA: non-contiguous (RMS) norm support (llama/11659)
...
* CUDA: non-contiguous (RMS) norm support
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-02-27 08:55:36 +02:00
dbeb7916b8
CUDA: fix Volta FlashAttention logic (llama/11615)
2025-02-03 22:00:57 +02:00
fad2806352
HIP: fix flash_attn_stream_k_fixup warning (llama/11604)
2025-02-03 22:00:57 +02:00
9906792ec3
CUDA/HIP: add support for selectable warp size to mmv (llama/11519)
...
CUDA/HIP: add support for selectable warp size to mmv
2025-02-03 22:00:57 +02:00
c49ee07ff4
HIP: add GGML_CUDA_CC_IS_* for amd familys as increasing cc archtectures for amd gpus are not supersets of eatch other (llama/11601)
...
This fixes a bug where RDNA1 gpus other than gfx1010 where not handled correctly
2025-02-03 22:00:57 +02:00
f8a831779e
CUDA: use mma PTX instructions for FlashAttention (llama/11583)
...
* CUDA: use mma PTX instructions for FlashAttention
* __shfl_sync workaround for movmatrix
* add __shfl_sync to HIP
Co-authored-by: Diego Devesa <slarengh@gmail.com >
2025-02-03 22:00:57 +02:00
fc2e44490d
HIP: Prepare reduction operators for wave 64
2025-02-03 22:00:57 +02:00
f41fdad200
CUDA/HIP: add warp_size to cuda_device_info
2025-02-03 22:00:57 +02:00
682a6f5f87
HIP: Supress transformation warning in softmax.cu
...
loops with bounds not known at compile time can not be unrolled.
when ncols_template == 0, the bounds of the loop are not constexpr, thus llvm cant unroll the loops here.
2025-02-03 22:00:57 +02:00
115716d109
HIP: Only call rocblas_initialize on rocblas versions with the multiple instantation bug (llama/11080)
...
This disables the workaround on rocblas fixed versions (>=4.0.0) to eliminate the runtime cost and unnecessary VRAM allocation of loading all tensile objects.
2025-02-03 22:00:57 +02:00
028511d349
AMD: parse the architecture as supplied by gcnArchName (llama/11244)
...
The value provided by minor doesn't include stepping for AMD, parse the value returned by gcnArchName instead to retrieve an accurate ID.
2025-02-03 22:00:57 +02:00
a160fa0f3a
Hip: disable VMM on hip as it seams that it dosent work in some configurations (llama/11420)
2025-02-03 22:00:57 +02:00
0282ad8fd1
hip : Add hipGraph and VMM support to ROCM (llama/11362)
...
* Add hipGraph support
* Enable VMM on rocm
2025-02-03 22:00:57 +02:00
9e467815d4
CUDA: fix FP16 cuBLAS GEMM (llama/11396)
2025-02-03 22:00:57 +02:00
727891d9bf
rocBLAS: Avoid fp32->fp16->fp32 conversion on cdna (llama/11356)
2025-02-03 22:00:57 +02:00
c262dc80e2
CPU/CUDA: fix (GQA) mul mat back, add CUDA support (llama/11380)
2025-02-03 22:00:57 +02:00
de49024e49
CUDA: backwards pass for misc. ops, add tests (llama/11257)
...
* CUDA: backwards pass for misc. ops, add tests
* remove restrict from pointers
2025-02-03 22:00:57 +02:00
54a2ee648f
RoPE: fix back, CUDA support for back + noncont. (llama/11240)
...
* RoPE: fix back, CUDA support for back + noncont.
* fix comments reg. non-cont. RoPE support [no-ci]
2025-02-03 22:00:57 +02:00
2425caf4fd
cuda : CUDA Graph Compute Function Refactor (precursor for performance improvements) (llama/11042)
...
* Refactor: Moves cuda graph executable update step to separate function.
* Refactor: Moves cuda graph update check to separate function.
* Refactor: Moves cuda graph maintenance (update or adjusting copy parameters) to separate function for improved readability.
* Fix: Adds missing reference to maintain_cuda_graph() definition.
* Refactor: Improves structure and abstractions by moving CUDA graph evaluation and capture to its own function.
* Refactor: Moves node graph checks and copy ops into individual function for improved readability.
* Refactor: Removes code permanently excluded from compilation to increase readability.
* Style: Adds missing newline
* Style: Consolidates several neighboring '#ifdef USE_CUDA_GRAPH' into a single one
* Refactor: Makes 'cuda_graph_update_required' a local variable
* remove double lines between functions
---------
Co-authored-by: slaren <slarengh@gmail.com >
2025-01-14 10:38:01 +02:00
06209f6683
llama: add support for QRWKV6 model architecture (llama/11001)
...
llama: add support for QRWKV6 model architecture (llama/11001)
* WIP: Add support for RWKV6Qwen2
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* RWKV: Some graph simplification
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* Add support for RWKV6Qwen2 with cpu and cuda GLA
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* RWKV6[QWEN2]: Concat lerp weights together to reduce cpu overhead
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* Fix some typos
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* code format changes
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* Fix wkv test & add gla test
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* Fix cuda warning
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* Update README.md
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* Update ggml/src/ggml-cuda/gla.cu
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* Fix fused lerp weights loading with RWKV6
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
* better sanity check skipping for QRWKV6 in llama-quant
thanks @compilade
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
Co-authored-by: compilade <git@compilade.net >
---------
Signed-off-by: Molly Sophia <mollysophia379@gmail.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
Co-authored-by: compilade <git@compilade.net >
2025-01-14 10:38:01 +02:00
262d0abc87
fix: add missing msg in static_assert (llama/11143)
...
Signed-off-by: hydai <z54981220@gmail.com >
2025-01-14 10:38:01 +02:00
341f5c28e6
CUDA: add BF16 support (llama/11093)
...
* CUDA: add BF16 support
2025-01-14 10:38:01 +02:00
6576af00d7
files : remove old sources
2024-12-18 12:52:16 +02:00
e22d38e4f2
llama : add Qwen2VL support + multimodal RoPE (llama/10361)
...
* Barebone Qwen2VL LLM convertor
* Add Qwen2VL cli entrypoint
* [WIP] add qwen2vl arch
* Verify m-rope output
* Add vl-rope/2d-rope support for qwen2vl ViT
* update qwen2vl cli tool
* update 5D tensor op workaround
* [WIP] qwen2vl vision model
* make batch and clip utils compatible with qwen2vl
* [WIP] create inference workflow, gguf convert script but fix
* correcting vision-rope behavior, add the missing last layer back to ViT
* add arg parser to qwen2vl_surgery
* replace variable size array with vector
* cuda-gdb cmake preset
* add fp32 mrope, vision rope kernel
* add fp16 support for qwen2vl and m-rope
* add `GGML_ROPE_TYPE_MROPE`, `GGML_ROPE_TYPE_VISION`
* fix rope op mode switching, out dated func args
* update `llama_hparams`
* update to keep up stream changes
* resolve linter, test errors
* add makefile entry, update speical image padding token
* add mrope unit test, fix few compiler warnings
* rename `mrope` related function, params
* minor updates on debug util, bug fixs
* add `m-rope` testcase to `test-backend-ops`
* Apply suggestions from code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix traililng whitespce
* store `llama_hparams.rope_sections` with fixed size array
* update position id tensor size check in GGML_OP_ROPE
* minor updates
* update `ggml_backend_*_supports_op` of unsupported backends
* remote old `rope_section` compare operator
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-12-18 12:52:16 +02:00
abe3102cb7
CUDA: faster non-contiguous concat (llama/10760)
...
* faster uncontiguous concat
* Use a lambda to avoid code duplication
Co-authored-by: Diego Devesa <slarengh@gmail.com >
* Update ggml/src/ggml-cuda/concat.cu
* add constexpr and static assert
---------
Co-authored-by: Diego Devesa <slarengh@gmail.com >
2024-12-18 12:52:16 +02:00
b82c8d76dc
CUDA: rename macros to avoid conflicts with WinAPI (llama/10736)
...
* Renames NVIDIA GPU-architecture flags to avoid name clashes with WinAPI. (e.g. CC_PASCAL, GPU architecture or WinAPI pascal compiler flag?)
* Reverts erroneous rename in SYCL-code.
* Renames GGML_CUDA_MIN_CC_DP4A to GGML_CUDA_CC_DP4A.
* Renames the rest of the compute capability macros for consistency.
2024-12-18 12:52:16 +02:00
eb27e0d834
CUDA: fix shared memory access condition for mmv (llama/10740)
2024-12-18 12:52:16 +02:00
e990d1b791
ggml : refactor online repacking (llama/10446)
...
* rename ggml-cpu-aarch64.c to .cpp
* reformat extra cpu backend.
- clean Q4_0_N_M and IQ4_0_N_M
- remove from "file" tensor type
- allow only with dynamic repack
- extract cpu extra bufts and convert to C++
- hbm
- "aarch64"
- more generic use of extra buffer
- generalise extra_supports_op
- new API for "cpu-accel":
- amx
- aarch64
* clang-format
* Clean Q4_0_N_M ref
Enable restrict on C++
* add op GGML_OP_MUL_MAT_ID for Q4_0_N_M with runtime repack
* added/corrected control on tensor size for Q4 repacking.
* Update ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* Update ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* add debug logs on repacks.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-12-18 12:52:16 +02:00
7d134e3737
ggml : remove old files (skip) ( #0 )
2024-12-08 23:04:26 +02:00
9df53b357e
ggml : sync remnants (skip) ( #0 )
2024-12-08 22:48:25 +02:00
230e985633
Add some minimal optimizations for CDNA (llama/10498)
...
* Add some minimal optimizations for CDNA
* ggml_cuda: set launch bounds also for GCN as it helps there too
2024-12-08 20:14:35 +02:00
48f421de23
cmake : enable warnings in llama (llama/10474)
...
* cmake : enable warnings in llama
ggml-ci
* cmake : add llama_get_flags and respect LLAMA_FATAL_WARNINGS
* cmake : get_flags -> ggml_get_flags
* speculative-simple : fix warnings
* cmake : reuse ggml_get_flags
ggml-ci
* speculative-simple : fix compile warning
ggml-ci
2024-12-08 20:14:35 +02:00
77e3e4a090
ggml : add support for dynamic loading of backends (llama/10469)
...
* ggml : add support for dynamic loading of backends
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-12-08 20:14:35 +02:00
2a4b5c9d7e
cuda : optimize argmax (llama/10441)
...
* cuda : optimize argmax
* remove unused parameter
ggml-ci
* fixup : use full warps
ggml-ci
* Apply suggestions from code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
* fix ub
* ggml : check ne00 <= INT32_MAX in argmax and argsort
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2024-12-08 20:14:35 +02:00
4af9626702
CUDA: remove unnecessary warp reduce in FA (ggml/1032)
...
* kqmax_new_j in every thread within warp is same after operate at line 199,this reduce can be omit
* same problem in vec32
---------
Co-authored-by: ZhaoXiaoYu <zhao.xiaoyu@zte.com.cn >
2024-12-08 20:14:35 +02:00
f4c1d7df39
ggml : sync resolve (skip) ( #0 )
2024-11-20 21:00:08 +02:00
5f6d6919b4
cuda : fix CUDA_FLAGS not being applied (llama/10403)
2024-11-20 21:00:08 +02:00
7ac2f17fac
cuda : only use native when supported by cmake (llama/10389)
2024-11-20 21:00:08 +02:00
161b443514
CUDA: fix MMV kernel being used for FP16 src1 (llama/10357)
2024-11-20 21:00:08 +02:00
