Bump vLLM version + more options when loading models in vLLM (#1782)

* Bump vLLM version to 0.3.2 * Add vLLM model loading options * Remove transformers-exllama * Fix install exllama
2025-06-12 20:18:11 +00:00 · 2024-03-01 16:48:53 -05:00
parent 1c312685aa
commit 939411300a
28 changed files with 736 additions and 641 deletions
--- a/backend/python/autogptq/autogptq.yml
+++ b/backend/python/autogptq/autogptq.yml
@ -71,7 +71,7 @@ dependencies:
      - regex==2023.10.3
      - requests==2.31.0
      - rouge==1.0.1
-      - safetensors==0.3.3
+      - safetensors>=0.3.3
      - six==1.16.0
      - sympy==1.12
      - tokenizers==0.14.0
--- a/backend/python/autogptq/backend_pb2.py
+++ b/backend/python/autogptq/backend_pb2.py