ExternalVendorCode/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2025-06-12 20:18:11 +00:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity

Bump vLLM version + more options when loading models in vLLM (#1782)

Browse Source 
* Bump vLLM version to 0.3.2

* Add vLLM model loading options

* Remove transformers-exllama

* Fix install exllama
This commit is contained in:
Ludovic Leroux
2024-03-01 16:48:53 -05:00
committed by GitHub
parent 1c312685aa
commit 939411300a
28 changed files with 736 additions and 641 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
backend/backend.proto
View File

@ -126,6 +126,11 @@ message ModelOptions {
// vllm
string Quantization = 40;
float GPUMemoryUtilization = 50;
bool TrustRemoteCode = 51;
bool EnforceEager = 52;
int32 SwapSpace = 53;
int32 MaxModelLen = 54;
string MMProj = 41;