ExternalVendorCode/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2025-06-12 20:18:11 +00:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity

Add tensor_parallel_size setting to vllm setting items (#2085)

Browse Source 
Signed-off-by: Taikono-Himazin <kazu@po.harenet.ne.jp>
This commit is contained in:
Taikono-Himazin
2024-04-20 23:37:02 +09:00
committed by GitHub
parent b319ed58b0
commit 03adc1f60d
4 changed files with 5 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
backend/python/vllm/backend_vllm.py
View File

@ -95,6 +95,8 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
engine_args.trust_remote_code = request.TrustRemoteCode
if request.EnforceEager:
engine_args.enforce_eager = request.EnforceEager
if request.TensorParallelSize:
engine_args.tensor_parallel_size = request.TensorParallelSize
if request.SwapSpace != 0:
engine_args.swap_space = request.SwapSpace
if request.MaxModelLen != 0: