2024-07-08 20:04:06 +00:00
<!DOCTYPE html>
< html lang = "en" >
{{template "views/partials/head" .}}
< body class = "bg-gray-900 text-gray-200" >
< div class = "flex flex-col min-h-screen" >
{{template "views/partials/navbar" .}}
< div class = "container mx-auto px-4 flex-grow" >
< div class = "workers mt-12 text-center" >
2024-08-19 16:05:02 +00:00
< div class = "animation-container" >
< canvas id = "networkCanvas" > < / canvas >
< div class = "text-overlay" >
< header class = "text-center py-12" >
< h1 class = "text-5xl font-bold text-gray-100" >
< i class = "fa-solid fa-circle-nodes mr-2" > < / i > Distributed inference with P2P
< / h1 >
< p class = "mt-4 text-lg" >
Distribute computation by sharing and loadbalancing instances or sharding model weights.
< a href = "https://localai.io/features/distribute/" target = "_blank" >
< i class = "fas fa-circle-info pr-2" > < / i >
< / a >
< / p >
< / header >
< / div >
< / div >
2024-07-08 20:04:06 +00:00
< h5 class = "mb-4 text-justify" > LocalAI uses P2P technologies to enable distribution of work between peers. It is possible to share an instance with Federation and/or split the weights of a model across peers (only available with llama.cpp models). You can now share computational resources between your devices or your friends!< / h5 >
2024-07-23 08:42:51 +00:00
<!-- Warning box if p2p token is empty and p2p is enabled -->
{{ if and .IsP2PEnabled (eq .P2PToken "") }}
< div class = "bg-red-500 p-4 rounded-lg shadow-lg mb-12 text-left" >
< p class = "text-xl font-semibold text-white" > < i class = "fa-solid fa-exclamation-triangle" > < / i > Warning: P2P mode is disabled or no token was specified< / p >
< p class = "mb-4" > You have to enable P2P mode by starting LocalAI with < code > --p2p< / code > . Please restart the server with < code > --p2p< / code > to generate a new token automatically that can be used to automatically discover other nodes. If you already have a token specify it with < code > export TOKEN=".."< / code > < a href = "https://localai.io/features/distribute/" target = "_blank" >
Check out the documentation for more information.
< / a > < / p >
< / div >
{{ else }}
2024-07-08 20:04:06 +00:00
<!-- Federation Box -->
< div class = "bg-gray-800 p-6 rounded-lg shadow-lg mb-12 text-left" >
< p class = "text-xl font-semibold text-gray-200" > < i class = "text-gray-200 fa-solid fa-circle-nodes" > < / i > Federated Nodes: < span hx-get = "/p2p/ui/workers-federation-stats" hx-trigger = "every 1s" > < / span > < / p >
< p class = "mb-4" > You can start LocalAI in federated mode to share your instance, or start the federated server to balance requests between nodes of the federation.< / p >
< div class = "grid grid-cols-1 sm:grid-cols-2 md:grid-cols-3 gap-4 mb-12" >
< div hx-get = "/p2p/ui/workers-federation" hx-trigger = "every 1s" > < / div >
< / div >
< hr class = "border-gray-700 mb-12" >
< h3 class = "text-2xl font-semibold text-gray-100 mb-6" > < i class = "fa-solid fa-book" > < / i > Start a federated instance< / h3 >
<!-- Tabs navigation -->
< ul class = "mb-5 flex list-none flex-row flex-wrap ps-0" role = "tablist" data-twe-nav-ref >
< li role = "presentation" class = "flex-auto text-center" >
< a href = "#tabs-federated-cli" class = "tablink my-2 block border-0 bg-gray-800 px-7 pb-3.5 pt-4 text-xs font-medium uppercase leading-tight text-white hover:bg-gray-700 focus:bg-gray-700 data-[twe-nav-active]:border-yellow-500 data-[twe-nav-active]:text-yellow-500 data-[twe-nav-active]:bg-gray-700 active" data-twe-toggle = "pill" data-twe-target = "#tabs-federated-cli" data-twe-nav-active role = "tab" aria-controls = "tabs-federated-cli" aria-selected = "true" > < i class = "fa-solid fa-terminal" > < / i > CLI< / a >
< / li >
< li role = "presentation" class = "flex-auto text-center" >
< a href = "#tabs-federated-docker" class = "tablink my-2 block border-0 bg-gray-800 px-7 pb-3.5 pt-4 text-xs font-medium uppercase leading-tight text-white hover:bg-gray-700 focus:bg-gray-700 data-[twe-nav-active]:border-yellow-500 data-[twe-nav-active]:text-yellow-500 data-[twe-nav-active]:bg-gray-700" data-twe-toggle = "pill" data-twe-target = "#tabs-federated-docker" role = "tab" aria-controls = "tabs-federated-docker" aria-selected = "false" > < i class = "fa-solid fa-box-open" > < / i > Container images< / a >
< / li >
< / ul >
<!-- Tabs content -->
< div class = "mb-6" >
< div class = "tabcontent hidden opacity-100 transition-opacity duration-150 ease-linear data-[twe-tab-active]:block p-4" id = "tabs-federated-cli" role = "tabpanel" aria-labelledby = "tabs-federated-cli" data-twe-tab-active >
< p class = "mb-2" > To start a new instance to share:< / p >
< code class = "block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words" >
# Start a new instance to share with --federated and a TOKEN< br >
export TOKEN="< span class = "token" > {{.P2PToken}}< / span > "< br >
local-ai run --federated --p2p
< / code >
< p class = "mt-2" > Note: If you don't have a token do not specify it and use the generated one that you can find in this page.< / p >
< p class = "mb-2" > To start a new federated load balancer:< / p >
< code class = "block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words" >
export TOKEN="< span class = "token" > {{.P2PToken}}< / span > "< br >
local-ai federated
< / code >
< p class = "mt-2" > Note: Token is needed when starting the federated server.< / p >
< p class = "mt-2" > For all the options available, please refer to the < a href = "https://localai.io/features/distribute/#starting-workers" target = "_blank" class = "text-yellow-300 hover:text-yellow-400" > documentation< / a > .< / p >
< / div >
< div class = "tabcontent hidden opacity-0 transition-opacity duration-150 ease-linear data-[twe-tab-active]:block p-4" id = "tabs-federated-docker" role = "tabpanel" aria-labelledby = "tabs-federated-docker" >
< p class = "mb-2" > To start a new federated instance:< / p >
< code class = "block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words" >
docker run -ti --net host -e TOKEN="< span class = "token" > {{.P2PToken}}< / span > " --name local-ai -p 8080:8080 localai/localai:latest-cpu run --federated --p2p
< / code >
< p class = "mb-2" > To start a new federated server (port to 9090):< / p >
< code class = "block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words" >
docker run -ti --net host -e TOKEN="< span class = "token" > {{.P2PToken}}< / span > " --name local-ai -p 9090:8080 localai/localai:latest-cpu federated
< / code >
< p class = "mt-2" > For all the options available and see what image to use, please refer to the < a href = "https://localai.io/basics/container/" target = "_blank" class = "text-yellow-300 hover:text-yellow-400" > Container images documentation< / a > and < a href = "https://localai.io/advanced/#cli-parameters" target = "_blank" class = "text-yellow-300 hover:text-yellow-400" > CLI parameters documentation< / a > .< / p >
< / div >
< / div >
< / div >
<!-- Llama.cpp Box -->
< div class = "bg-gray-800 p-6 rounded-lg shadow-lg mb-12 text-left" >
< p class = "text-xl font-semibold text-gray-200" > < i class = "text-gray-200 fa-solid fa-circle-nodes" > < / i > Workers (llama.cpp): < span hx-get = "/p2p/ui/workers-stats" hx-trigger = "every 1s" > < / span > < / p >
< p class = "mb-4" > You can start llama.cpp workers to distribute weights between the workers and offload part of the computation. To start a new worker, you can use the CLI or Docker.< / p >
< div class = "grid grid-cols-1 sm:grid-cols-2 md:grid-cols-3 gap-4 mb-12" >
< div hx-get = "/p2p/ui/workers" hx-trigger = "every 1s" > < / div >
< / div >
< hr class = "border-gray-700 mb-12" >
< h3 class = "text-2xl font-semibold text-gray-100 mb-6" > < i class = "fa-solid fa-book" > < / i > Start a new llama.cpp P2P worker< / h3 >
<!-- Tabs navigation -->
< ul class = "mb-5 flex list-none flex-row flex-wrap ps-0" role = "tablist" data-twe-nav-ref >
< li role = "presentation" class = "flex-auto text-center" >
< a href = "#tabs-cli" class = "tablink my-2 block border-0 bg-gray-800 px-7 pb-3.5 pt-4 text-xs font-medium uppercase leading-tight text-white hover:bg-gray-700 focus:bg-gray-700 data-[twe-nav-active]:border-yellow-500 data-[twe-nav-active]:text-yellow-500 data-[twe-nav-active]:bg-gray-700 active" data-twe-toggle = "pill" data-twe-target = "#tabs-cli" data-twe-nav-active role = "tab" aria-controls = "tabs-cli" aria-selected = "true" > < i class = "fa-solid fa-terminal" > < / i > CLI< / a >
< / li >
< li role = "presentation" class = "flex-auto text-center" >
< a href = "#tabs-docker" class = "tablink my-2 block border-0 bg-gray-800 px-7 pb-3.5 pt-4 text-xs font-medium uppercase leading-tight text-white hover:bg-gray-700 focus:bg-gray-700 data-[twe-nav-active]:border-yellow-500 data-[twe-nav-active]:text-yellow-500 data-[twe-nav-active]:bg-gray-700" data-twe-toggle = "pill" data-twe-target = "#tabs-docker" role = "tab" aria-controls = "tabs-docker" aria-selected = "false" > < i class = "fa-solid fa-box-open" > < / i > Container images< / a >
< / li >
< / ul >
<!-- Tabs content -->
< div class = "mb-6" >
< div class = "tabcontent hidden opacity-100 transition-opacity duration-150 ease-linear data-[twe-tab-active]:block p-4" id = "tabs-cli" role = "tabpanel" aria-labelledby = "tabs-cli" data-twe-tab-active >
< p class = "mb-2" > To start a new worker, run the following command:< / p >
< code class = "block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words" >
export TOKEN="< span class = "token" > {{.P2PToken}}< / span > "< br >
local-ai worker p2p-llama-cpp-rpc
< / code >
< p class = "mt-2" > For all the options available, please refer to the < a href = "https://localai.io/features/distribute/#starting-workers" target = "_blank" class = "text-yellow-300 hover:text-yellow-400" > documentation< / a > .< / p >
< / div >
< div class = "tabcontent hidden opacity-0 transition-opacity duration-150 ease-linear data-[twe-tab-active]:block p-4" id = "tabs-docker" role = "tabpanel" aria-labelledby = "tabs-docker" >
< p class = "mb-2" > To start a new worker with docker, run the following command:< / p >
< code class = "block bg-gray-700 text-yellow-300 p-4 rounded-lg break-words" >
docker run -ti --net host -e TOKEN="< span class = "token" > {{.P2PToken}}< / span > " --name local-ai -p 8080:8080 localai/localai:latest-cpu worker p2p-llama-cpp-rpc
< / code >
< p class = "mt-2" > For all the options available and see what image to use, please refer to the < a href = "https://localai.io/basics/container/" target = "_blank" class = "text-yellow-300 hover:text-yellow-400" > Container images documentation< / a > and < a href = "https://localai.io/advanced/#cli-parameters" target = "_blank" class = "text-yellow-300 hover:text-yellow-400" > CLI parameters documentation< / a > .< / p >
< / div >
< / div >
< / div >
2024-07-23 08:42:51 +00:00
<!-- Llama.cpp Box END -->
{{ end }}
2024-07-08 20:04:06 +00:00
< / div >
< / div >
{{template "views/partials/footer" .}}
< / div >
2024-08-19 16:05:02 +00:00
< script src = "/static/p2panimation.js" > < / script >
2024-07-08 20:04:06 +00:00
< style >
.token {
word-break: break-all;
}
.workers .grid div {
display: flex;
flex-direction: column;
justify-content: space-between;
}
< / style >
< / body >
< / html >
