fdb45153fe
feat(llama.cpp): Totally decentralized, private, distributed, p2p inference ( #2343 )
...
* feat(llama.cpp): Enable decentralized, distributed inference
As https://github.com/mudler/LocalAI/pull/2324 introduced distributed inferencing thanks to
@rgerganov implementation in https://github.com/ggerganov/llama.cpp/pull/6829 in upstream llama.cpp, now
it is possible to distribute the workload to remote llama.cpp gRPC server.
This changeset now uses mudler/edgevpn to establish a secure, distributed network between the nodes using a shared token.
The token is generated automatically when starting the server with the `--p2p` flag, and can be used by starting the workers
with `local-ai worker p2p-llama-cpp-rpc` by passing the token via environment variable (TOKEN) or with args (--token).
As per how mudler/edgevpn works, a network is established between the server and the workers with dht and mdns discovery protocols,
the llama.cpp rpc server is automatically started and exposed to the underlying p2p network so the API server can connect on.
When the HTTP server is started, it will discover the workers in the network and automatically create the port-forwards to the service locally.
Then llama.cpp is configured to use the services.
This feature is behind the "p2p" GO_FLAGS
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* go mod tidy
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* ci: add p2p tag
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* better message
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
2024-05-20 19:17:59 +02:00
c89271b2e4
feat(llama.cpp): add distributed llama.cpp inferencing ( #2324 )
...
* feat(llama.cpp): support distributed llama.cpp
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* feat: let tweak how chat messages are merged together
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* refactor
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* Makefile: register to ALL_GRPC_BACKENDS
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* refactoring, allow disable auto-detection of backends
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* minor fixups
Signed-off-by: mudler <mudler@localai.io >
* feat: add cmd to start rpc-server from llama.cpp
Signed-off-by: mudler <mudler@localai.io >
* ci: add ccache
Signed-off-by: mudler <mudler@localai.io >
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
Signed-off-by: mudler <mudler@localai.io >
2024-05-15 01:17:02 +02:00
2c5a46bc34
feat(ux): Add chat, tts, and image-gen pages to the WebUI ( #2222 )
...
* feat(webui): Add chat page
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* feat(webui): Add image-gen page
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
* feat(webui): Add tts page
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
2024-05-02 21:14:10 +02:00
11c48a0004
fix: security scanner warning noise: error handlers part 2 ( #2145 )
...
check off a few more error handlers
Signed-off-by: Dave Lee <dave@gray101.com >
2024-04-29 15:11:42 +02:00
220958a87c
fix: typo in models.go ( #2099 )
2024-04-22 04:34:59 +00:00
b319ed58b0
models(gallery): add gallery ( #2078 )
...
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
2024-04-20 15:22:54 +02:00
8d30b39811
feat: fiber logs with zerlog and add trace level ( #2082 )
...
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
2024-04-20 10:43:37 +02:00
b9e7708643
feat: enable polling configs for systems with broken fsnotify (docker volumes on windows) ( #2081 )
...
* feat: enable polling configs for systems with broken fsnotify (docker volumes on windows)
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* fix: update logging to make it clear that the config file is being polled
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
---------
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
2024-04-19 19:31:15 -05:00
502c1eedaa
feat: refactor the dynamic json configs for api_keys and external_backends ( #2055 )
...
* feat: refactor the dynamic json configs for api_keys and external_backends
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* fix: remove commented code
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
---------
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com >
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com >
2024-04-18 03:21:55 +00:00
af9e5a2d05
Revert #1963 ( #2056 )
...
* Revert "fix(fncall): fix regression introduced in #1963 (#2048 )"
This reverts commit 6b06d4e0af#2038 )"
This reverts commit fdec8a9d00#2035 )"
This reverts commit e843d7df0e#1963 )"
This reverts commit eed5706994Fixes : #1920
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io >
2024-04-17 23:33:49 +02:00
538a086309
fix: previous CLI rework broke debug logging ( #2036 )
...
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
Co-authored-by: Dave <dave@gray101.com >
2024-04-15 22:13:59 +02:00
eed5706994
refactor: backend/service split, channel-based llm flow ( #1963 )
...
Refactor: channel based llm flow and services split
---------
Signed-off-by: Dave Lee <dave@gray101.com >
2024-04-13 09:45:34 +02:00
24d7dadfed
feat: kong cli refactor fixes #1955 ( #1974 )
...
* feat: migrate to alecthomas/kong for CLI
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* feat: bring in new flag for granular log levels
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* chore: go mod tidy
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* feat: allow loading cli flag values from ["./localai.yaml", "~/.config/localai.yaml", "/etc/localai.yaml"] in that order
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* feat: load from .env file instead of a yaml file
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* feat: better loading for environment files
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* feat(doc): add initial documentation about configuration
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* fix: remove test log lines
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* feat: integrate new documentation into existing pages
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* feat: add documentation on .env files
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* fix: cleanup some documentation table errors
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
* feat: refactor CLI logic out to it's own package under core/cli
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
---------
Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com >
2024-04-11 09:19:24 +02:00
