Commit Graph

194 Commits

Author SHA1 Message Date
Ettore Di Giacinto
a3d69872e3
feat(api): list loaded models in /system (#3661)
feat(api): list loaded models in /system

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-09-25 18:00:23 +02:00
Ettore Di Giacinto
0893d3cbbe
fix(health): do not require auth for /healthz and /readyz (#3656)
* fix(health): do not require auth for /healthz and /readyz

Fixes: #3655

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Comment so I don’t forget

Adding a reminder here...

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Dave <dave@gray101.com>
2024-09-24 18:25:59 +00:00
Dave
90cacb9692
test: preliminary tests and merge fix for authv2 (#3584)
* add api key to existing app tests, add preliminary auth test

Signed-off-by: Dave Lee <dave@gray101.com>

* small fix, run test

Signed-off-by: Dave Lee <dave@gray101.com>

* status on non-opaque

Signed-off-by: Dave Lee <dave@gray101.com>

* tweak auth error

Signed-off-by: Dave Lee <dave@gray101.com>

* exp

Signed-off-by: Dave Lee <dave@gray101.com>

* quick fix on real laptop

Signed-off-by: Dave Lee <dave@gray101.com>

* add downloader version that allows providing an auth header

Signed-off-by: Dave Lee <dave@gray101.com>

* stash some devcontainer fixes during testing

Signed-off-by: Dave Lee <dave@gray101.com>

* s2

Signed-off-by: Dave Lee <dave@gray101.com>

* s

Signed-off-by: Dave Lee <dave@gray101.com>

* done with experiment

Signed-off-by: Dave Lee <dave@gray101.com>

* done with experiment

Signed-off-by: Dave Lee <dave@gray101.com>

* after merge fix

Signed-off-by: Dave Lee <dave@gray101.com>

* rename and fix

Signed-off-by: Dave Lee <dave@gray101.com>

---------

Signed-off-by: Dave Lee <dave@gray101.com>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
2024-09-24 09:32:48 +02:00
Sertaç Özercan
ee21b00a8d
feat: auto load into memory on startup (#3627)
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
2024-09-22 10:03:30 +02:00
Ettore Di Giacinto
191bc2e50a
feat(api): allow to pass audios to backends (#3603)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-09-19 12:26:53 +02:00
Ettore Di Giacinto
fbb9facda4
feat(api): allow to pass videos to backends (#3601)
This prepares the API to receive videos as well for video understanding.

It works similarly to images, where the request should be in the form:

{
 "type": "video_url",
 "video_url": { "url": "url or base64 data" }
}

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-09-19 11:21:59 +02:00
Ettore Di Giacinto
a53392f919
chore(refactor): drop duplicated shutdown logics (#3589)
* chore(refactor): drop duplicated shutdown logics

- Handle locking in Shutdown and CheckModelIsLoaded in a more go-idiomatic way
- Drop duplicated code and re-organize shutdown code

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix: drop leftover

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore: improve logging and add missing locks

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-09-17 16:51:40 +02:00
Dave
db1159b651
feat: auth v2 - supersedes #2894 (#3476)
feat: auth v2 - supercedes #2894, metrics to follow later

Signed-off-by: Dave Lee <dave@gray101.com>
2024-09-16 23:29:07 -04:00
Ettore Di Giacinto
cf747bcdec
feat: extract output with regexes from LLMs (#3491)
* feat: extract output with regexes from LLMs

This changset adds `extract_regex` to the LLM config. It is a list of
regexes that can match output and will be used to re extract text from
the LLM output. This is particularly useful for LLMs which outputs final
results into tags.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add tests, enhance output in case of configuration error

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-09-13 13:27:36 +02:00
Dave
300f2779e4
fix: speedup and improve cachability of docker build of builder-sd (#3430)
fix: speedup and improve cachability of docker build of `builder-sd` (#3430)
---------

Signed-off-by: Dave Lee <dave@gray101.com>
2024-09-10 08:57:16 +02:00
Ettore Di Giacinto
791c3ace72
feat: add endpoint to list system informations (#3449)
* feat: add endpoint to list system informations

For now, it lists the available backends, but can be expanded later on
to include more system informations (such as GPU devices detected, RAM,
threads configured, and so on so forth).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* show also external backends

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* add test

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-09-05 20:44:30 +02:00
Dave
c2804c42fe
fix: untangle pkg/grpc and core/schema for Transcription (#3419)
untangle pkg/grpc and core/schema in Transcribe

Signed-off-by: Dave Lee <dave@gray101.com>
2024-09-02 15:48:53 +02:00
Ettore Di Giacinto
11d960b2a6
chore(cli): be consistent between workers and expose ExtraLLamaCPPArgs to both (#3428)
* chore(cli): be consistent between workers and expose ExtraLLamaCPPArgs to both

Fixes: https://github.com/mudler/LocalAI/issues/3427

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* bump grpcio

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-30 00:10:17 +02:00
Ettore Di Giacinto
da3bc8077d
fix(tts): check error before inspecting result (#3415)
Otherwise we panic when a res is nil

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-27 17:35:16 +02:00
Ettore Di Giacinto
7f06954425
fix(model-loading): keep track of open GRPC Clients (#3377)
Due to a previous refactor we moved the client constructor tight to the
model address, however that was just a string which we would use to
build the client each time.

With this change we make the loader to return a *Model which carries a
constructor for the client and stores the client on the first
connection.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-25 14:36:09 +02:00
Ettore Di Giacinto
ce827139bb
fix(p2p): correctly allow to pass extra args to llama.cpp (#3368)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-24 10:30:24 +02:00
Dave
81ae92f017
feat: elevenlabs sound-generation api (#3355)
* initial version of elevenlabs compatible soundgeneration api and cli command

Signed-off-by: Dave Lee <dave@gray101.com>

* minor cleanup

Signed-off-by: Dave Lee <dave@gray101.com>

* restore TTS, add test

Signed-off-by: Dave Lee <dave@gray101.com>

* remove stray s

Signed-off-by: Dave Lee <dave@gray101.com>

* fix

Signed-off-by: Dave Lee <dave@gray101.com>

---------

Signed-off-by: Dave Lee <dave@gray101.com>
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
2024-08-24 00:20:28 +00:00
Ettore Di Giacinto
fbaae8528d
fix(chat): re-generated uuid, created, and text on each request (#3359)
This was noticed by models returning content besides function calls.
Sadly we can't test that easily in the CI so it got unnoticed.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-22 10:56:05 +02:00
Ettore Di Giacinto
023ce59d44
feat(p2p): allow to set intervals (#3353)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-21 18:23:51 +02:00
Ettore Di Giacinto
7822d944b5
chore(p2p): single-node when sharing federated instance (#3354)
* chore(p2p): single-node when sharing federated instance

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore: refactor out and extract into functions

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-21 18:23:42 +02:00
Ettore Di Giacinto
b510352393
chore(anime.js): drop unused (#3351)
* fix(anime.js): correctly set the static path

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Drop anime.js (unused)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-21 13:10:09 +02:00
Ettore Di Giacinto
af095204fa
fix(p2p): avoid starting the node twice (#3349)
* fix(p2p): avoid starting the node twice

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(p2p): keep exposing service if we don't start the llama.cpp runner

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-21 10:30:56 +02:00
Ettore Di Giacinto
2669f4738a
fix(p2p): re-use p2p host when running federated mode (#3341)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-20 20:14:17 +02:00
Ettore Di Giacinto
6d350ccce0
feat(federation): do not allocate local services for load balancing (#3337)
* refactor: extract proxy into functions

* feat(federation): do not allocate services, directly connect with libp2p

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-20 14:45:36 +02:00
Ettore Di Giacinto
0c84c7b1cc
chore(ux): allow to create and drag dots in the animation (#3287)
Make the animation more interactive!

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-19 20:40:55 +02:00
Ettore Di Giacinto
73c9b3598d
chore(p2p): make commands easier to copy-paste (#3273)
chore(p2p): make box easier to copy-paste

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-19 19:58:17 +02:00
Ettore Di Giacinto
13cb7960bd
chore(ux): add animated header with anime.js in p2p sections (#3271)
feat(p2p): add animated header with anime.js

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-19 18:05:02 +02:00
Ettore Di Giacinto
e4c696d966 chore(p2p): customize defaults
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-19 12:45:51 +02:00
Ettore Di Giacinto
d58f9c333b chore(p2p): lower default dht interval default
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-19 12:41:13 +02:00
Ettore Di Giacinto
1dbb3b8abc
fix(gallery): be consistent and disable UI routes as well (#3262)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-18 09:26:29 +02:00
Ettore Di Giacinto
27b03a52f3
fix(p2p): allocate tunnels only when needed (#3259)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-17 15:03:55 +02:00
Ettore Di Giacinto
7278bf3de8
chore: allow to disable gallery endpoints, improve p2p connection handling (#3256)
* Add more debug messages

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat: allow to disable gallery endpoints

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* improve p2p messaging

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* improve error handling

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Make sure to close the listening socket when context is exhausted

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-17 08:28:52 +02:00
Ettore Di Giacinto
3457acc48b
chore(explorer): add join instructions (#3255)
* feat(explorer): give CLI instructions to join federated clusters

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* debug message

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-16 19:34:36 +02:00
Ettore Di Giacinto
c50e0edcb8
feat(gallery): lazy load images (#3246)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-14 12:53:42 +02:00
Ettore Di Giacinto
d6c4e751f2
feat(explorer): visual improvements (#3247)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-14 12:53:29 +02:00
Ettore Di Giacinto
02de274e00
feat(federated): allow to pickup a specific worker, improve loadbalancing (#3243)
* feat(explorer): allow to specify a worker target

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(explorer): correctly load balance requests

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(explorer): mark load balanced by default

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix: make sure to delete tunnels that might not exist anymore

If a worker goes off and on might change tunnel address, and we want to
load balance only on the active tunnels.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-13 16:17:18 +02:00
Ettore Di Giacinto
9729d2ae37
feat(explorer): make possible to run sync in a separate process (#3224)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-12 19:25:44 +02:00
Ettore Di Giacinto
c4534cd908
chore(deps): update edgevpn (#3214)
* chore(deps): update edgevpn

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix: initialize failure map

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-11 10:46:17 +02:00
Ettore Di Giacinto
7ba4a78fcc
fix(explorer): reset counter when network is active (#3213)
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
2024-08-11 00:59:58 +02:00
Ettore Di Giacinto
8627bc2dd4
feat(explorer): relax token deletion with error threshold (#3211)
feat(explorer): relax token deletion with error threashold

Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
2024-08-10 20:50:57 +02:00
Ettore Di Giacinto
9e3e892ac7
feat(p2p): add network explorer and community pools (#3125)
* WIP

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fixups

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Wire up a simple explorer DB

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* wip

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* WIP

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor: group services id so can be identified easily in the ledger table

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(discovery): discovery service now gather worker informations correctly

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(explorer): display network token

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(explorer): display form to add new networks

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(explorer): stop from overwriting networks

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(explorer): display only networks with active workers

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(explorer): list only clusters in a network if it has online workers

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* remove invalid and inactive networks

if networks have no workers delete them from the database, similarly,
if invalid.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci: add workflow to deploy new explorer versions automatically

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* build-api: build with p2p tag

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Allow to specify a connection timeout

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* logging

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Better p2p defaults

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Set loglevel

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fix dht enable

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Default to info for loglevel

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add navbar

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Slightly improve rendering

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Allow to copy the token easily

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci fixups

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-09 20:12:01 +02:00
Ettore Di Giacinto
8814b31805
chore: drop gpt4all.cpp (#3106)
chore: drop gpt4all

gpt4all is already supported in llama.cpp - the backend was kept for
keeping compatibility with old gpt4all models (prior to gguf format).

It is good time now to clean up and remove it to slim the compilation
process.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-07 23:35:55 +02:00
Ettore Di Giacinto
36e185ba63
feat(p2p): allow to run multiple clusters in the same p2p network (#3128)
feat(p2p): allow to run multiple clusters in the same network

Allow to specify a network ID via CLI which allows to run multiple
clusters, logically separated within the same network (by using the same
shared token).

Note: This segregation is not "secure" by any means, anyone having the
network token can see the services available in all the network,
however, this provides a way to separate the inference endpoints.

This allows for instance to have a node which is both federated and
having attached a set of llama.cpp workers.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-07 23:35:44 +02:00
Ettore Di Giacinto
e198347886
feat(openai): add json_schema format type and strict mode (#3193)
* feat(openai): add json_schema and strict mode

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* handle err vs _

security scanners prefer if we put these branches in, and I tend to agree.

Signed-off-by: Dave <dave@gray101.com>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Dave <dave@gray101.com>
Co-authored-by: Dave <dave@gray101.com>
2024-08-07 15:27:02 -04:00
Sertaç Özercan
d1a123954b
feat(guesser): add gemma2 (#3118)
* feat(guesser): add gemma2

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

* update

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>

---------

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
2024-08-04 09:45:42 +02:00
Ettore Di Giacinto
a36b721ca6
fix: be consistent in downloading files, check for scanner errors (#3108)
* fix(downloader): be consistent in downloading files

This PR puts some order in the downloader such as functions are re-used
across several places.

This fixes an issue with having uri's inside the model YAML file, it
would resolve to MD5 rather then using the filename

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(scanner): do raise error only if unsafeFiles are found

Fixes: https://github.com/mudler/LocalAI/issues/3114

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-02 20:06:25 +02:00
Ettore Di Giacinto
d792cf115b
fix(ui): do not show duplicate entries if not installed by gallery (#3107)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-08-01 17:27:40 +02:00
Ettore Di Giacinto
d6a7a77f6b
fix(gallery): do clear out errors once displayed (#3033)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-07-28 10:28:10 +02:00
Ettore Di Giacinto
2a839e1432
fix(gallery): do not attempt to delete duplicate files (#3031)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-07-28 10:27:56 +02:00
Ettore Di Giacinto
2169c3497d
feat(grammar): add llama3.1 schema (#3015)
* wip

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* get rid of panics

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* expose it properly from the config

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Simplify

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* forgot to commit

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Remove focus on test

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Small fixups

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2024-07-26 20:11:29 +02:00