91 Commits

Author	SHA1	Message	Date
Richard Palethorpe	bf6426aef2	feat: Realtime API support reboot (#5392) ... * feat(realtime): Initial Realtime API implementation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: go mod tidy Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat: Implement transcription only mode for realtime API Reduce the scope of the real time API for the initial realease and make transcription only mode functional. Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(build): Build backends on a separate layer to speed up core only changes Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-25 22:25:05 +02:00
Ettore Di Giacinto	3b0cf52f6a	feat(llama.cpp): add reranking (#5396) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-22 21:49:30 +02:00
Ettore Di Giacinto	21bdfe5fa4	fix: use rice when embedding large binaries (#5309) ... * fix(embed): use go-rice for large backend assets Golang embed FS has a hard limit that we might exceed when providing many binary alternatives. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * simplify golang deps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(tests): switch to testcontainers and print logs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tests): do not build a test binary Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * small fixup Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-04 16:42:42 +02:00
Ettore Di Giacinto	72111c597d	fix(gpu): do not assume gpu being returned has node and mem (#5310) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-03 19:00:24 +02:00
Ettore Di Giacinto	b2f9fc870b	chore(defaults): enlarge defaults, drop gpu layers which is infered (#5308) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-03 18:44:51 +02:00
Ettore Di Giacinto	5c6cd50ed6	feat(llama.cpp): estimate vram usage (#5299) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-02 17:40:26 +02:00
Ettore Di Giacinto	2c9279a542	feat(video-gen): add endpoint for video generation (#5247) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-26 18:05:01 +02:00
Ettore Di Giacinto	9628860c0e	feat(llama.cpp/clip): inject gpu options if we detect GPUs (#5243) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-26 00:04:47 +02:00
Ettore Di Giacinto	61cc76c455	chore(autogptq): drop archived backend (#5214) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-19 15:52:29 +02:00
Gianluca Boiano	f67e5dec68	fix: bark-cpp: assign FLAG_TTS to bark-cpp backend (#5186) ... Signed-off-by: Gianluca Boiano <morf3089@gmail.com>	2025-04-16 08:21:30 +02:00
Ettore Di Giacinto	9c74d74f7b	feat(gguf): guess default context size from file (#5089) ... feat(gguf): guess default config file from files Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-29 14:42:14 +01:00
Ettore Di Giacinto	a7b4001b75	feat: allow to specify a reply prefix (#4931) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-02 16:07:32 +01:00
Ettore Di Giacinto	bbbb28e3ca	fix(models): unify usecases identifications (#4914) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-27 15:51:12 +01:00
Brandon Beiler	6a6e1a0ea9	feat(vllm): Additional vLLM config options (Disable logging, dtype, and Per-Prompt media limits) (#4855) ... * Adding the following vLLM config options: disable_log_status, dtype, limit_mm_per_prompt Signed-off-by: TheDropZone <brandonbeiler@gmail.com> * using " marks in the config.yaml file Signed-off-by: TheDropZone <brandonbeiler@gmail.com> * adding in missing colon Signed-off-by: TheDropZone <brandonbeiler@gmail.com> --------- Signed-off-by: TheDropZone <brandonbeiler@gmail.com>	2025-02-18 19:27:58 +01:00
Ettore Di Giacinto	5b19af99ff	feat(ui): detect model usage and display link (#4864) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-18 19:27:07 +01:00
Dave	3cddf24747	feat: Centralized Request Processing middleware (#3847) ... * squash past, centralize request middleware PR Signed-off-by: Dave Lee <dave@gray101.com> * migrate bruno request files to examples repo Signed-off-by: Dave Lee <dave@gray101.com> * fix Signed-off-by: Dave Lee <dave@gray101.com> * Update tests/e2e-aio/e2e_test.go Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> --------- Signed-off-by: Dave Lee <dave@gray101.com> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2025-02-10 12:06:16 +01:00
Ettore Di Giacinto	cc1f6f913f	fix(llama.cpp): disable mirostat as default (#2911) ... Even if increasing the quality of the output, it has shown to have performance drawbacks to be so noticeable that the confuses users about speed of LocalAI ( see also https://github.com/mudler/LocalAI/issues/2780 ). This changeset disables Mirostat by default (which can be still enabled manually). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Dave <dave@gray101.com>	2025-02-06 19:39:59 +01:00
Ettore Di Giacinto	af41436f1b	fix(tests): pin to branch for config used in tests (#4721) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-31 09:57:58 +01:00
Ettore Di Giacinto	72e52c4f6a	chore: drop embedded models (#4715) ... Since the remote gallery was introduced this is now completely superseded by it. In order to keep the code clean and remove redudant parts let's simplify the usage. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-30 00:03:01 +01:00
Ettore Di Giacinto	e15d29aba2	chore(stablediffusion-ncn): drop in favor of ggml implementation (#4652) ... * chore(stablediffusion-ncn): drop in favor of ggml implementation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ci): drop stablediffusion build Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(tests): add Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(tests): try to fixup current tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Try to fix tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Tests improvements Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(tests): use quality to specify step Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(tests): switch to sd-1.5 also increase prep time for downloading models Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-22 19:34:16 +01:00
Gianluca Boiano	032a33de49	chore: remove deprecated tinydream backend (#4631) ... Signed-off-by: Gianluca Boiano <morf3089@gmail.com>	2025-01-18 18:35:30 +01:00
mintyleaf	96f8ec0402	feat: add machine tag and inference timings (#4577) ... * Add machine tag option, add extraUsage option, grpc-server -> proto -> endpoint extraUsage data is broken for now Signed-off-by: mintyleaf <mintyleafdev@gmail.com> * remove redurant timing fields, fix not working timings output Signed-off-by: mintyleaf <mintyleafdev@gmail.com> * use middleware for Machine-Tag only if tag is specified Signed-off-by: mintyleaf <mintyleafdev@gmail.com> --------- Signed-off-by: mintyleaf <mintyleafdev@gmail.com>	2025-01-17 17:05:58 +01:00
Ettore Di Giacinto	7d0ac1ea3f	chore(vall-e-x): Drop backend (#4619) ... There are many new architectures that are SOTA and replaces vall-e-x nowadays. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-17 09:35:10 +01:00
Ettore Di Giacinto	cea5a0ea42	feat(template): read jinja templates from gguf files (#4332) ... * Read jinja templates as fallback Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Move templating out of model loader Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Test TemplateMessages Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Set role and content from transformers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Tests: be more flexible Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * More jinja Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small refactoring and adaptations Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-08 13:50:33 +01:00
Ettore Di Giacinto	d4c1746c7d	feat(llama.cpp): expose cache_type_k and cache_type_v for quant of kv cache (#4329) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-06 10:23:59 +01:00
Ettore Di Giacinto	44a5dac312	feat(backend): add stablediffusion-ggml (#4289) ... * feat(backend): add stablediffusion-ggml Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ci): track stablediffusion-ggml Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Use default scheduler and sampler if not specified Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Move cfg scale out of diffusers block Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Make it working Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: set free_params_immediately to false to call the model in sequence https://github.com/leejet/stable-diffusion.cpp/issues/366 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-03 22:41:22 +01:00
Ettore Di Giacinto	4f1ab2366d	chore(refactor): imply modelpath (#4208) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-20 18:06:35 +01:00
Ettore Di Giacinto	947224b952	feat(diffusers): allow multiple lora adapters (#4081) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-05 15:14:33 +01:00
Ettore Di Giacinto	ae1ec4e096	feat(vllm): expose 'load_format' (#3943) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-23 15:34:57 +02:00
Ettore Di Giacinto	8737a65760	feat: allow to disable '/metrics' endpoints for local stats (#3945) ... Seem the "/metrics" endpoint that is source of confusion as people tends to believe we collect telemetry data just because we import "opentelemetry", however it is still a good idea to allow to disable even local metrics if not really required. See also: https://github.com/mudler/LocalAI/issues/3942 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-23 15:34:32 +02:00
Ettore Di Giacinto	ccc7cb0287	feat(templates): use a single template for multimodals messages (#3892) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-22 09:34:05 +02:00
Ettore Di Giacinto	648ffdf449	feat(multimodal): allow to template placeholders (#3728) ... feat(multimodal): allow to template image placeholders Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-04 18:32:29 +02:00
Dave	307a835199	groundwork: ListModels Filtering Upgrade (#2773) ... * seperate the filtering from the middleware changes --------- Signed-off-by: Dave Lee <dave@gray101.com>	2024-10-01 18:55:46 +00:00
Sertaç Özercan	ee21b00a8d	feat: auto load into memory on startup (#3627) ... Signed-off-by: Sertac Ozercan <sozercan@gmail.com>	2024-09-22 10:03:30 +02:00
Dave	db1159b651	feat: auth v2 - supersedes #2894 (#3476) ... feat: auth v2 - supercedes #2894, metrics to follow later Signed-off-by: Dave Lee <dave@gray101.com>	2024-09-16 23:29:07 -04:00
Ettore Di Giacinto	cf747bcdec	feat: extract output with regexes from LLMs (#3491) ... * feat: extract output with regexes from LLMs This changset adds `extract_regex` to the LLM config. It is a list of regexes that can match output and will be used to re extract text from the LLM output. This is particularly useful for LLMs which outputs final results into tags. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add tests, enhance output in case of configuration error Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-09-13 13:27:36 +02:00
Ettore Di Giacinto	7278bf3de8	chore: allow to disable gallery endpoints, improve p2p connection handling (#3256) ... * Add more debug messages Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: allow to disable gallery endpoints Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * improve p2p messaging Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * improve error handling Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Make sure to close the listening socket when context is exhausted Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-08-17 08:28:52 +02:00
Ettore Di Giacinto	36e185ba63	feat(p2p): allow to run multiple clusters in the same p2p network (#3128) ... feat(p2p): allow to run multiple clusters in the same network Allow to specify a network ID via CLI which allows to run multiple clusters, logically separated within the same network (by using the same shared token). Note: This segregation is not "secure" by any means, anyone having the network token can see the services available in all the network, however, this provides a way to separate the inference endpoints. This allows for instance to have a node which is both federated and having attached a set of llama.cpp workers. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-08-07 23:35:44 +02:00
Sertaç Özercan	d1a123954b	feat(guesser): add gemma2 (#3118) ... * feat(guesser): add gemma2 Signed-off-by: Sertac Ozercan <sozercan@gmail.com> * update Signed-off-by: Sertac Ozercan <sozercan@gmail.com> --------- Signed-off-by: Sertac Ozercan <sozercan@gmail.com>	2024-08-04 09:45:42 +02:00
Ettore Di Giacinto	a36b721ca6	fix: be consistent in downloading files, check for scanner errors (#3108) ... * fix(downloader): be consistent in downloading files This PR puts some order in the downloader such as functions are re-used across several places. This fixes an issue with having uri's inside the model YAML file, it would resolve to MD5 rather then using the filename Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(scanner): do raise error only if unsafeFiles are found Fixes: https://github.com/mudler/LocalAI/issues/3114 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-08-02 20:06:25 +02:00
Ettore Di Giacinto	35561edb6e	feat(llama.cpp): support embeddings endpoints (#2871) ... * feat(llama.cpp): add embeddings Also enable embeddings by default for llama.cpp models Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(Makefile): prepare llama.cpp sources only once Otherwise we keep cloning llama.cpp for each of the variants Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * do not set embeddings to false Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add embeddings to the YAML config reference Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-07-15 22:54:16 +02:00
Ettore Di Giacinto	b7c0d46170	feat: models(gallery): add deepseek-v2-lite (#2658) ... * models(gallery): add deepseek-v2-lite Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Update deepseek.yaml The trailing space here is presumably part of the template string - try use a chomp keep to get yaml lint to accept it? Signed-off-by: Dave <dave@gray101.com> * Update deepseek.yaml chomp didn't fix, erase the space and see what happens. Signed-off-by: Dave <dave@gray101.com> * Update deepseek.yaml Signed-off-by: Dave <dave@gray101.com> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: Dave <dave@gray101.com> Co-authored-by: Dave <dave@gray101.com>	2024-07-13 17:09:59 -04:00
Dave	133987b1fb	feat: HF /scan endpoint (#2566) ... * start by checking /scan during the checksum update Signed-off-by: Dave Lee <dave@gray101.com> * add back in golang side features: downloader/uri gets struct and scan function, gallery uses it, and secscan/models calls it. Signed-off-by: Dave Lee <dave@gray101.com> * add a param to scan specific urls - useful for debugging Signed-off-by: Dave Lee <dave@gray101.com> * helpful printouts Signed-off-by: Dave Lee <dave@gray101.com> * fix offsets Signed-off-by: Dave Lee <dave@gray101.com> * fix error and naming Signed-off-by: Dave Lee <dave@gray101.com> * expose error Signed-off-by: Dave Lee <dave@gray101.com> * fix json tags Signed-off-by: Dave Lee <dave@gray101.com> * slight wording change Signed-off-by: Dave Lee <dave@gray101.com> * go mod tidy - getting warnings Signed-off-by: Dave Lee <dave@gray101.com> * split out python to make editing easier, add some simple code to delete contaminated entries from gallery Signed-off-by: Dave Lee <dave@gray101.com> * o7 to my favorite part of our old name, go-skynet Signed-off-by: Dave Lee <dave@gray101.com> * merge fix Signed-off-by: Dave Lee <dave@gray101.com> * merge fix Signed-off-by: Dave Lee <dave@gray101.com> * merge fix Signed-off-by: Dave Lee <dave@gray101.com> * address review comments Signed-off-by: Dave Lee <dave@gray101.com> * forgot secscan could accept multiple URL at once Signed-off-by: Dave Lee <dave@gray101.com> * invert naming and actually use it Signed-off-by: Dave Lee <dave@gray101.com> * missed cli/models.go Signed-off-by: Dave Lee <dave@gray101.com> * Update .github/check_and_update.py Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Signed-off-by: Dave <dave@gray101.com> --------- Signed-off-by: Dave Lee <dave@gray101.com> Signed-off-by: Dave <dave@gray101.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2024-07-10 13:18:32 +02:00
Ettore Di Giacinto	cca881ec49	feat(p2p): Federation and AI swarms (#2723) ... * Wip p2p enhancements * get online state * Pass-by token to show in the dashboard Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Style * Minor fixups * parametrize SearchID * Refactoring * Allow to expose/bind more services Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add federation * Display federated mode in the WebUI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * make federated nodes visible from the WebUI * Fix version display * improve web page * live page update * visual enhancements * enhancements * visual enhancements --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-07-08 22:04:06 +02:00
Ettore Di Giacinto	a181dd0ebc	refactor: gallery inconsistencies (#2647) ... * refactor(gallery): move under core/ Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(unarchive): do not allow symlinks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-24 17:32:12 +02:00
Sertaç Özercan	5866fc8ded	chore: fix go.mod module (#2635) ... Signed-off-by: Sertac Ozercan <sozercan@gmail.com>	2024-06-23 08:24:36 +00:00
Ettore Di Giacinto	94cfaad7f4	feat(libpath): refactor and expose functions for external library paths (#2578) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-16 13:58:28 +02:00
Ettore Di Giacinto	8f952d90b0	feat(guesser): identify gemma models (#2561) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-13 19:12:37 +02:00
Ettore Di Giacinto	14b41be057	feat(detection): detect by template in gguf file, add qwen2, phi, mistral and chatml (#2536) ... feat(detection): detect by template in gguf file, add qwen and chatml Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-10 22:58:04 +02:00
Ettore Di Giacinto	d9109ffafb	feat(defaults): add defaults for Command-R models (#2529) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-09 20:00:16 +02:00