437 Commits

Author	SHA1	Message	Date
Ettore Di Giacinto	fb9a09d49c	chore(backend gallery): add description for remaining backends (#5679) ... * chore(backend gallery): add description for remaining backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(backend gallery): add linter Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-06-17 22:21:44 +02:00
Ettore Di Giacinto	0a78f0ad2d	chore(backend gallery): re-order and add description for vLLM (#5676) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-06-17 17:31:53 +02:00
Ettore Di Giacinto	d68660bd5a	chore(deps): bump llama.cpp to 'e434e69183fd9e1031f4445002083178c331a28b (#5665) ... chore(deps): bump llama.cpp to 'e434e69183fd9e1031f4445002083178c331a28b' Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-06-17 17:00:10 +02:00
Ettore Di Giacinto	89040ff6f7	fix: add python symlink, use absolute python env path when running backends (#5664) ... * fix: add python symlink, use absolute python env path when running backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): do not push images when building PRs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-06-16 23:00:53 +02:00
Ettore Di Giacinto	2d64269763	feat: Add backend gallery (#5607) ... * feat: Add backend gallery This PR add support to manage backends as similar to models. There is now available a backend gallery which can be used to install and remove extra backends. The backend gallery can be configured similarly as a model gallery, and API calls allows to install and remove new backends in runtime, and as well during the startup phase of LocalAI. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add backends docs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * wip: Backend Dockerfile for python backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: drop extras images, build python backends separately Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixup on all backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test CI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Tweaks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Drop old backends leftovers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixup CI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Move dockerfile upper Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix proto Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Feature dropped for consistency - we prefer model galleries Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add missing packages in the build image Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * exllama is ponly available on cublas Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * pin torch on chatterbox Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups to index Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * CI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Debug CI * Install accellerators deps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add target arch * Add cuda minor version Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Use self-hosted runners Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci: use quay for test images Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups for vllm and chatterbox Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small fixups on CI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chatterbox is only available for nvidia Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Simplify CI builds Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Adapt test, use qwen3 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(model gallery): add jina-reranker-v1-tiny-en-gguf Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gguf-parser): recover from potential panics that can happen while reading ggufs with gguf-parser Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Use reranker from llama.cpp in AIO images Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Limit concurrent jobs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2025-06-15 14:56:52 +02:00
fuder.eth	eb8c29f90a	Minor Documentation Updates: Clarified Comments in Python and Go Files (#5641) ... * Update ui.go Signed-off-by: fuder.eth <139509124+vtjl10@users.noreply.github.com> * Update backend.py Signed-off-by: fuder.eth <139509124+vtjl10@users.noreply.github.com> --------- Signed-off-by: fuder.eth <139509124+vtjl10@users.noreply.github.com>	2025-06-13 19:55:25 +02:00
Ettore Di Giacinto	88e570b5de	fix(deps): pin grpcio (#5621) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-06-10 14:21:51 +02:00
Ettore Di Giacinto	8b889955b4	chore(deps): bump pytorch to 2.7 in vllm (#5576) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-06-04 08:56:45 +02:00
Ettore Di Giacinto	cd3cd899ad	chore(deps): bump llama.cpp to '363757628848a27a435bbf22ff9476e9aeda5f40' (#5571) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-06-03 12:19:16 +02:00
Ettore Di Giacinto	ec0868e691	chore(deps): bump grpcio from 1.72.0 to 1.72.1 (#5570) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-06-03 09:59:43 +02:00
Ettore Di Giacinto	80f7f17843	chore(deps): bump llama.cpp to 'e562eece7cb476276bfc4cbb18deb7c0369b2233' (#5552) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-31 12:46:32 +02:00
Ettore Di Giacinto	d5c9c717b5	feat(chatterbox): add new backend (#5524) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-30 10:52:55 +02:00
Ettore Di Giacinto	dd7fa6b9f7	chore(deps): bump llama.cpp to 'e83ba3e460651b20a594e9f2f0f0bffb998d3ce1 (#5527) ... chore(deps): bump llama.cpp to 'e83ba3e460651b20a594e9f2f0f0bffb998d3ce1' Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-30 10:29:01 +02:00
Ettore Di Giacinto	5ffad3b004	chore(deps): remove pin on transformers (#5501) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-27 09:24:27 +02:00
Ettore Di Giacinto	88de2ea01a	feat(llama.cpp): add support for audio input (#5466) ... * feat(llama.cpp): add support for audio input Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Adapt tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-26 16:06:03 +02:00
Richard Palethorpe	bf6426aef2	feat: Realtime API support reboot (#5392) ... * feat(realtime): Initial Realtime API implementation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: go mod tidy Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat: Implement transcription only mode for realtime API Reduce the scope of the real time API for the initial realease and make transcription only mode functional. Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(build): Build backends on a separate layer to speed up core only changes Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-25 22:25:05 +02:00
Ettore Di Giacinto	3b0cf52f6a	feat(llama.cpp): add reranking (#5396) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-22 21:49:30 +02:00
Ettore Di Giacinto	6a382a1afe	fix(transformers): try to pin to working release (#5426) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-22 12:50:51 +02:00
Ettore Di Giacinto	82811a9630	fix(transformers): pin protobuf (#5421) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-20 20:28:31 +02:00
Ettore Di Giacinto	6d5bde860b	feat(llama.cpp): upgrade and use libmtmd (#5379) ... * WIP * wip * wip * Make it compile * Update json.hpp * this shouldn't be private for now * Add logs * Reset auto detected template Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Re-enable grammars * This seems to be broken - 360a9c98e1 (diff-a18a8e64e12a01167d8e98fc)[…]cccf0d4eed09d76d879L2998-L3207 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Placeholder * Simplify image loading * use completion type * disable streaming Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * correctly return timings Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Remove some debug logging * Adapt tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Keep header * embedding: do not use oai type Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Sync from server.cpp * Use utils and json directly from llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Sync with upstream Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: copy json.hpp from the correct location Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: add httplib * sync llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Embeddiongs: set OAICOMPAT_TYPE_EMBEDDING Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: sync with server.cpp by including it Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * make it darwin-compatible Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-17 16:02:53 +02:00
omahs	0f365ac204	fix: typos (#5376) ... Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>	2025-05-16 12:45:48 +02:00
Ettore Di Giacinto	dc21604741	chore(deps): bump whisper.cpp (#5338) ... * chore(deps): bump whisper.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * add libggml-metal Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups macOS arm64 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * adjust cublas for whisper.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-09 08:17:45 +02:00
Ettore Di Giacinto	adb24214c6	chore(deps): bump llama.cpp to b34c859146630dff136943abc9852ca173a7c9d6 (#5323) ... chore(deps): bump llama.cpp to 'b34c859146630dff136943abc9852ca173a7c9d6' Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-06 11:21:25 +02:00
Ettore Di Giacinto	1fc6d469ac	chore(deps): bump llama.cpp to '1d36b3670b285e69e58b9d687c770a2a0a192194 (#5307) ... chore(deps): bump llama.cpp to '1d36b3670b285e69e58b9d687c770a2a0a192194' Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-05-03 18:44:40 +02:00
Wyatt Neal	4076ea0494	fix: vllm missing logprobs (#5279) ... * working to address missing items referencing #3436, #2930 - if i could test it, this might show that the output from the vllm backend is processed and returned to the user Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com> * adding in vllm tests to test-extras Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com> * adding in tests to pipeline for execution Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com> * removing todo block, test via pipeline Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com> --------- Signed-off-by: Wyatt Neal <wyatt.neal+git@gmail.com>	2025-04-30 12:55:07 +00:00
Ettore Di Giacinto	6e8f4f584b	fix(diffusers): consider options only in form of key/value (#5277) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-29 17:08:55 +02:00
Ettore Di Giacinto	2c9279a542	feat(video-gen): add endpoint for video generation (#5247) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-26 18:05:01 +02:00
Ettore Di Giacinto	cae9bf1308	chore(deps): bump grpcio to 1.72.0 (#5244) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-25 21:32:37 +02:00
Richard Palethorpe	7f61d397d5	fix(stablediffusion-ggml): Build with DSD CUDA, HIP and Metal flags (#5236) ... Signed-off-by: Richard Palethorpe <io@richiejp.com>	2025-04-24 10:27:17 +02:00
Ettore Di Giacinto	61cc76c455	chore(autogptq): drop archived backend (#5214) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-19 15:52:29 +02:00
Ettore Di Giacinto	8abecb4a18	chore: bump grpc limits to 50MB (#5212) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-19 08:53:24 +02:00
Richard Palethorpe	0f0fafacd9	fix(stablediffusion): Avoid overwriting SYCL specific flags from outer make call (#5181) ... Signed-off-by: Richard Palethorpe <io@richiejp.com>	2025-04-15 19:31:25 +02:00
Richard Palethorpe	1b899e1a68	feat(stablediffusion): Enable SYCL (#5144) ... * feat(sycl): Enable SYCL for stable diffusion This is a pain because we compile with CGO, but SD is compiled with CMake. I don't think we can easily use CMake to set the linker flags necessary. Also I could not find pkg-config calls that would fully set the flags, so some of them are set manually. See https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl-link-line-advisor.html for reference. I also resorted to searching the shared object files in MKLROOT/lib for the symbols. Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): Don't set nproc on cmake Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2025-04-10 15:20:53 +02:00
Ettore Di Giacinto	d484028532	feat(diffusers): add support for Lumina2Text2ImgPipeline (#4806) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-10 09:55:51 +02:00
Ettore Di Giacinto	25e6f21322	chore(deps): bump llama.cpp to 4ccea213bc629c4eef7b520f7f6c59ce9bbdaca0 (#5143) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-08 11:26:06 +02:00
Ettore Di Giacinto	ece239966f	chore: ⬆️ Update ggml-org/llama.cpp to 6bf28f0111ff9f21b3c1b1eace20c590281e7ba6 (#5127) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-06 14:01:51 +02:00
Richard Palethorpe	d2cf8ef070	fix(sycl): kernel not found error by forcing -fsycl (#5115) ... * chore(sycl): Update oneapi to 2025:1 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(sycl): Pass -fsycl flag as workaround -fsycl should be set by llama.cpp's cmake file, but something goes wrong and it doesn't appear to get added Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(build): Speed up llama build by using all CPUs Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2025-04-03 16:22:59 +02:00
Ettore Di Giacinto	18b320d577	chore(deps): bump llama.cpp to 'f01bd02376f919b05ee635f438311be8dfc91d7c (#5110) ... chore(deps): bump llama.cpp to 'f01bd02376f919b05ee635f438311be8dfc91d7c' Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-03 10:23:14 +02:00
Ettore Di Giacinto	c2a39e3639	fix(llama.cpp): properly handle sigterm (#5099) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-30 18:08:29 +02:00
Ettore Di Giacinto	423514a5a5	fix(clip): do not imply GPU offload by default (#5010) ... * fix(clip): do not imply GPUs by default Until a better solution is found upstream, be conservative and default to GPU. https://github.com/ggml-org/llama.cpp/pull/12322 https://github.com/ggml-org/llama.cpp/pull/12322#issuecomment-2720970695 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * allow to override gpu via backend options Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-13 15:14:11 +01:00
Ettore Di Giacinto	1db2b9943c	chore(deps): Bump grpcio to 1.71.0 (#4993) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-11 09:44:21 +01:00
Ettore Di Giacinto	879dc73eba	Revert "chore(deps): Bump intel-extension-for-pytorch from 2.3.110+xpu to 2.6.10+xpu in /backend/python/diffusers" (#4992) ... Revert "chore(deps): Bump intel-extension-for-pytorch from 2.3.110+xpu to 2.6…" This reverts commit 1dfc52de16.	2025-03-11 08:29:05 +01:00
dependabot[bot]	1dfc52de16	chore(deps): Bump intel-extension-for-pytorch from 2.3.110+xpu to 2.6.10+xpu in /backend/python/diffusers (#4973) ... chore(deps): Bump intel-extension-for-pytorch Bumps intel-extension-for-pytorch from 2.3.110+xpu to 2.6.10+xpu. --- updated-dependencies: - dependency-name: intel-extension-for-pytorch dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2025-03-10 21:14:43 +00:00
Ettore Di Giacinto	e4fa894153	fix(llama.cpp): correctly handle embeddings in batches (#4957) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-07 19:29:52 +01:00
Ettore Di Giacinto	67f7bffd18	chore(deps): update llama.cpp and sync with upstream changes (#4950) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-06 00:40:58 +01:00
Ettore Di Giacinto	30bf6c962f	chore(stable-diffusion-ggml): update, adapt upstream changes (#4889) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-23 08:36:41 +01:00
Ettore Di Giacinto	af3bb64e42	fix(coqui): pin transformers (#4875) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-20 16:16:54 +01:00
Brandon Beiler	6a6e1a0ea9	feat(vllm): Additional vLLM config options (Disable logging, dtype, and Per-Prompt media limits) (#4855) ... * Adding the following vLLM config options: disable_log_status, dtype, limit_mm_per_prompt Signed-off-by: TheDropZone <brandonbeiler@gmail.com> * using " marks in the config.yaml file Signed-off-by: TheDropZone <brandonbeiler@gmail.com> * adding in missing colon Signed-off-by: TheDropZone <brandonbeiler@gmail.com> --------- Signed-off-by: TheDropZone <brandonbeiler@gmail.com>	2025-02-18 19:27:58 +01:00
Ettore Di Giacinto	9e32fda304	fix(llama.cpp): improve context shift handling (#4820) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-14 14:55:03 +01:00
Ettore Di Giacinto	f5638a6354	feat(diffusers): allow to override image gen options (#4807) ... Use the options field in the model to override kwargs if needed. This allows to specify from the model yaml config: ```yaml options: - foo:bar ``` And each option will be used directly when calling the diffusers pipeline, e.g: ```python pipe( foo="bar", ) ``` Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-11 10:16:32 +01:00