76 Commits

Author	SHA1	Message	Date
Ettore Di Giacinto	8abecb4a18	chore: bump grpc limits to 50MB (#5212) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-19 08:53:24 +02:00
Richard Palethorpe	1b899e1a68	feat(stablediffusion): Enable SYCL (#5144) ... * feat(sycl): Enable SYCL for stable diffusion This is a pain because we compile with CGO, but SD is compiled with CMake. I don't think we can easily use CMake to set the linker flags necessary. Also I could not find pkg-config calls that would fully set the flags, so some of them are set manually. See https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl-link-line-advisor.html for reference. I also resorted to searching the shared object files in MKLROOT/lib for the symbols. Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): Don't set nproc on cmake Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2025-04-10 15:20:53 +02:00
Ettore Di Giacinto	25e6f21322	chore(deps): bump llama.cpp to 4ccea213bc629c4eef7b520f7f6c59ce9bbdaca0 (#5143) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-08 11:26:06 +02:00
Ettore Di Giacinto	ece239966f	chore: ⬆️ Update ggml-org/llama.cpp to 6bf28f0111ff9f21b3c1b1eace20c590281e7ba6 (#5127) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-06 14:01:51 +02:00
Richard Palethorpe	d2cf8ef070	fix(sycl): kernel not found error by forcing -fsycl (#5115) ... * chore(sycl): Update oneapi to 2025:1 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(sycl): Pass -fsycl flag as workaround -fsycl should be set by llama.cpp's cmake file, but something goes wrong and it doesn't appear to get added Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(build): Speed up llama build by using all CPUs Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2025-04-03 16:22:59 +02:00
Ettore Di Giacinto	18b320d577	chore(deps): bump llama.cpp to 'f01bd02376f919b05ee635f438311be8dfc91d7c (#5110) ... chore(deps): bump llama.cpp to 'f01bd02376f919b05ee635f438311be8dfc91d7c' Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-04-03 10:23:14 +02:00
Ettore Di Giacinto	c2a39e3639	fix(llama.cpp): properly handle sigterm (#5099) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-30 18:08:29 +02:00
Ettore Di Giacinto	423514a5a5	fix(clip): do not imply GPU offload by default (#5010) ... * fix(clip): do not imply GPUs by default Until a better solution is found upstream, be conservative and default to GPU. https://github.com/ggml-org/llama.cpp/pull/12322 https://github.com/ggml-org/llama.cpp/pull/12322#issuecomment-2720970695 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * allow to override gpu via backend options Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-13 15:14:11 +01:00
Ettore Di Giacinto	e4fa894153	fix(llama.cpp): correctly handle embeddings in batches (#4957) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-07 19:29:52 +01:00
Ettore Di Giacinto	67f7bffd18	chore(deps): update llama.cpp and sync with upstream changes (#4950) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-06 00:40:58 +01:00
Ettore Di Giacinto	9e32fda304	fix(llama.cpp): improve context shift handling (#4820) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-14 14:55:03 +01:00
Shraddha	03974a4dd4	feat: tokenization with llama.cpp (#4724) ... feat: tokenization Signed-off-by: shraddhazpy <shraddha@shraddhafive.in>	2025-02-02 17:39:43 +00:00
Ettore Di Giacinto	1d6afbd65d	feat(llama.cpp): Add support to grammar triggers (#4733) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-02 13:25:03 +01:00
Ettore Di Giacinto	958f6eb722	chore(llama.cpp): update dependency (#4628) ... Update to '3edfa7d3753c29e44b964c0ff424d2ea8d5fdee6' and adapt to upstream changes Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-18 11:55:13 +01:00
mintyleaf	96f8ec0402	feat: add machine tag and inference timings (#4577) ... * Add machine tag option, add extraUsage option, grpc-server -> proto -> endpoint extraUsage data is broken for now Signed-off-by: mintyleaf <mintyleafdev@gmail.com> * remove redurant timing fields, fix not working timings output Signed-off-by: mintyleaf <mintyleafdev@gmail.com> * use middleware for Machine-Tag only if tag is specified Signed-off-by: mintyleaf <mintyleafdev@gmail.com> --------- Signed-off-by: mintyleaf <mintyleafdev@gmail.com>	2025-01-17 17:05:58 +01:00
Ettore Di Giacinto	ab5adf40af	chore(deps): bump llama.cpp to '924518e2e5726e81f3aeb2518fb85963a500e… (#4592) ... chore(deps): bump llama.cpp to '924518e2e5726e81f3aeb2518fb85963a500e93a' Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-13 17:33:06 +01:00
Ettore Di Giacinto	c553d73748	chore(deps): bump llama.cpp to 4b0c638b9 (#4532) ... deps(llama.cpp): bump to 4b0c638b9 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-04 09:40:08 +01:00
Ettore Di Giacinto	0eb2911aad	chore(llava): update clip.patch (#4453) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-23 19:11:31 +01:00
Ettore Di Giacinto	708cba0c1b	chore(llama.cpp): bump, drop penalize_nl (#4418) ... deps(llama.cpp): bump, drop penalize_nl Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-17 00:47:52 +01:00
Ettore Di Giacinto	fc4a714992	feat(llama.cpp): bump and adapt to upstream changes (#4378) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-14 00:30:52 +01:00
Ettore Di Giacinto	d4c1746c7d	feat(llama.cpp): expose cache_type_k and cache_type_v for quant of kv cache (#4329) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-06 10:23:59 +01:00
Ettore Di Giacinto	cbedf2f428	fix(llama.cpp): embed metal file into result binary for darwin (#4279) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-28 04:17:00 +00:00
Ettore Di Giacinto	2b62260b6d	feat(models): use rwkv from llama.cpp (#4264) ... feat(rwkv): use rwkv from llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-26 14:22:55 +01:00
Ettore Di Giacinto	404ca3cc23	chore(deps): bump llama.cpp to 47f931c8f9a26c072d71224bc8013cc66ea9e445 (#4263) ... chore(deps): bump llama.cpp to '47f931c8f9a26c072d71224bc8013cc66ea9e445' Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-26 11:12:57 +01:00
Ettore Di Giacinto	939fbe59cc	chore(deps): bump llama-cpp to ae8de6d50a09d49545e0afab2e50cc4acfb280e2 (#4157) ... * chore(deps): bump llama-cpp to ae8de6d50a09d49545e0afab2e50cc4acfb280e2 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(metal): metal file has moved Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-15 12:51:43 +01:00
Ettore Di Giacinto	3d4bb757d2	chore(deps): bump llama-cpp to 8f275a7c4593aa34147595a90282cf950a853690 (#4016) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-30 08:31:13 +01:00
Ettore Di Giacinto	32db787991	chore(deps): bump llama-cpp to cda0e4b648dde8fac162b3430b14a99597d3d74f (#3884) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-20 00:26:49 +02:00
Ettore Di Giacinto	6257e2f510	chore(deps): bump llama-cpp to 96776405a17034dcfd53d3ddf5d142d34bdbb657 (#3793) ... This adapts also to upstream changes Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-12 01:25:03 +02:00
siddimore	f84b55d1ef	feat: Add Get Token Metrics to GRPC server (#3687) ... * Add Get Token Metrics to GRPC server Signed-off-by: Siddharth More <siddimore@gmail.com> * Expose LocalAI endpoint Signed-off-by: Siddharth More <siddimore@gmail.com> --------- Signed-off-by: Siddharth More <siddimore@gmail.com>	2024-10-01 14:41:20 +02:00
siddimore	50a3b54e34	feat(api): add correlationID to Track Chat requests (#3668) ... * Add CorrelationID to chat request Signed-off-by: Siddharth More <siddimore@gmail.com> * remove get_token_metrics Signed-off-by: Siddharth More <siddimore@gmail.com> * Add CorrelationID to proto Signed-off-by: Siddharth More <siddimore@gmail.com> * fix correlation method name Signed-off-by: Siddharth More <siddimore@gmail.com> * Update core/http/endpoints/openai/chat.go Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Signed-off-by: Siddharth More <siddimore@gmail.com> * Update core/http/endpoints/openai/chat.go Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Signed-off-by: Siddharth More <siddimore@gmail.com> --------- Signed-off-by: Siddharth More <siddimore@gmail.com> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2024-09-28 17:23:56 +02:00
Ettore Di Giacinto	25deb4ba95	chore(deps): update llama.cpp to 6262d13e0b2da91f230129a93a996609a2fa2f2 (#3549) ... chore(deps): update llama.cpp to 6262d13e0b2da91f230129a93a996609a2f5a2f2 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-09-16 10:29:20 +02:00
Ettore Di Giacinto	d51444d606	chore(deps): update llama.cpp (#3497) ... * Apply llava patch Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-09-12 20:55:27 +02:00
Ettore Di Giacinto	b8e7a76524	chore(deps): update llama.cpp (#3438) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-08-31 01:21:45 +02:00
Ettore Di Giacinto	409e2d348e	chore(deps): bump llama.cpp, rename llama_add_bos_token (#3253) ... deps(llama.cpp): bump, rename llama_add_bos_token Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-08-16 01:20:21 +02:00
Ettore Di Giacinto	abcf0ff000	chore: ⬆️ Update ggerganov/llama.cpp to 1e6f6554aa11fa10160a5fda689e736c3c34169f (#3189) ... * arrow_up: Update ggerganov/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(llama.cpp): adapt to upstream naming changes Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>	2024-08-07 01:10:21 +02:00
Ettore Di Giacinto	4e11ca55fd	chore: ⬆️ Update ggerganov/llama.cpp (#3166) ... * arrow_up: Update ggerganov/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(llama.cpp): adapt init function call Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>	2024-08-06 11:39:35 +02:00
Ettore Di Giacinto	bd900945f7	fix(llama.cpp): do not set anymore lora_base (#2999) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-07-24 12:35:52 +02:00
Ettore Di Giacinto	35561edb6e	feat(llama.cpp): support embeddings endpoints (#2871) ... * feat(llama.cpp): add embeddings Also enable embeddings by default for llama.cpp models Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(Makefile): prepare llama.cpp sources only once Otherwise we keep cloning llama.cpp for each of the variants Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * do not set embeddings to false Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add embeddings to the YAML config reference Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-07-15 22:54:16 +02:00
Dave	405794d4ca	fix: speedup git submodule update with --single-branch (#2847) ... add --single-branch to submodule update commands for speed Signed-off-by: Dave Lee <dave@gray101.com>	2024-07-13 22:32:25 +02:00
Loric	a00e9a82ae	Update remaining git clones to git fetch (#2779) ... Signed-off-by: Loric <117862619+LoricOSC@users.noreply.github.com>	2024-07-12 06:43:58 +00:00
cryptk	c047c19145	fix: make sure the GNUMake jobserver is passed to cmake for the llama.cpp build (#2697) ... Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>	2024-07-02 08:46:59 +02:00
Ettore Di Giacinto	7b1e792732	deps(llama.cpp): bump to latest, update build variables (#2669) ... * arrow_up: Update ggerganov/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * deps(llama.cpp): update build variables to follow upstream Update build recipes with https://github.com/ggerganov/llama.cpp/pull/8006 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Disable shared libs by default in llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Disable shared libs in llama.cpp Makefile Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Disable metal embedding for now, until it is tested Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(mac): explicitly enable metal Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * debug Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix typo Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>	2024-06-27 23:10:04 +02:00
Ettore Di Giacinto	a8bfb6f9c2	feat(options): add repeat_last_n (#2660) ... feat(options): add repeat_last_n Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-26 14:58:50 +02:00
Ettore Di Giacinto	b783c811db	feat(build): only build llama.cpp relevant targets (#2659) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-26 14:58:38 +02:00
Ettore Di Giacinto	3a9408363b	deps(llama.cpp): update and adapt API changes (#2381) ... deps(llama.cpp): update and rename function Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-05-23 01:02:11 +02:00
Ettore Di Giacinto	c89271b2e4	feat(llama.cpp): add distributed llama.cpp inferencing (#2324) ... * feat(llama.cpp): support distributed llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: let tweak how chat messages are merged together Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Makefile: register to ALL_GRPC_BACKENDS Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactoring, allow disable auto-detection of backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * minor fixups Signed-off-by: mudler <mudler@localai.io> * feat: add cmd to start rpc-server from llama.cpp Signed-off-by: mudler <mudler@localai.io> * ci: add ccache Signed-off-by: mudler <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: mudler <mudler@localai.io>	2024-05-15 01:17:02 +02:00
Ettore Di Giacinto	e49ea0123b	feat(llama.cpp): add flash_attention and no_kv_offloading (#2310) ... feat(llama.cpp): add flash_attn and no_kv_offload Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-05-13 19:07:51 +02:00
cryptk	28a421cb1d	feat: migrate python backends from conda to uv (#2215) ... * feat: migrate diffusers backend from conda to uv - replace conda with UV for diffusers install (prototype for all extras backends) - add ability to build docker with one/some/all extras backends instead of all or nothing Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate autogtpq bark coqui from conda to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: convert exllama over to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate exllama2 to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate mamba to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate parler to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate petals to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: fix tests Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate rerankers to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate sentencetransformers to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: install uv for tests-linux Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: make sure file exists before installing on intel images Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate transformers backend to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate transformers-musicgen to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate vall-e-x to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: migrate vllm to uv Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: add uv install to the rest of test-extra.yml Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: adjust file perms on all install/run/test scripts Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: add missing acclerate dependencies Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: add some more missing dependencies to python backends Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: parler tests venv py dir fix Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: correct filename for transformers-musicgen tests Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: adjust the pwd for valle tests Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: cleanup and optimization work for uv migration Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: add setuptools to requirements-install for mamba Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: more size optimization work Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * feat: make installs and tests more consistent, cleanup some deps Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: cleanup Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: mamba backend is cublas only Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> * fix: uncomment lines in makefile Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com> --------- Signed-off-by: Chris Jowett <421501+cryptk@users.noreply.github.com>	2024-05-10 15:08:08 +02:00
Ettore Di Giacinto	530bec9c64	feat(llama.cpp): do not specify backends to autoload and add llama.cpp variants (#2232) ... * feat(initializer): do not specify backends to autoload We can simply try to autoload the backends extracted in the asset dir. This will allow to build variants of the same backend (for e.g. with different instructions sets), so to have a single binary for all the variants. Signed-off-by: mudler <mudler@localai.io> * refactor(prepare): refactor out llama.cpp prepare steps Make it so are idempotent and that we can re-build Signed-off-by: mudler <mudler@localai.io> * [TEST] feat(build): build noavx version along Signed-off-by: mudler <mudler@localai.io> * build: make build parallel Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build: do not override CMAKE_ARGS Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build: add fallback variant Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(huggingface-langchain): fail if no token is set Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(huggingface-langchain): rename Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: do not autoload local-store Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: give priority between the listed backends Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: mudler <mudler@localai.io> Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-05-04 17:56:12 +02:00
Ettore Di Giacinto	e843d7df0e	feat(grpc): return consumed token count and update response accordingly (#2035) ... Fixes: #1920	2024-04-15 19:47:11 +02:00