60 Commits

Author	SHA1	Message	Date
Ettore Di Giacinto	67f7bffd18	chore(deps): update llama.cpp and sync with upstream changes (#4950) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-06 00:40:58 +01:00
Ettore Di Giacinto	a7b4001b75	feat: allow to specify a reply prefix (#4931) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-03-02 16:07:32 +01:00
Brandon Beiler	6a6e1a0ea9	feat(vllm): Additional vLLM config options (Disable logging, dtype, and Per-Prompt media limits) (#4855) ... * Adding the following vLLM config options: disable_log_status, dtype, limit_mm_per_prompt Signed-off-by: TheDropZone <brandonbeiler@gmail.com> * using " marks in the config.yaml file Signed-off-by: TheDropZone <brandonbeiler@gmail.com> * adding in missing colon Signed-off-by: TheDropZone <brandonbeiler@gmail.com> --------- Signed-off-by: TheDropZone <brandonbeiler@gmail.com>	2025-02-18 19:27:58 +01:00
Dave	3cddf24747	feat: Centralized Request Processing middleware (#3847) ... * squash past, centralize request middleware PR Signed-off-by: Dave Lee <dave@gray101.com> * migrate bruno request files to examples repo Signed-off-by: Dave Lee <dave@gray101.com> * fix Signed-off-by: Dave Lee <dave@gray101.com> * Update tests/e2e-aio/e2e_test.go Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> --------- Signed-off-by: Dave Lee <dave@gray101.com> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2025-02-10 12:06:16 +01:00
Shraddha	03974a4dd4	feat: tokenization with llama.cpp (#4724) ... feat: tokenization Signed-off-by: shraddhazpy <shraddha@shraddhafive.in>	2025-02-02 17:39:43 +00:00
Ettore Di Giacinto	1d6afbd65d	feat(llama.cpp): Add support to grammar triggers (#4733) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-02-02 13:25:03 +01:00
mintyleaf	96f8ec0402	feat: add machine tag and inference timings (#4577) ... * Add machine tag option, add extraUsage option, grpc-server -> proto -> endpoint extraUsage data is broken for now Signed-off-by: mintyleaf <mintyleafdev@gmail.com> * remove redurant timing fields, fix not working timings output Signed-off-by: mintyleaf <mintyleafdev@gmail.com> * use middleware for Machine-Tag only if tag is specified Signed-off-by: mintyleaf <mintyleafdev@gmail.com> --------- Signed-off-by: mintyleaf <mintyleafdev@gmail.com>	2025-01-17 17:05:58 +01:00
Ettore Di Giacinto	7d0ac1ea3f	chore(vall-e-x): Drop backend (#4619) ... There are many new architectures that are SOTA and replaces vall-e-x nowadays. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-01-17 09:35:10 +01:00
mintyleaf	2bc4b56a79	feat: stream tokens usage (#4415) ... * Use pb.Reply instead of []byte with Reply.GetMessage() in llama grpc to get the proper usage data in reply streaming mode at the last [DONE] frame * Fix 'hang' on empty message from the start Seems like that empty message marker trick was unnecessary --------- Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2024-12-18 09:48:50 +01:00
Ettore Di Giacinto	f943c4b803	Revert "feat: include tokens usage for streamed output" (#4336) ... Revert "feat: include tokens usage for streamed output (#4282)" This reverts commit 0d6c3a7d57101428aec4100d0f7bca765ee684a7.	2024-12-08 17:53:36 +01:00
Ettore Di Giacinto	d4c1746c7d	feat(llama.cpp): expose cache_type_k and cache_type_v for quant of kv cache (#4329) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-06 10:23:59 +01:00
Ettore Di Giacinto	44a5dac312	feat(backend): add stablediffusion-ggml (#4289) ... * feat(backend): add stablediffusion-ggml Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(ci): track stablediffusion-ggml Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Use default scheduler and sampler if not specified Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Move cfg scale out of diffusers block Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Make it working Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix: set free_params_immediately to false to call the model in sequence https://github.com/leejet/stable-diffusion.cpp/issues/366 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-12-03 22:41:22 +01:00
mintyleaf	0d6c3a7d57	feat: include tokens usage for streamed output (#4282) ... Use pb.Reply instead of []byte with Reply.GetMessage() in llama grpc to get the proper usage data in reply streaming mode at the last [DONE] frame Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2024-11-28 14:47:56 +01:00
Ettore Di Giacinto	6daef00d30	chore(refactor): drop unnecessary code in loader (#4096) ... * chore: simplify passing options to ModelOptions Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(refactor): do not expose internal backend Loader Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-08 21:54:25 +01:00
Ettore Di Giacinto	e2a8dd64db	fix(tts): correctly pass backend config when generating model options (#4091) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-07 18:30:22 +01:00
Ettore Di Giacinto	947224b952	feat(diffusers): allow multiple lora adapters (#4081) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-11-05 15:14:33 +01:00
Ettore Di Giacinto	ae1ec4e096	feat(vllm): expose 'load_format' (#3943) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-23 15:34:57 +02:00
Ettore Di Giacinto	d5da8c3509	feat(templates): extract text from multimodal requests (#3866) ... When offloading template construction to the backend, we want to keep text around in case of multimodal requests. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-17 17:33:50 +02:00
Ettore Di Giacinto	3acd767ac4	chore: simplify model loading (#3715) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-10-02 08:59:06 +02:00
Shraddha	5488fc3bc1	feat: tokenization endpoint (#3710) ... endpoint to access the tokenizer Signed-off-by: shraddhazpy <shraddha@shraddhafive.in> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Co-authored-by: Dave <dave@gray101.com>	2024-10-02 08:56:18 +02:00
Ettore Di Giacinto	0965c6cd68	feat: track internally started models by ID (#3693) ... * chore(refactor): track internally started models by ID Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Just extend options, no need to copy Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Improve debugging for rerankers failures Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Simplify model loading with rerankers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Be more consistent when generating model options Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Uncommitted code Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Make deleteProcess more idiomatic Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Adapt CLI for sound generation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixup threads definition Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Handle corner case where c.Seed is nil Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Consistently use ModelOptions Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Adapt new code to refactoring Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Dave <dave@gray101.com>	2024-10-02 08:55:58 +02:00
siddimore	f84b55d1ef	feat: Add Get Token Metrics to GRPC server (#3687) ... * Add Get Token Metrics to GRPC server Signed-off-by: Siddharth More <siddimore@gmail.com> * Expose LocalAI endpoint Signed-off-by: Siddharth More <siddimore@gmail.com> --------- Signed-off-by: Siddharth More <siddimore@gmail.com>	2024-10-01 14:41:20 +02:00
Sertaç Özercan	ee21b00a8d	feat: auto load into memory on startup (#3627) ... Signed-off-by: Sertac Ozercan <sozercan@gmail.com>	2024-09-22 10:03:30 +02:00
Ettore Di Giacinto	191bc2e50a	feat(api): allow to pass audios to backends (#3603) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-09-19 12:26:53 +02:00
Ettore Di Giacinto	fbb9facda4	feat(api): allow to pass videos to backends (#3601) ... This prepares the API to receive videos as well for video understanding. It works similarly to images, where the request should be in the form: { "type": "video_url", "video_url": { "url": "url or base64 data" } } Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-09-19 11:21:59 +02:00
Ettore Di Giacinto	cf747bcdec	feat: extract output with regexes from LLMs (#3491) ... * feat: extract output with regexes from LLMs This changset adds `extract_regex` to the LLM config. It is a list of regexes that can match output and will be used to re extract text from the LLM output. This is particularly useful for LLMs which outputs final results into tags. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add tests, enhance output in case of configuration error Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-09-13 13:27:36 +02:00
Dave	300f2779e4	fix: speedup and improve cachability of docker build of builder-sd (#3430) ... fix: speedup and improve cachability of docker build of `builder-sd` (#3430) --------- Signed-off-by: Dave Lee <dave@gray101.com>	2024-09-10 08:57:16 +02:00
Dave	c2804c42fe	fix: untangle pkg/grpc and core/schema for Transcription (#3419) ... untangle pkg/grpc and core/schema in Transcribe Signed-off-by: Dave Lee <dave@gray101.com>	2024-09-02 15:48:53 +02:00
Ettore Di Giacinto	da3bc8077d	fix(tts): check error before inspecting result (#3415) ... Otherwise we panic when a res is nil Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-08-27 17:35:16 +02:00
Dave	81ae92f017	feat: elevenlabs sound-generation api (#3355) ... * initial version of elevenlabs compatible soundgeneration api and cli command Signed-off-by: Dave Lee <dave@gray101.com> * minor cleanup Signed-off-by: Dave Lee <dave@gray101.com> * restore TTS, add test Signed-off-by: Dave Lee <dave@gray101.com> * remove stray s Signed-off-by: Dave Lee <dave@gray101.com> * fix Signed-off-by: Dave Lee <dave@gray101.com> --------- Signed-off-by: Dave Lee <dave@gray101.com> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2024-08-24 00:20:28 +00:00
Ettore Di Giacinto	35561edb6e	feat(llama.cpp): support embeddings endpoints (#2871) ... * feat(llama.cpp): add embeddings Also enable embeddings by default for llama.cpp models Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(Makefile): prepare llama.cpp sources only once Otherwise we keep cloning llama.cpp for each of the variants Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * do not set embeddings to false Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add embeddings to the YAML config reference Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-07-15 22:54:16 +02:00
Dave	133987b1fb	feat: HF /scan endpoint (#2566) ... * start by checking /scan during the checksum update Signed-off-by: Dave Lee <dave@gray101.com> * add back in golang side features: downloader/uri gets struct and scan function, gallery uses it, and secscan/models calls it. Signed-off-by: Dave Lee <dave@gray101.com> * add a param to scan specific urls - useful for debugging Signed-off-by: Dave Lee <dave@gray101.com> * helpful printouts Signed-off-by: Dave Lee <dave@gray101.com> * fix offsets Signed-off-by: Dave Lee <dave@gray101.com> * fix error and naming Signed-off-by: Dave Lee <dave@gray101.com> * expose error Signed-off-by: Dave Lee <dave@gray101.com> * fix json tags Signed-off-by: Dave Lee <dave@gray101.com> * slight wording change Signed-off-by: Dave Lee <dave@gray101.com> * go mod tidy - getting warnings Signed-off-by: Dave Lee <dave@gray101.com> * split out python to make editing easier, add some simple code to delete contaminated entries from gallery Signed-off-by: Dave Lee <dave@gray101.com> * o7 to my favorite part of our old name, go-skynet Signed-off-by: Dave Lee <dave@gray101.com> * merge fix Signed-off-by: Dave Lee <dave@gray101.com> * merge fix Signed-off-by: Dave Lee <dave@gray101.com> * merge fix Signed-off-by: Dave Lee <dave@gray101.com> * address review comments Signed-off-by: Dave Lee <dave@gray101.com> * forgot secscan could accept multiple URL at once Signed-off-by: Dave Lee <dave@gray101.com> * invert naming and actually use it Signed-off-by: Dave Lee <dave@gray101.com> * missed cli/models.go Signed-off-by: Dave Lee <dave@gray101.com> * Update .github/check_and_update.py Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Signed-off-by: Dave <dave@gray101.com> --------- Signed-off-by: Dave Lee <dave@gray101.com> Signed-off-by: Dave <dave@gray101.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2024-07-10 13:18:32 +02:00
Ettore Di Giacinto	a8bfb6f9c2	feat(options): add repeat_last_n (#2660) ... feat(options): add repeat_last_n Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-26 14:58:50 +02:00
Ettore Di Giacinto	03b1cf51fd	feat(whisper): add translate option (#2649) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-24 19:21:22 +02:00
Ettore Di Giacinto	a181dd0ebc	refactor: gallery inconsistencies (#2647) ... * refactor(gallery): move under core/ Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(unarchive): do not allow symlinks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-24 17:32:12 +02:00
Sertaç Özercan	5866fc8ded	chore: fix go.mod module (#2635) ... Signed-off-by: Sertac Ozercan <sozercan@gmail.com>	2024-06-23 08:24:36 +00:00
Ettore Di Giacinto	7b205510f9	feat(gallery): uniform download from CLI (#2559) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-06-13 16:12:46 +02:00
Chakib Benziane	b99182c8d4	TTS API improvements (#2308) ... * update doc on COQUI_LANGUAGE env variable Signed-off-by: blob42 <contact@blob42.xyz> * return errors from tts gRPC backend Signed-off-by: blob42 <contact@blob42.xyz> * handle speaker_id and language in coqui TTS backend Signed-off-by: blob42 <contact@blob42.xyz> * TTS endpoint: add optional language paramter Signed-off-by: blob42 <contact@blob42.xyz> * tts fix: empty language string breaks non-multilingual models Signed-off-by: blob42 <contact@blob42.xyz> * allow tts param definition in config file - consolidate TTS options under `tts` config entry Signed-off-by: blob42 <contact@blob42.xyz> * tts: update doc Signed-off-by: blob42 <contact@blob42.xyz> --------- Signed-off-by: blob42 <contact@blob42.xyz> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2024-06-01 18:26:27 +00:00
Ettore Di Giacinto	e49ea0123b	feat(llama.cpp): add flash_attention and no_kv_offloading (#2310) ... feat(llama.cpp): add flash_attn and no_kv_offload Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-05-13 19:07:51 +02:00
Dave	c4f958e11b	refactor(application): introduce application global state (#2072) ... * start breaking up the giant channel refactor now that it's better understood - easier to merge bites Signed-off-by: Dave Lee <dave@gray101.com> * add concurrency and base64 back in, along with new base64 tests. Signed-off-by: Dave Lee <dave@gray101.com> * Automatic rename of whisper.go's Result to TranscriptResult Signed-off-by: Dave Lee <dave@gray101.com> * remove pkg/concurrency - significant changes coming in split 2 Signed-off-by: Dave Lee <dave@gray101.com> * fix comments Signed-off-by: Dave Lee <dave@gray101.com> * add list_model service as another low-risk service to get it out of the way Signed-off-by: Dave Lee <dave@gray101.com> * split backend config loader into seperate file from the actual config struct. No changes yet, just reduce cognative load with smaller files of logical blocks Signed-off-by: Dave Lee <dave@gray101.com> * rename state.go ==> application.go Signed-off-by: Dave Lee <dave@gray101.com> * fix lost import? Signed-off-by: Dave Lee <dave@gray101.com> --------- Signed-off-by: Dave Lee <dave@gray101.com>	2024-04-29 17:42:37 +00:00
Dave	2cd4936c99	fix: security scanner warning noise: error handlers part 1 (#2141) ... first group of error handlers to reduce security scanner warning noise level Signed-off-by: Dave Lee <dave@gray101.com>	2024-04-26 10:34:31 +02:00
Dave	c8dd8e5ef4	fix: reduce chmod permissions for created files and directories (#2137) ... quiet more security scanner issues: pass one of chmod restriction to remove group and other permissions Signed-off-by: Dave Lee <dave@gray101.com>	2024-04-26 00:47:06 +02:00
Ettore Di Giacinto	b664edde29	feat(rerankers): Add new backend, support jina rerankers API (#2121) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-04-25 00:19:02 +02:00
Taikono-Himazin	03adc1f60d	Add tensor_parallel_size setting to vllm setting items (#2085) ... Signed-off-by: Taikono-Himazin <kazu@po.harenet.ne.jp>	2024-04-20 14:37:02 +00:00
Ettore Di Giacinto	af9e5a2d05	Revert #1963 (#2056) ... * Revert "fix(fncall): fix regression introduced in #1963 (#2048)" This reverts commit 6b06d4e0af4db7a8aa8e131ec2b3af171934862e. * Revert "fix: action-tmate back to upstream, dead code removal (#2038)" This reverts commit fdec8a9d00a034ccd8e075008edd165147edf328. * Revert "feat(grpc): return consumed token count and update response accordingly (#2035)" This reverts commit e843d7df0e8b177ab122a9f7bfa7196274ccd204. * Revert "refactor: backend/service split, channel-based llm flow (#1963)" This reverts commit eed5706994a3e770a0194cad9d1cfd724ba1b10a. * feat(grpc): return consumed token count and update response accordingly Fixes: #1920 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-04-17 23:33:49 +02:00
Ettore Di Giacinto	e843d7df0e	feat(grpc): return consumed token count and update response accordingly (#2035) ... Fixes: #1920	2024-04-15 19:47:11 +02:00
Dave	eed5706994	refactor: backend/service split, channel-based llm flow (#1963) ... Refactor: channel based llm flow and services split --------- Signed-off-by: Dave Lee <dave@gray101.com>	2024-04-13 09:45:34 +02:00
Ludovic Leroux	12c0d9443e	feat: use tokenizer.apply_chat_template() in vLLM (#1990) ... Use tokenizer.apply_chat_template() in vLLM Signed-off-by: Ludovic LEROUX <ludovic@inpher.io>	2024-04-11 19:20:22 +02:00
Ettore Di Giacinto	8342553214	fix(llama.cpp): set better defaults for llama.cpp (#1961) ... fix(defaults): set better defaults for llama.cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-04-06 22:56:45 +02:00
Ettore Di Giacinto	ff77d3bc22	fix(seed): generate random seed per-request if -1 is set (#1952) ... * fix(seed): generate random seed per-request if -1 is set Also update ci with new workflows and allow the aio tests to run with an api key Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(openvino): Add OpenVINO example Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2024-04-03 22:25:47 +02:00