diff --git a/README.md b/README.md index 69c74acb..dbda44de 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ > :bulb: Get help - [❓FAQ](https://localai.io/faq/) [πŸ’­Discussions](https://github.com/go-skynet/LocalAI/discussions) [:speech_balloon: Discord](https://discord.gg/uJAeKSAGDy) [:book: Documentation website](https://localai.io/) > -> [πŸ’» Quickstart](https://localai.io/basics/getting_started/) [πŸ“£ News](https://localai.io/basics/news/) [ πŸ›« Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ πŸ–ΌοΈ Models ](https://localai.io/models/) +> [πŸ’» Quickstart](https://localai.io/basics/getting_started/) [πŸ“£ News](https://localai.io/basics/news/) [ πŸ›« Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ πŸ–ΌοΈ Models ](https://localai.io/models/) [ πŸš€ Roadmap ](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap) [![tests](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml)[![Build and Release](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml)[![build container images](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml)[![Bump dependencies](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml)[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/localai)](https://artifacthub.io/packages/search?repo=localai) @@ -79,7 +79,12 @@ LocalAI was created by [Ettore Di Giacinto](https://github.com/mudler/) and is a Note that this started just as a [fun weekend project](https://localai.io/#backstory) in order to try to create the necessary pieces for a full AI assistant like `ChatGPT`: the community is growing fast and we are working hard to make it better and more stable. If you want to help, please consider contributing (see below)! -## πŸ”₯πŸ”₯ [Hot topics / Roadmap](https://localai.io/#-hot-topics--roadmap) +## πŸ”₯πŸ”₯ Hot topics / Roadmap + +- [Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap) + +Hot topics: +- https://github.com/mudler/LocalAI/issues/1126 ## πŸš€ [Features](https://localai.io/features/) @@ -91,7 +96,7 @@ Note that this started just as a [fun weekend project](https://localai.io/#backs - 🧠 [Embeddings generation for vector databases](https://localai.io/features/embeddings/) - ✍️ [Constrained grammars](https://localai.io/features/constrained_grammars/) - πŸ–ΌοΈ [Download Models directly from Huggingface ](https://localai.io/models/) - +- πŸ†• [Vision API](https://localai.io/features/gpt-vision/) ## :book: πŸŽ₯ [Media, Blogs, Social](https://localai.io/basics/news/#media-blogs-social) diff --git a/docs/content/_index.en.md b/docs/content/_index.en.md index a2ced5b3..121e4782 100644 --- a/docs/content/_index.en.md +++ b/docs/content/_index.en.md @@ -20,7 +20,7 @@ title = "LocalAI" > πŸ’‘ Get help - [❓FAQ](https://localai.io/faq/) [❓How tos](https://localai.io/howtos/) [πŸ’­Discussions](https://github.com/go-skynet/LocalAI/discussions) [πŸ’­Discord](https://discord.gg/uJAeKSAGDy) > -> [πŸ’» Quickstart](https://localai.io/basics/getting_started/) [πŸ“£ News](https://localai.io/basics/news/) [ πŸ›« Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ πŸ–ΌοΈ Models ](https://localai.io/models/) +> [πŸ’» Quickstart](https://localai.io/basics/getting_started/) [πŸ“£ News](https://localai.io/basics/news/) [ πŸ›« Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ πŸ–ΌοΈ Models ](https://localai.io/models/) [ πŸš€ Roadmap ](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap) **LocalAI** is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format. Does not require GPU. It is maintained by [mudler](https://github.com/mudler). @@ -84,37 +84,15 @@ Note that this started just as a [fun weekend project](https://localai.io/#backs - 🧠 [Embeddings generation for vector databases](https://localai.io/features/embeddings/) - ✍️ [Constrained grammars](https://localai.io/features/constrained_grammars/) - πŸ–ΌοΈ [Download Models directly from Huggingface ](https://localai.io/models/) +- πŸ†• [Vision API](https://localai.io/features/gpt-vision/) + ## πŸ”₯πŸ”₯ Hot topics / Roadmap -- Backend rework for LocalAI v2 https://github.com/go-skynet/LocalAI/issues/1126 -- Roadmap: https://github.com/go-skynet/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap - -Some items: - -- [ ] Enable gallery management directly from the webui. https://github.com/go-skynet/LocalAI/issues/918 -- [x] llama.cpp lora adapters: https://github.com/go-skynet/LocalAI/issues/919 -- [ ] image variants and edits: https://github.com/go-skynet/LocalAI/issues/921 -- [ ] esrgan for diffusers: https://github.com/go-skynet/LocalAI/issues/917 -- [ ] ggml-stablediffusion: https://github.com/go-skynet/LocalAI/issues/916 -- [ ] SAM: https://github.com/go-skynet/LocalAI/issues/915 -- [x] diffusers lora adapters: https://github.com/go-skynet/LocalAI/issues/914 -- [ ] resource management and control: https://github.com/go-skynet/LocalAI/issues/912 -- [ ] ChatGLM: https://github.com/go-skynet/LocalAI/issues/754 -- [ ] text-to-video : https://github.com/go-skynet/LocalAI/issues/933 -- [ ] rustformers: https://github.com/go-skynet/LocalAI/issues/939 -- [x] Vall-e: https://github.com/go-skynet/LocalAI/issues/985 -- [ ] Speculative sampling: https://github.com/go-skynet/LocalAI/issues/1013 -- [ ] Falcon/GPTNeoX on llama.cpp: https://github.com/go-skynet/LocalAI/issues/1009 -- [ ] transformers/vllm: https://github.com/go-skynet/LocalAI/issues/1015 -- [ ] TortoiseTTS: https://github.com/go-skynet/LocalAI/issues/1016 -- [ ] Exllama2: https://github.com/go-skynet/LocalAI/issues/1053 -- [ ] ctransformers: https://github.com/go-skynet/LocalAI/issues/1056 -- [ ] GPTQ for LLama: https://github.com/go-skynet/LocalAI/issues/1055 -- [ ] LLaVA and miniGPT-4: https://github.com/go-skynet/LocalAI/issues/1054 -- [ ] Test generation inference: https://github.com/go-skynet/LocalAI/issues/1042 -- [ ] docs - extending LocalAI with external backends: https://github.com/go-skynet/LocalAI/issues/1057 +- [Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap) +Hot topics: +- https://github.com/mudler/LocalAI/issues/1126 ## How does it work? diff --git a/docs/content/features/_index.en.md b/docs/content/features/_index.en.md index 173050d4..a0f4dc5d 100644 --- a/docs/content/features/_index.en.md +++ b/docs/content/features/_index.en.md @@ -13,4 +13,5 @@ This section contains the documentation for the features supported by LocalAI. - [🎨 Image generation]({{%relref "features/image-generation" %}}) - [🧠 Embeddings]({{%relref "features/embeddings" %}}) - [πŸ”₯ OpenAI functions]({{%relref "features/openai-functions" %}}) +- [πŸ†• GPT Vision API]({{%relref "features/gpt-vision" %}}) - [✍️ Constrained grammars]({{%relref "features/constrained_grammars" %}}) diff --git a/docs/content/features/gpt-vision.md b/docs/content/features/gpt-vision.md new file mode 100644 index 00000000..3b37fe8f --- /dev/null +++ b/docs/content/features/gpt-vision.md @@ -0,0 +1,30 @@ + ++++ +disableToc = false +title = "πŸ†• GPT Vision" +weight = 2 ++++ + +{{% notice note %}} +Available only on `master` builds +{{% /notice %}} + +LocalAI supports understanding images by using [LLaVA](https://llava.hliu.cc/), and implements the [GPT Vision API](https://platform.openai.com/docs/guides/vision) from OpenAI. + +![llava](https://github.com/mudler/LocalAI/assets/2420543/cb0a0897-3b58-4350-af66-e6f4387b58d3) + +## Usage + +OpenAI docs: https://platform.openai.com/docs/guides/vision + +To let LocalAI understand and reply with what sees in the image, use the `/v1/chat/completions` endpoint, for example with curl: + +```bash +curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "llava", + "messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}' +``` + +### Setup + +To setup the LLaVa models, follow the full example in the [configuration examples](https://github.com/mudler/LocalAI/blob/master/examples/configurations/README.md#llava). \ No newline at end of file diff --git a/docs/content/model-compatibility/vall-e-x.md b/docs/content/model-compatibility/vall-e-x.md index dd514131..5184efe6 100644 --- a/docs/content/model-compatibility/vall-e-x.md +++ b/docs/content/model-compatibility/vall-e-x.md @@ -1,7 +1,7 @@ +++ disableToc = false -title = "πŸ†• Vall-E-X" +title = "Vall-E-X" weight = 4 +++ diff --git a/docs/content/model-compatibility/vllm.md b/docs/content/model-compatibility/vllm.md index ae58ea7c..56c876ee 100644 --- a/docs/content/model-compatibility/vllm.md +++ b/docs/content/model-compatibility/vllm.md @@ -1,7 +1,7 @@ +++ disableToc = false -title = "πŸ†• vLLM" +title = "vLLM" weight = 4 +++