docs: Add llava, update hot topics (#1322)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto 2023-11-23 18:54:55 +01:00 committed by GitHub
parent 69f53211a1
commit ba5ab26f2e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 47 additions and 33 deletions

View File

@ -22,7 +22,7 @@
> :bulb: Get help - [❓FAQ](https://localai.io/faq/) [💭Discussions](https://github.com/go-skynet/LocalAI/discussions) [:speech_balloon: Discord](https://discord.gg/uJAeKSAGDy) [:book: Documentation website](https://localai.io/)
>
> [💻 Quickstart](https://localai.io/basics/getting_started/) [📣 News](https://localai.io/basics/news/) [ 🛫 Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ 🖼️ Models ](https://localai.io/models/)
> [💻 Quickstart](https://localai.io/basics/getting_started/) [📣 News](https://localai.io/basics/news/) [ 🛫 Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ 🖼️ Models ](https://localai.io/models/) [ 🚀 Roadmap ](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
[![tests](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml)[![Build and Release](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml)[![build container images](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml)[![Bump dependencies](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml)[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/localai)](https://artifacthub.io/packages/search?repo=localai)
@ -79,7 +79,12 @@ LocalAI was created by [Ettore Di Giacinto](https://github.com/mudler/) and is a
Note that this started just as a [fun weekend project](https://localai.io/#backstory) in order to try to create the necessary pieces for a full AI assistant like `ChatGPT`: the community is growing fast and we are working hard to make it better and more stable. If you want to help, please consider contributing (see below)!
## 🔥🔥 [Hot topics / Roadmap](https://localai.io/#-hot-topics--roadmap)
## 🔥🔥 Hot topics / Roadmap
- [Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
Hot topics:
- https://github.com/mudler/LocalAI/issues/1126
## 🚀 [Features](https://localai.io/features/)
@ -91,7 +96,7 @@ Note that this started just as a [fun weekend project](https://localai.io/#backs
- 🧠 [Embeddings generation for vector databases](https://localai.io/features/embeddings/)
- ✍️ [Constrained grammars](https://localai.io/features/constrained_grammars/)
- 🖼️ [Download Models directly from Huggingface ](https://localai.io/models/)
- 🆕 [Vision API](https://localai.io/features/gpt-vision/)
## :book: 🎥 [Media, Blogs, Social](https://localai.io/basics/news/#media-blogs-social)

View File

@ -20,7 +20,7 @@ title = "LocalAI"
> 💡 Get help - [❓FAQ](https://localai.io/faq/) [❓How tos](https://localai.io/howtos/) [💭Discussions](https://github.com/go-skynet/LocalAI/discussions) [💭Discord](https://discord.gg/uJAeKSAGDy)
>
> [💻 Quickstart](https://localai.io/basics/getting_started/) [📣 News](https://localai.io/basics/news/) [ 🛫 Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ 🖼️ Models ](https://localai.io/models/)
> [💻 Quickstart](https://localai.io/basics/getting_started/) [📣 News](https://localai.io/basics/news/) [ 🛫 Examples ](https://github.com/go-skynet/LocalAI/tree/master/examples/) [ 🖼️ Models ](https://localai.io/models/) [ 🚀 Roadmap ](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
**LocalAI** is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format. Does not require GPU. It is maintained by [mudler](https://github.com/mudler).
@ -84,37 +84,15 @@ Note that this started just as a [fun weekend project](https://localai.io/#backs
- 🧠 [Embeddings generation for vector databases](https://localai.io/features/embeddings/)
- ✍️ [Constrained grammars](https://localai.io/features/constrained_grammars/)
- 🖼️ [Download Models directly from Huggingface ](https://localai.io/models/)
- 🆕 [Vision API](https://localai.io/features/gpt-vision/)
## 🔥🔥 Hot topics / Roadmap
- Backend rework for LocalAI v2 https://github.com/go-skynet/LocalAI/issues/1126
- Roadmap: https://github.com/go-skynet/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap
Some items:
- [ ] Enable gallery management directly from the webui. https://github.com/go-skynet/LocalAI/issues/918
- [x] llama.cpp lora adapters: https://github.com/go-skynet/LocalAI/issues/919
- [ ] image variants and edits: https://github.com/go-skynet/LocalAI/issues/921
- [ ] esrgan for diffusers: https://github.com/go-skynet/LocalAI/issues/917
- [ ] ggml-stablediffusion: https://github.com/go-skynet/LocalAI/issues/916
- [ ] SAM: https://github.com/go-skynet/LocalAI/issues/915
- [x] diffusers lora adapters: https://github.com/go-skynet/LocalAI/issues/914
- [ ] resource management and control: https://github.com/go-skynet/LocalAI/issues/912
- [ ] ChatGLM: https://github.com/go-skynet/LocalAI/issues/754
- [ ] text-to-video : https://github.com/go-skynet/LocalAI/issues/933
- [ ] rustformers: https://github.com/go-skynet/LocalAI/issues/939
- [x] Vall-e: https://github.com/go-skynet/LocalAI/issues/985
- [ ] Speculative sampling: https://github.com/go-skynet/LocalAI/issues/1013
- [ ] Falcon/GPTNeoX on llama.cpp: https://github.com/go-skynet/LocalAI/issues/1009
- [ ] transformers/vllm: https://github.com/go-skynet/LocalAI/issues/1015
- [ ] TortoiseTTS: https://github.com/go-skynet/LocalAI/issues/1016
- [ ] Exllama2: https://github.com/go-skynet/LocalAI/issues/1053
- [ ] ctransformers: https://github.com/go-skynet/LocalAI/issues/1056
- [ ] GPTQ for LLama: https://github.com/go-skynet/LocalAI/issues/1055
- [ ] LLaVA and miniGPT-4: https://github.com/go-skynet/LocalAI/issues/1054
- [ ] Test generation inference: https://github.com/go-skynet/LocalAI/issues/1042
- [ ] docs - extending LocalAI with external backends: https://github.com/go-skynet/LocalAI/issues/1057
- [Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
Hot topics:
- https://github.com/mudler/LocalAI/issues/1126
## How does it work?

View File

@ -13,4 +13,5 @@ This section contains the documentation for the features supported by LocalAI.
- [🎨 Image generation]({{%relref "features/image-generation" %}})
- [🧠 Embeddings]({{%relref "features/embeddings" %}})
- [🔥 OpenAI functions]({{%relref "features/openai-functions" %}})
- [🆕 GPT Vision API]({{%relref "features/gpt-vision" %}})
- [✍️ Constrained grammars]({{%relref "features/constrained_grammars" %}})

View File

@ -0,0 +1,30 @@
+++
disableToc = false
title = "🆕 GPT Vision"
weight = 2
+++
{{% notice note %}}
Available only on `master` builds
{{% /notice %}}
LocalAI supports understanding images by using [LLaVA](https://llava.hliu.cc/), and implements the [GPT Vision API](https://platform.openai.com/docs/guides/vision) from OpenAI.
![llava](https://github.com/mudler/LocalAI/assets/2420543/cb0a0897-3b58-4350-af66-e6f4387b58d3)
## Usage
OpenAI docs: https://platform.openai.com/docs/guides/vision
To let LocalAI understand and reply with what sees in the image, use the `/v1/chat/completions` endpoint, for example with curl:
```bash
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "llava",
"messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}'
```
### Setup
To setup the LLaVa models, follow the full example in the [configuration examples](https://github.com/mudler/LocalAI/blob/master/examples/configurations/README.md#llava).

View File

@ -1,7 +1,7 @@
+++
disableToc = false
title = "🆕 Vall-E-X"
title = "Vall-E-X"
weight = 4
+++

View File

@ -1,7 +1,7 @@
+++
disableToc = false
title = "🆕 vLLM"
title = "vLLM"
weight = 4
+++