LocalAI/examples/configurations
Ettore Di Giacinto bc8f648a91
fix(doc/examples): set defaults to mirostat ()
The default sampler on some models don't return enough candidates which
leads to a false sense of randomness. Tracing back the code it looks
that with the temperature sampler there might not be enough
candidates to pick from, and since the seed and "randomness" take effect
while picking a good candidate this yields to the same results over and
over.

Fixes https://github.com/mudler/LocalAI/issues/1723 by updating the
examples and documentation to use mirostat instead.
2024-03-11 19:49:03 +01:00
..
llava Delete examples/configurations/llava/README.md 2023-11-18 15:01:39 +01:00
mistral docs(examples): Add mistral example () 2023-10-25 20:56:12 +02:00
mixtral docs(mixtral): add mixtral example () 2023-12-16 17:44:43 +01:00
phi-2.yaml fix(doc/examples): set defaults to mirostat () 2024-03-11 19:49:03 +01:00
README.md docs(phi-2): add example () 2024-01-01 10:51:47 +01:00

Advanced configuration

This section contains examples on how to install models manually with config files.

Prerequisites

First clone LocalAI:

git clone https://github.com/go-skynet/LocalAI

cd LocalAI

Setup the model you prefer from the examples below and then start LocalAI:

docker compose up -d --pull always

If LocalAI is already started, you can restart it with

docker compose restart

See also the getting started: https://localai.io/basics/getting_started/

You can also start LocalAI just with docker:

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm quay.io/go-skynet/local-ai:master --models-path /models --threads 4

Mistral

To setup mistral copy the files inside mistral in the models folder:

cp -r examples/configurations/mistral/* models/

Now download the model:

wget https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF/resolve/main/mistral-7b-openorca.Q6_K.gguf -O models/mistral-7b-openorca.Q6_K.gguf

LLaVA

llava

Setup

cp -r examples/configurations/llava/* models/
wget https://huggingface.co/mys/ggml_bakllava-1/resolve/main/ggml-model-q4_k.gguf -O models/ggml-model-q4_k.gguf
wget https://huggingface.co/mys/ggml_bakllava-1/resolve/main/mmproj-model-f16.gguf -O models/mmproj-model-f16.gguf

Try it out

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "llava",
     "messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}'

Phi-2

cp -r examples/configurations/phi-2.yaml models/

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "phi-2",
     "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
}'

Mixtral

cp -r examples/configuration/mixtral/* models/
wget https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/resolve/main/mixtral-8x7b-instruct-v0.1.Q2_K.gguf -O models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf

Test it out

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "mixtral",
     "prompt": "How fast is light?",                                                                                    
     "temperature": 0.1 }'