LocalAI/embedded/models/mixtral-instruct.yaml
Ettore Di Giacinto bc8f648a91
fix(doc/examples): set defaults to mirostat ()
The default sampler on some models don't return enough candidates which
leads to a false sense of randomness. Tracing back the code it looks
that with the temperature sampler there might not be enough
candidates to pick from, and since the seed and "randomness" take effect
while picking a good candidate this yields to the same results over and
over.

Fixes https://github.com/mudler/LocalAI/issues/1723 by updating the
examples and documentation to use mirostat instead.
2024-03-11 19:49:03 +01:00

25 lines
564 B
YAML

name: mixtral-instruct
mmap: true
parameters:
model: huggingface://TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/mixtral-8x7b-instruct-v0.1.Q2_K.gguf
temperature: 0.2
top_k: 40
seed: -1
top_p: 0.95
mirostat: 2
mirostat_eta: 1.0
mirostat_tau: 1.0
template:
chat: &chat |
[INST] {{.Input}} [/INST]
completion: *chat
context_size: 4096
f16: true
gpu_layers: 90
usage: |
curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
"model": "mixtral-instruct",
"prompt": "How are you doing?"
}'