text-generation-webui

Mirrors/text-generation-webui

Fork 0

mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-12-04 04:50:30 +01:00

Table of Contents

OpenAI compatible API

Starting the API
Examples

Completions
Chat completions
Chat completions with characters
SSE streaming
Logits
Logits after sampling parameters
Python chat example
Python chat example with streaming
Python completions example with streaming

Environment variables

Persistent settings with settings.yaml

Third-party application setup
Embeddings (alpha)
Compatibility & not so compatibility

Applications

OpenAI compatible API

The main API for this project is meant to be a drop-in replacement to the OpenAI API, including Chat and Completions endpoints.

It is 100% offline and private.
It doesn't create any logs.
It doesn't connect to OpenAI.
It doesn't use the openai-python library.

Starting the API

Add --api to your command-line flags.

To create a public Cloudflare URL, add the --public-api flag.
To listen on your local network, add the --listen flag.
To change the port, which is 5000 by default, use --api-port 1234 (change 1234 to your desired port number).
To use SSL, add --ssl-keyfile key.pem --ssl-certfile cert.pem. Note that it doesn't work with --public-api.
To use an API key for authentication, add --api-key yourkey.

Examples

For the documentation with all the parameters and their types, consult http://127.0.0.1:5000/docs or the typing.py file.

The official examples in the OpenAI documentation should also work, and the same parameters apply (although the API here has more optional parameters).

Completions

curl http://127.0.0.1:5000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "This is a cake recipe:\n\n1.",
    "max_tokens": 200,
    "temperature": 1,
    "top_p": 0.9,
    "seed": 10
  }'

Chat completions

Works best with instruction-following models. If the "instruction_template" variable is not provided, it will be guessed automatically based on the model name using the regex patterns in models/config.yaml.

curl http://127.0.0.1:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ],
    "mode": "instruct",
    "instruction_template": "Alpaca"
  }'

Chat completions with characters

curl http://127.0.0.1:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Hello! Who are you?"
      }
    ],
    "mode": "chat",
    "character": "Example"
  }'

SSE streaming

curl http://127.0.0.1:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ],
    "mode": "instruct",
    "instruction_template": "Alpaca",
    "stream": true
  }'

Logits

curl -k http://127.0.0.1:5000/v1/internal/logits \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Who is best, Asuka or Rei? Answer:",
    "use_samplers": false
  }'

Logits after sampling parameters

curl -k http://127.0.0.1:5000/v1/internal/logits \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Who is best, Asuka or Rei? Answer:",
    "use_samplers": true,
    "top_k": 3
  }'

Python chat example

import requests

url = "http://127.0.0.1:5000/v1/chat/completions"

headers = {
    "Content-Type": "application/json"
}

history = []

while True:
    user_message = input("> ")
    history.append({"role": "user", "content": user_message})
    data = {
        "mode": "chat",
        "character": "Example",
        "messages": history
    }

    response = requests.post(url, headers=headers, json=data, verify=False)
    assistant_message = response.json()['choices'][0]['message']['content']
    history.append({"role": "assistant", "content": assistant_message})
    print(assistant_message)

Python chat example with streaming

Start the script with python -u to see the output in real time.

import requests
import sseclient  # pip install sseclient-py
import json

url = "http://127.0.0.1:5000/v1/chat/completions"

headers = {
    "Content-Type": "application/json"
}

history = []

while True:
    user_message = input("> ")
    history.append({"role": "user", "content": user_message})
    data = {
        "mode": "instruct",
        "stream": True,
        "messages": history
    }

    stream_response = requests.post(url, headers=headers, json=data, verify=False, stream=True)
    client = sseclient.SSEClient(stream_response)

    assistant_message = ''
    for event in client.events():
        payload = json.loads(event.data)
        chunk = payload['choices'][0]['message']['content']
        assistant_message += chunk
        print(chunk, end='')

    print()
    history.append({"role": "assistant", "content": assistant_message})

Python completions example with streaming

Start the script with python -u to see the output in real time.

import json
import requests
import sseclient  # pip install sseclient-py

url = "http://127.0.0.1:5000/v1/completions"

headers = {
    "Content-Type": "application/json"
}

data = {
    "prompt": "This is a cake recipe:\n\n1.",
    "max_tokens": 200,
    "temperature": 1,
    "top_p": 0.9,
    "seed": 10,
    "stream": True,
}

stream_response = requests.post(url, headers=headers, json=data, verify=False, stream=True)
client = sseclient.SSEClient(stream_response)

print(data['prompt'], end='')
for event in client.events():
    payload = json.loads(event.data)
    print(payload['choices'][0]['text'], end='')

print()

Environment variables

The following environment variables can be used (they take precedence over everything else):

Variable Name	Description	Example Value
`OPENEDAI_PORT`	Port number	5000
`OPENEDAI_CERT_PATH`	SSL certificate file path	cert.pem
`OPENEDAI_KEY_PATH`	SSL key file path	key.pem
`OPENEDAI_DEBUG`	Enable debugging (set to 1)	1
`SD_WEBUI_URL`	WebUI URL (used by endpoint)	http://127.0.0.1:7861
`OPENEDAI_EMBEDDING_MODEL`	Embedding model (if applicable)	sentence-transformers/all-mpnet-base-v2
`OPENEDAI_EMBEDDING_DEVICE`	Embedding device (if applicable)	cuda

Persistent settings with `settings.yaml`

You can also set the following variables in your settings.yaml file:

openai-embedding_device: cuda
openai-embedding_model: "sentence-transformers/all-mpnet-base-v2"
openai-sd_webui_url: http://127.0.0.1:7861
openai-debug: 1

Third-party application setup

You can usually force an application that uses the OpenAI API to connect to the local API by using the following environment variables:

OPENAI_API_HOST=http://127.0.0.1:5000

OPENAI_API_KEY=sk-111111111111111111111111111111111111111111111111
OPENAI_API_BASE=http://127.0.0.1:5000/v1

With the official python openai client, the address can be set like this:

import openai

openai.api_key = "..."
openai.api_base = "http://127.0.0.1:5000/v1"
openai.api_version = "2023-05-15"

If using .env files to save the OPENAI_API_BASE and OPENAI_API_KEY variables, make sure the .env file is loaded before the openai module is imported:

from dotenv import load_dotenv
load_dotenv() # make sure the environment variables are set before import
import openai

With the official Node.js openai client it is slightly more more complex because the environment variables are not used by default, so small source code changes may be required to use the environment variables, like so:

const openai = OpenAI(
  Configuration({
    apiKey: process.env.OPENAI_API_KEY,
    basePath: process.env.OPENAI_API_BASE
  })
);

For apps made with the chatgpt-api Node.js client library:

const api = new ChatGPTAPI({
  apiKey: process.env.OPENAI_API_KEY,
  apiBaseUrl: process.env.OPENAI_API_BASE
});

Embeddings (alpha)

Embeddings requires sentence-transformers installed, but chat and completions will function without it loaded. The embeddings endpoint is currently using the HuggingFace model: sentence-transformers/all-mpnet-base-v2 for embeddings. This produces 768 dimensional embeddings (the same as the text-davinci-002 embeddings), which is different from OpenAI's current default text-embedding-ada-002 model which produces 1536 dimensional embeddings. The model is small-ish and fast-ish. This model and embedding size may change in the future.

model name	dimensions	input max tokens	speed	size	Avg. performance
text-embedding-ada-002	1536	8192	-	-	-
text-davinci-002	768	2046	-	-	-
all-mpnet-base-v2	768	384	2800	420M	63.3
all-MiniLM-L6-v2	384	256	14200	80M	58.8

In short, the all-MiniLM-L6-v2 model is 5x faster, 5x smaller ram, 2x smaller storage, and still offers good quality. Stats from (https://www.sbert.net/docs/pretrained_models.html). To change the model from the default you can set the environment variable OPENEDAI_EMBEDDING_MODEL, ex. "OPENEDAI_EMBEDDING_MODEL=all-MiniLM-L6-v2".

Warning: You cannot mix embeddings from different models even if they have the same dimensions. They are not comparable.

Compatibility & not so compatibility

Note: the table below may be obsolete.

API endpoint	tested with	notes
/v1/chat/completions	openai.ChatCompletion.create()	Use it with instruction following models
/v1/embeddings	openai.Embedding.create()	Using SentenceTransformer embeddings
/v1/images/generations	openai.Image.create()	Bare bones, no model configuration, response_format='b64_json' only.
/v1/moderations	openai.Moderation.create()	Basic initial support via embeddings
/v1/models	openai.Model.list()	Lists models, Currently loaded model first, plus some compatibility options
/v1/models/{id}	openai.Model.get()	returns whatever you ask for
/v1/edits	openai.Edit.create()	Removed, use /v1/chat/completions instead
/v1/text_completion	openai.Completion.create()	Legacy endpoint, variable quality based on the model
/v1/completions	openai api completions.create	Legacy endpoint (v0.25)
/v1/engines/*/embeddings	python-openai v0.25	Legacy endpoint
/v1/engines/*/generate	openai engines.generate	Legacy endpoint
/v1/engines	openai engines.list	Legacy Lists models
/v1/engines/{model_name}	openai engines.get -i {model_name}	You can use this legacy endpoint to load models via the api or command line
/v1/images/edits	openai.Image.create_edit()	not yet supported
/v1/images/variations	openai.Image.create_variation()	not yet supported
/v1/audio/*	openai.Audio.*	supported
/v1/files*	openai.Files.*	not yet supported
/v1/fine-tunes*	openai.FineTune.*	not yet supported
/v1/search	openai.search, engines.search	not yet supported

Applications

Almost everything needs the OPENAI_API_KEY and OPENAI_API_BASE environment variable set, but there are some exceptions.