LiteLLM

17 August 202613 min read

LiteLLM is an LLM gateway and Python SDK that exposes many model providers through an OpenAI-compatible interface. It translates requests to provider-specific APIs while keeping request and response handling consistent across OpenAI, Anthropic, Azure OpenAI, Vertex AI, Hugging Face, Ollama, OpenRouter, and other providers.

It can be used directly as a Python library or deployed as a proxy server for centralized access to models. In practice, it is a "one API for AI" layer: apps talk to one OpenAI-shaped API, while LiteLLM handles provider routing, credentials, fallbacks, spend limits, and local/self-hosted model backends.

Features

• OpenAI-compatible APIs for chat, responses, text completion, embeddings, image generation/editing, audio transcription/speech, files, batches, fine-tuning, and model listing

• Supports many commercial, hosted open model, and local model providers

• Router support for retries, fallbacks, load balancing, model groups, and latency/cost-aware routing

• Proxy server mode for using LiteLLM as an LLM gateway

• Virtual keys, organizations/teams, rate limits, budgets, and spend tracking

• Observability callbacks for tools such as Langfuse, MLflow, Helicone, and Lunary

• Maps provider errors to OpenAI-compatible exception types

• Useful for replacing hardcoded provider SDKs with a single API contract

Python SDK

python

from litellm import completion

response = completion(

model="openai/gpt-5",

messages=[{"role": "user", "content": "Hello"}],

)

The provider is selected through the model prefix, for example openai/..., anthropic/..., azure/..., ollama/..., openrouter/..., vertex_ai/..., bedrock/..., groq/..., or together_ai/....

Supported API Endpoints

LiteLLM aims to expose the most useful OpenAI API surface while translating each call to the closest provider-native API.

Endpoint family	Common endpoints	Use for	Notes
Chat	`/v1/chat/completions`	General LLM calls, tool calling, structured output, streaming	Most mature and widely supported path. Usually the safest default for apps.
Responses	`/v1/responses`	Newer OpenAI-style unified text/multimodal calls	Useful when clients are built around the OpenAI Responses API. Provider support varies.
Legacy completions	`/v1/completions`	Prompt-in/text-out models and older clients	Prefer chat unless maintaining legacy code.
Embeddings	`/v1/embeddings`	Vector search, RAG, clustering, reranking pipelines	Maps to OpenAI, Cohere, Voyage, Bedrock, Vertex, Hugging Face, Ollama, and other embedding providers.
Images	`/v1/images/generations`, `/v1/images/edits`, `/v1/images/variations`	Image generation/editing	Depends heavily on backend model/provider capabilities.
Audio	`/v1/audio/transcriptions`, `/v1/audio/translations`, `/v1/audio/speech`	Speech-to-text, translation, text-to-speech	Often backed by OpenAI Whisper/TTS, Azure, or self-hosted audio services.
Models	`/v1/models`	Discover configured model aliases	In proxy mode, this usually returns configured `model_list` entries rather than every upstream model.
Files/Batches/Fine-tuning	`/v1/files`, `/v1/batches`, `/v1/fine_tuning/jobs`	Async jobs, batch inference, provider fine-tuning workflows	Support depends on provider; treat these as provider-specific even if the route is OpenAI-shaped.
Moderation	`/v1/moderations`	Safety classification	Useful when centralizing policy checks through the same gateway.
Health/Admin	LiteLLM proxy admin routes	Key management, spend tracking, health, callbacks	These are LiteLLM-specific and should usually be kept internal.

The important mental model: LiteLLM normalizes request/response shapes, but it does not make every backend support every feature. Tool calling, JSON schema output, context length, image input, audio, caching, and batch behavior still depend on the selected model/provider.

Beyond Text: AI Capability Map

LiteLLM is most famous for chat/completions, but the bigger idea is to make application code depend on a small set of AI capability interfaces instead of dozens of provider SDKs. Useful capability buckets:

Capability	What the app asks for	Example backends	Wiring idea
Chat/reasoning	Conversational text, coding, planning, extraction, tool calls	OpenAI, Anthropic, Gemini, Bedrock, Groq, Together, vLLM, Ollama, llama.cpp	Expose aliases like `chat-fast`, `chat-smart`, `chat-local`, `code`, `reasoning`.
Vision understanding	Image question-answering, OCR-ish extraction, screenshot/UI analysis, document images	GPT-4o/4.1 vision, Gemini, Claude vision, LLaVA/Qwen-VL/Gemma 3 via local servers	Use chat/responses with multimodal message content; keep `vision-default` separate from text-only chat.
Embeddings	Turn text/images into vectors for search, memory, clustering, deduplication	OpenAI embeddings, Voyage, Cohere, Bedrock, Vertex, Ollama, Infinity, sentence-transformers	Expose stable `embed-text` and maybe `embed-code`; never swap embedding dimensions without rebuilding indexes.
Reranking	Reorder search/RAG candidates by relevance	Cohere Rerank, Voyage rerank, Jina, BGE rerankers, Infinity	Often sits beside LiteLLM rather than fully inside it; still give apps a central AI gateway interface.
Image generation/editing	Create/edit images from prompts, product mockups, backgrounds, avatars	OpenAI images, Stability/Stable Diffusion, ComfyUI, AUTOMATIC1111, Replicate, Fal.ai	Route hosted image APIs through LiteLLM where supported; use sidecar services for richer Stable Diffusion workflows.
Audio transcription	Speech-to-text, meeting notes, voice commands	OpenAI Whisper, Azure Speech, whisper.cpp, faster-whisper	Use `/audio/transcriptions` for OpenAI-compatible providers; keep local Whisper as a private service when needed.
Audio generation	Text-to-speech, voice UI, narration	OpenAI TTS, ElevenLabs, Azure, Piper, Chatterbox, CosyVoice, Dia	Some can go through OpenAI-compatible audio routes; advanced self-hosted TTS usually needs an adapter service.
Moderation/safety	Classify unsafe content, PII, policy checks	OpenAI moderation, Llama Guard, Detoxify, Presidio	Centralize before/after model calls; do not rely on every model provider having the same safety behavior.
Structured extraction	JSON, schemas, forms, records, entity extraction	Tool calling/function calling, JSON mode, grammar-constrained llama.cpp	Treat as a model capability; validate outputs in app code even when the model claims schema support.
Agents/tools	Model decides to call tools/APIs/functions	OpenAI tools, Anthropic tools, Gemini function calling, LangChain/LlamaIndex/Semantic Kernel	LiteLLM normalizes model access; the orchestration framework still owns tool execution and state.

A good architecture is to name capabilities, not vendors. Apps should ask for vision-default, embed-text, transcribe, image-gen, or chat-smart; LiteLLM and nearby adapter services decide which provider actually fulfills the request.

Model Naming

LiteLLM uses two related model names:

• Provider model ID: the actual upstream model or deployment, usually with a provider prefix such as openai/gpt-4o, anthropic/claude-3-5-sonnet-20241022, ollama/llama3.1, or hosted_vllm/meta-llama/Llama-3.1-70B-Instruct.

• Proxy model alias: the stable name your applications use, configured in model_list, such as fast-chat, cheap-chat, code, embeddings, or local-llama.

Prefer exposing aliases to applications. That lets you swap providers, add fallbacks, or move from hosted to self-hosted inference without touching application code.

Example naming pattern:

yaml

model_list:

- model_name: fast-chat

litellm_params:

model: groq/llama-3.1-70b-versatile

api_key: os.environ/GROQ_API_KEY

- model_name: local-chat

litellm_params:

model: ollama/llama3.1

api_base: http://localhost:11434

- model_name: embeddings

litellm_params:

model: openai/text-embedding-3-large

api_key: os.environ/OPENAI_API_KEY

Common Model/Provider Categories

Category	Example LiteLLM model IDs	Good fit
OpenAI	`openai/gpt-4o`, `openai/gpt-4o-mini`, `openai/o1`, `openai/text-embedding-3-large`	Strong general models, OpenAI-native API compatibility, embeddings, multimodal, image/audio APIs.
Anthropic	`anthropic/claude-3-5-sonnet-20241022`, `anthropic/claude-3-5-haiku-20241022`	Long-context reasoning, coding, writing, analysis, vision.
Azure OpenAI	`azure/<deployment-name>`	Enterprise OpenAI deployments with Azure networking, identity, and compliance controls.
Google Vertex/Gemini	`vertex_ai/gemini-1.5-pro`, `gemini/gemini-1.5-flash`	Google Cloud-native apps, long context, multimodal, video/image understanding.
AWS Bedrock	`bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0`, `bedrock/meta.llama3-1-70b-instruct-v1:0`	AWS-native access to Anthropic, Meta, Mistral, Amazon Nova/Titan, embeddings, guardrails.
Hosted open models	`groq/llama-3.1-70b-versatile`, `together_ai/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo`, `fireworks_ai/accounts/fireworks/models/llama-v3p1-70b-instruct`	Fast or cost-effective open-weight models without running GPUs.
OpenRouter	`openrouter/anthropic/claude-3.5-sonnet`, `openrouter/meta-llama/llama-3.1-405b-instruct`	Brokered access to many models behind one upstream provider.
Local/self-hosted	`ollama/llama3.1`, `hosted_vllm/meta-llama/Llama-3.1-70B-Instruct`, `text-completion-openai/<model>`	Privacy, local development, custom open-weight deployments, cost control.
Vision-local	`ollama/llava`, `ollama/qwen2.5vl`, llama.cpp multimodal server models	Local screenshot/image analysis where supported by backend.
Embeddings	`openai/text-embedding-3-small`, `cohere/embed-english-v3.0`, `voyage/voyage-3`, `ollama/nomic-embed-text`	RAG and semantic search.
Audio/image generation	Provider-specific OpenAI-compatible audio/image routes	Voice, transcription, TTS, image generation/editing, creative workflows.

Proxy Server

LiteLLM can run as a local or shared HTTP service. Clients use the regular OpenAI SDK and point base_url at the LiteLLM proxy:

python

from openai import OpenAI

client = OpenAI(

api_key="sk-example",

base_url="http://localhost:4000",

)

response = client.chat.completions.create(

model="gpt-5",

messages=[{"role": "user", "content": "Write a short poem"}],

)

A proxy deployment is useful when model access, authentication, cost tracking, and provider fallback should be managed centrally instead of embedded separately in every application.

Configuration

Proxy deployments are commonly configured with a litellm_config.yaml that maps logical model names to provider-specific deployments and credentials:

yaml

model_list:

- model_name: gpt-5

litellm_params:

model: azure/my-gpt-5-deployment

api_base: os.environ/AZURE_API_BASE

api_key: os.environ/AZURE_API_KEY

api_version: "2023-07-01-preview"

litellm_settings:

master_key: sk-example

Run the proxy with:

bash

litellm --config litellm_config.yaml --port 4000

Applications then call http://localhost:4000/v1/... with an OpenAI-compatible client and a LiteLLM virtual key.

Best Open Source Backends to Wire Behind LiteLLM

These are strong open-source implementations to pair with LiteLLM when you want local or self-hosted AI infrastructure.

Backend	Best for	LiteLLM wiring
Ollama	Easiest local models on laptops/workstations; quick dev setup; GGUF-style local serving.	Use `model: ollama/<model>` and `api_base: http://localhost:11434`.
vLLM	High-throughput OpenAI-compatible serving on NVIDIA/AMD GPUs; production inference for open-weight LLMs.	Run vLLM's OpenAI server and use `hosted_vllm/<hf-model>` or OpenAI-compatible `api_base`.
llama.cpp	CPU/GPU local inference, quantized GGUF models, edge devices, embeddings, and local multimodal vision models.	Run `llama-server` with OpenAI-compatible endpoints and point LiteLLM at its base URL.
SGLang	Fast serving for structured generation, agents, tool use, vision-language models, and complex decoding workflows.	Run its OpenAI-compatible server and expose it as a LiteLLM model alias.
Hugging Face TGI	Production Hugging Face model serving with streaming and tensor parallelism.	Use LiteLLM's Hugging Face/TGI-compatible provider settings or OpenAI-compatible route when enabled.
LocalAI	Drop-in local OpenAI-compatible API for LLMs, embeddings, images, and audio on commodity hardware.	Point LiteLLM to the LocalAI OpenAI-compatible `api_base`.
Open WebUI	User-facing chat UI for local or remote model gateways.	Put Open WebUI in front of LiteLLM by setting OpenAI API base URL to LiteLLM.
TabbyAPI	ExLlamaV2-based serving for quantized LLMs, often popular for local GPU inference.	Use its OpenAI-compatible endpoint as an upstream backend.
Infinity	Fast embedding and reranking model serving.	Wire as an OpenAI-compatible embeddings backend or provider-specific endpoint.
Xinference	Unified local serving for LLMs, embeddings, rerankers, image, audio, and multimodal models.	Point LiteLLM at its OpenAI-compatible endpoint.
whisper.cpp	Local speech-to-text/transcription using Whisper-style models.	Usually run as a sidecar service; expose through an adapter if you need OpenAI-compatible `/audio/transcriptions`.
ComfyUI	Node-based Stable Diffusion/image/video generation workflows.	Usually sits beside LiteLLM; call it from tools or an adapter service for image generation.
AUTOMATIC1111 Stable Diffusion WebUI	Popular local Stable Diffusion UI/API.	Use as an image sidecar behind an internal `image-gen` service.
Piper	Lightweight local text-to-speech; original `rhasspy/piper` repo is archived.	Use as a local TTS sidecar; wrap with an OpenAI-compatible speech endpoint if needed.

Wiring Patterns

1. App → LiteLLM → Hosted Providers

Use this when you want one internal API key and one client shape while still using OpenAI, Anthropic, Gemini, Bedrock, Groq, Together, etc.

yaml

model_list:

- model_name: general

litellm_params:

model: openai/gpt-4o-mini

api_key: os.environ/OPENAI_API_KEY

- model_name: reasoning

litellm_params:

model: anthropic/claude-3-5-sonnet-20241022

api_key: os.environ/ANTHROPIC_API_KEY

- model_name: cheap-open

litellm_params:

model: together_ai/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo

api_key: os.environ/TOGETHER_API_KEY

Application code stays provider-neutral:

python

from openai import OpenAI

client = OpenAI(api_key="sk-litellm-key", base_url="http://localhost:4000/v1")

response = client.chat.completions.create(

model="general",

messages=[{"role": "user", "content": "Summarize this design doc."}],

)

2. App → LiteLLM → Ollama

Good for local development and private experiments.

yaml

model_list:

- model_name: local-llama

litellm_params:

model: ollama/llama3.1

api_base: http://localhost:11434

- model_name: local-embeddings

litellm_params:

model: ollama/nomic-embed-text

api_base: http://localhost:11434

Then call model="local-llama" or model="local-embeddings" from any OpenAI-compatible client.

3. App → LiteLLM → vLLM/SGLang/llama.cpp

Good for self-hosted open-weight models on your own GPU machines.

yaml

model_list:

- model_name: selfhosted-chat

litellm_params:

model: hosted_vllm/meta-llama/Llama-3.1-70B-Instruct

api_base: http://vllm.internal:8000/v1

api_key: os.environ/VLLM_API_KEY

If the backend exposes a generic OpenAI-compatible API, the core idea is always the same:

1. Start the backend server with its OpenAI-compatible API enabled.

2. Add it to LiteLLM model_list with a stable alias.

3. Point apps at LiteLLM instead of the backend directly.

4. Add LiteLLM fallbacks, budgets, rate limits, and logging centrally.

4. Multimodal: App → LiteLLM → Vision Model

Vision is usually still a chat/responses call, but message content includes image parts.

python

from openai import OpenAI

client = OpenAI(api_key="sk-litellm-key", base_url="http://localhost:4000/v1")

response = client.chat.completions.create(

model="vision-default",

messages=[

{

"role": "user",

"content": [

{"type": "text", "text": "What is important in this screenshot?"},

{"type": "image_url", "image_url": {"url": "https://example.com/screenshot.png"}},

}

)

Suggested aliases:

yaml

model_list:

- model_name: vision-default

litellm_params:

model: openai/gpt-4o-mini

api_key: os.environ/OPENAI_API_KEY

- model_name: vision-local

litellm_params:

model: ollama/llava

api_base: http://localhost:11434

Keep vision aliases separate because many text models do not accept images, and local vision model quality varies widely.

5. Retrieval: App → LiteLLM Embeddings → Vector DB → Chat

For RAG, LiteLLM is useful for centralizing embedding models and chat models, but the vector database and retrieval logic still live in the application stack.

flowchart TD
    Docs[Documents]
    Embed[embed-text alias]
    VDB[Vector database]
    Query[User query]
    Retrieve[Retrieve and rerank]
    Chat[chat-smart alias]
    Answer[Grounded answer]

    Docs --> Embed --> VDB
    Query --> Embed
    Embed --> Retrieve
    VDB --> Retrieve
    Retrieve --> Chat --> Answer

Common companion projects:

• Vector databases: Qdrant, Weaviate, Milvus, Chroma, pgvector.

• Rerankers: Cohere Rerank, Voyage rerank, Jina rerankers, BGE rerankers through Infinity or sentence-transformers.

• Frameworks: LlamaIndex, LangChain, Haystack, DSPy.

6. Voice: App → Transcription → LiteLLM → Speech

A voice assistant usually needs three capabilities, not one model:

1. speech-to-text: /audio/transcriptions or a sidecar like whisper.cpp/faster-whisper;

2. reasoning/tool use: /chat/completions or /responses through LiteLLM;

3. text-to-speech: /audio/speech or a sidecar like Piper, Chatterbox, CosyVoice, Dia, or ElevenLabs.

flowchart TD
    Mic[Microphone audio]
    STT[Transcription]
    LLM[LiteLLM chat/tool model]
    TTS[Speech synthesis]
    Speaker[Audio output]

    Mic --> STT --> LLM --> TTS --> Speaker

LiteLLM can centralize the OpenAI-compatible parts, but local audio services often need thin adapter APIs.

7. Image/Video Generation: LiteLLM plus Creative Sidecars

Image APIs are less standardized than chat. For simple hosted image generation, use LiteLLM/OpenAI-compatible image routes where supported. For richer local workflows, wire specialized systems as sidecars:

• ComfyUI for node-based Stable Diffusion, ControlNet, LoRA, video/image pipelines.

• AUTOMATIC1111 for a broad local Stable Diffusion UI/API ecosystem.

• Replicate, Fal.ai, Stability, or provider APIs for hosted image/video jobs.

Pattern:

flowchart TD
    App[App]
    Gateway[AI gateway]
    LLM[LiteLLM for prompts/planning]
    Image[Image generation service]
    Store[Object storage]

    App --> Gateway
    Gateway --> LLM
    Gateway --> Image
    Image --> Store
    Store --> App

The LLM can plan or rewrite prompts, but the image service should own generation parameters, seeds, LoRAs, upscaling, and workflow graphs.

8. Agents and Tools: LiteLLM is the Model Layer

LiteLLM does not replace an agent runtime. It gives the runtime a consistent model API.

Typical layering:

• Agent runtime: LangChain, LlamaIndex, Semantic Kernel, CrewAI, AutoGen, custom loop.

• Model gateway: LiteLLM aliases like agent-fast, agent-smart, vision-default.

• Tools: database queries, browser automation, file search, vector search, code execution, image generation, ticket creation.

• State: app database, vector memory, object storage, traces.

Keep tool execution outside the model gateway. LiteLLM should route/model/observe; the agent framework should decide tool permissions, state transitions, retries, and side effects.

9. App → LiteLLM Router → Fallback Chain

Use fallbacks when apps should call one model name but LiteLLM can try backup backends if the primary fails.

yaml

model_list:

- model_name: chat-primary

litellm_params:

model: openai/gpt-4o-mini

api_key: os.environ/OPENAI_API_KEY

- model_name: chat-backup

litellm_params:

model: anthropic/claude-3-5-haiku-20241022

api_key: os.environ/ANTHROPIC_API_KEY

- model_name: chat-local

litellm_params:

model: ollama/llama3.1

api_base: http://localhost:11434

router_settings:

num_retries: 2

litellm_settings:

fallbacks:

- chat-primary:

- chat-backup

- chat-local

This lets callers request chat-primary while LiteLLM handles retries and fallback selection. For production, separate aliases like chat-fast, chat-premium, chat-local, and embed-default are usually easier to reason about than hiding every backend behind one name.

Practical Architecture

flowchart TD
    App[Apps, agents, automations]
    SDK[OpenAI-compatible SDKs]
    Proxy[LiteLLM proxy]
    Policy[Keys, budgets, routing, logs]
    Chat[Chat and reasoning models]
    Vision[Vision and multimodal models]
    Embed[Embeddings and rerankers]
    Audio[Speech-to-text and TTS]
    Image[Image/video generators]
    Tools[Tool and workflow services]
    Obs[Observability]

    App --> SDK --> Proxy
    Proxy --> Policy
    Proxy --> Chat
    Proxy --> Vision
    Proxy --> Embed
    Proxy --> Audio
    Proxy --> Image
    Proxy --> Tools
    Proxy --> Obs

Strictly, LiteLLM is strongest as the OpenAI-compatible model gateway. For capabilities that are not uniformly OpenAI-compatible, use small adapter services around specialized tools, then expose them to apps through the same internal AI gateway pattern.

Recommendations

• Think in capabilities, not models: chat-smart, chat-fast, vision-default, embed-text, rerank, transcribe, tts, image-gen, moderate.

• Put LiteLLM at the boundary between apps and model providers; do not scatter provider keys through every app.

• Give apps stable aliases instead of provider-specific IDs.

• Use hosted providers for reliability and frontier capability; use Ollama/llama.cpp for local dev; use vLLM/SGLang/TGI for production self-hosting.

• Use sidecars/adapters for AI systems that are not cleanly OpenAI-compatible: ComfyUI, whisper.cpp, Piper, custom rerankers, browser agents, code sandboxes.

• Keep embeddings as their own alias because embedding dimensions and vector indexes are model-specific.

• Keep multimodal aliases explicit; chat-default should not secretly become a vision/audio model unless callers know how to send those inputs.

• Treat tool calling, JSON schema, vision, audio, and context length as capability checks per model, not universal API guarantees.

• Add observability early so cost, latency, failures, and fallback behavior are visible.