ComfyUI
5 min read
ComfyUI is an open-source AI creation engine with a visual node/graph interface and API. It started as a highly controllable Stable Diffusion UI, but the current project is broader: images, image editing, video, 3D, audio, custom workflows, and production pipeline integration.
It is useful when prompt-in/image-out is not enough and you need explicit control over models, samplers, ControlNet, LoRAs, upscalers, masks, video nodes, batch jobs, and reusable workflow graphs.
Current state
The upstream project moved from comfyanonymous/ComfyUI to the Comfy organization namespace at Comfy-Org/ComfyUI. The old GitHub URL may redirect, but use the Comfy organization repo for current links.
ComfyUI now supports much more than classic Stable Diffusion:
• image generation: SD1.x, SDXL, SD3/3.5, Flux, HiDream, Qwen Image, Hunyuan Image, and many others;
• image editing/inpainting: Flux Kontext, Qwen Image Edit, Omnigen, HiDream editing models;
• video: Stable Video Diffusion, LTX-Video, Hunyuan Video, Wan models, and related workflows;
• audio: Stable Audio and similar generation workflows;
• 3D: Hunyuan3D-style workflows;
• API nodes for some hosted/closed-source models;
• custom nodes and ComfyUI-Manager;
• an async queue and workflow execution API.
Why it matters
• Graph control — every step is visible and tweakable.
• Reusable workflows — workflows can be saved as JSON and embedded in generated media metadata.
• Production-friendly — queue-based execution and API endpoints make it possible to call workflows from apps.
• Extensible — custom nodes are the ecosystem; ComfyUI-Manager helps install/update them.
• Efficient recomputation — only changed graph branches need to re-run.
• Model sharing — can be configured to use model folders from other Stable Diffusion installs.
Installation
Easiest paths
• Desktop app: easiest for Windows/macOS users.
• Windows portable package: common for local NVIDIA/AMD/Intel setups.
• Manual install: best for Linux, servers, custom Python/PyTorch versions, and production-ish workflows.
• Cloud: Comfy Cloud is the official paid hosted option.
comfy-cli
shell
pip install comfy-cli
comfy install
Manual install
shell
git clone https://github.com/Comfy-Org/ComfyUI.git
cd ComfyUI
pip install -r requirements.txt
python main.py
Then open:
text
http://127.0.0.1:8188
Model files usually go in subdirectories under models/, for example:
| Model type | Directory |
|---|---|
| Checkpoints / diffusion models | models/checkpoints |
| VAE | models/vae |
| LoRA | models/loras |
| ControlNet | models/controlnet |
| Upscalers | models/upscale_models |
| Embeddings / textual inversion | models/embeddings |
Use extra_model_paths.yaml if you want ComfyUI to reuse models from another UI or shared model directory.
Hardware notes
ComfyUI is PyTorch-based. Install the PyTorch build matching your hardware:
• NVIDIA: CUDA PyTorch builds.
• AMD Linux: ROCm PyTorch builds.
• Intel Arc: PyTorch XPU builds.
• Apple Silicon: PyTorch with MPS support; performance varies by workflow/model.
• CPU: possible with --cpu, but slow for most generation workflows.
Common launch commands:
shell
# Normal
python main.py
# Enable ComfyUI-Manager
python main.py --enable-manager
# CPU-only fallback
python main.py --cpu
# Higher-quality previews
python main.py --preview-method auto
ComfyUI-Manager
ComfyUI-Manager helps install and update custom nodes and models.
shell
pip install -r manager_requirements.txt
python main.py --enable-manager
Be careful with custom nodes in production: they are code execution plugins. Pin versions for reproducibility.
Using ComfyUI
Basic workflow:
1. Load or create a workflow graph.
2. Put models in the correct models/ directories.
3. Configure prompt nodes, sampler, seed, resolution, LoRAs, ControlNet, etc.
4. Press Ctrl+Enter to queue the graph.
5. Save the workflow as JSON or drag generated PNG/WebP/FLAC outputs back into ComfyUI to restore embedded workflows.
Useful shortcuts:
| Shortcut | Action |
|---|---|
Ctrl+Enter | Queue current graph |
Ctrl+Shift+Enter | Queue as first |
Ctrl+Alt+Enter | Cancel current generation |
Ctrl+S | Save workflow |
Ctrl+O | Load workflow |
| Double-click | Node search |
Ctrl+B | Bypass selected node |
Ctrl+M | Mute selected node |
API / app integration
ComfyUI can be treated as an image/video/audio generation sidecar behind an internal AI gateway. The usual pattern is:
flowchart TD
App[App or agent]
LLM[LiteLLM prompt/planning model]
Adapter[Workflow adapter service]
Comfy[ComfyUI queue/API]
Storage[Output storage]
App --> LLM
App --> Adapter
Adapter --> Comfy
Comfy --> Storage
Storage --> AppUse an adapter service rather than exposing raw ComfyUI directly to every app. The adapter can:
• choose a workflow template;
• fill prompt/seed/size/model parameters;
• submit to the ComfyUI queue;
• poll for completion;
• store outputs in object storage;
• return URLs/metadata to the caller;
• enforce quotas and safe model/workflow allowlists.
This works well beside LiteLLM: LiteLLM handles text/multimodal model calls, while ComfyUI handles graph-based creative generation.
Docker Compose
The old note used AbdBarho/stable-diffusion-webui-docker, which may still be useful, but upstream ComfyUI is easiest to keep current from the official repo or desktop/portable builds. For Docker, prefer an actively maintained image or build your own from the current Comfy-Org/ComfyUI repo and pin image/model versions.
Minimal conceptual compose shape:
yaml
services:
comfyui:
image: your-comfyui-image
ports:
- "8188:8188"
volumes:
- ./models:/app/ComfyUI/models
- ./input:/app/ComfyUI/input
- ./output:/app/ComfyUI/output
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
Relation to other tools
• Open WebUI can integrate image generation through APIs such as AUTOMATIC1111 and hosted image APIs; ComfyUI can fill a similar sidecar role with more workflow control.
• LiteLLM is a better fit for OpenAI-compatible text/vision/audio model routing. ComfyUI is better for graph-based generation workflows.
• AUTOMATIC1111 Stable Diffusion WebUI is simpler for many SD-only use cases; ComfyUI is more composable and pipeline-oriented.
Caveats
• Custom nodes can break across ComfyUI releases; pin versions for repeatable workflows.
• Workflows often depend on exact model files, custom node versions, and sampler settings.
• API integration is powerful but not as standardized as OpenAI-style chat APIs.
• Generated media workflows may have licensing concerns around model checkpoints, LoRAs, training data, and output usage.