CogVideo and CogVideoX are a family of open video generation models. CogVideoX is the open release of the model behind the QingYing service and covers three tasks: text-to-video, image-to-video (the I2V variants take an image as background input plus a prompt), and video continuation. The original CogVideo, a Transformer-based text-to-video model published at ICLR 2023, lives on a separate branch.
Inference and fine-tuning code ships in two forms. The SAT (SwissArmyTransformer) version exposes the model structure for research-style modification, while the diffusers version supports quantised inference through TorchAO or Optimum-quanto so that CogVideoX-2B runs on GPUs such as a GTX 1080 Ti and CogVideoX-5B on an RTX 3060. Models accept English prompts only, up to 224 or 226 tokens, and are trained on long descriptive prompts, so the project pairs generation with a prompt-expansion step using GLM-4 or a comparable LLM.
| Model | Resolution | Frames | Length and frame rate | Single-GPU memory (diffusers) |
|---|---|---|---|---|
| CogVideoX1.5-5B | 1360 × 768 | 16N + 1, N ≤ 10 (default 81) | 5 or 10 s at 16 fps | BF16 from 10 GB, INT8 from 7 GB |
| CogVideoX1.5-5B-I2V | min(W, H) = 768, max(W, H) 768–1360 | 16N + 1, N ≤ 10 | 5 or 10 s at 16 fps | BF16 from 10 GB, INT8 from 7 GB |
| CogVideoX-2B | 720 × 480 | 8N + 1, N ≤ 6 (default 49) | 6 s at 8 fps | FP16 from 4 GB, INT8 from 3.6 GB |
| CogVideoX-5B | 720 × 480 | 8N + 1, N ≤ 6 | 6 s at 8 fps | BF16 from 5 GB, INT8 from 4.4 GB |
| CogVideoX-5B-I2V | 720 × 480 | 8N + 1, N ≤ 6 | 6 s at 8 fps | BF16 from 5 GB, INT8 from 4.4 GB |
The memory figures assume the diffusers optimisations (sequential CPU offload, VAE slicing and tiling) are enabled; disabling them roughly triples peak memory and speeds up generation by 3–4×. Reported inference times at 50 steps are about 90 s (2B), 180 s (5B), and 1000 s (1.5-5B) on a single A100, roughly halved on an H100.
Features
- Text-to-video, image-to-video, and continuation: one model family covers all three generation tasks
- Two code paths: SAT weights for structural experimentation and a diffusers pipeline for deployment and quantisation
- Quantised inference: INT8 and FP8 through TorchAO and Optimum-quanto, compatible with torch.compile
- Multi-GPU inference: diffusers-based multi-GPU runs at 24 GB (1.5-5B), 15 GB (5B), and 10 GB (2B)
- Fine-tuning: LoRA fine-tuning on the diffusers model, a SAT fine-tuning path, and the CogKit framework for CogVideoX and CogView4
- DDIM inversion: supported for CogVideoX-5B and CogVideoX1.5-5B
- Prompt optimisation: a conversion script expands short prompts with GLM-4 or similar models before generation
- 3D Causal VAE: the video encoder-decoder used by CogVideoX-2B, released separately
- CogVLM2-Caption: the captioning model used to turn training videos into text descriptions
- Community adaptations: CogVideoX-Fun (flexible resolutions), RIFLEx (length extrapolation), CogStudio (Gradio web UI), Xorbits Inference, and LeMiCa (cache-based inference acceleration)
- Distribution: weights on Hugging Face, ModelScope, and WiseModel, plus Colab notebooks
