f06d824054
Co-authored-by: leffff <levnovitskiy@gmail.com> Co-authored-by: Kirill <kykozlov@gmail.com> Co-authored-by: shaoxiongduan <shaoxiongduan@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
84 lines
4.8 KiB
Markdown
84 lines
4.8 KiB
Markdown
# Kandinsky 6 Video Super-Resolution
|
|
|
|
`Kandinsky6SRPipeline` upscales a low-resolution video by x2, x2.25 or x4 (video-to-video, no text prompt). The clip is
|
|
encoded once with the SR VAE (KVAE); its latent is cut into overlapping tiles, each tile is enlarged by the *latent
|
|
upscaler*, refined by a text-free SR DiT in a few denoising steps and decoded, and the tiles are blended back together.
|
|
The source audio is kept. The clip can come from anywhere, for example from the [Kandinsky 6 T2IVA](kandinsky6.md)
|
|
pipeline.
|
|
|
|
## Models
|
|
|
|
| Variant | Hub repo | Scheduler | Default steps per tile |
|
|
|---|---|---|---|
|
|
| Flow matching | `kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers` | `FlowMatchEulerDiscreteScheduler` (shift 5.0) | 4 |
|
|
| Distilled | `kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers` | `PiflowScheduler` (shift 3.5, `n_grid` 10) | 2 |
|
|
|
|
Both repos have the same layout and share `vae/` and `latent_upscaler/`:
|
|
|
|
```text
|
|
model_index.json _class_name = Kandinsky6SRPipeline
|
|
transformer/ Kandinsky6SRTransformer3DModel (sr_params: trained base resolution, RoPE scale, noise level)
|
|
vae/ Kandinsky6SRVAE
|
|
latent_upscaler/ Kandinsky6SRLatentUpscalerBank (x2 and x4 models)
|
|
scheduler/ FlowMatchEulerDiscreteScheduler or PiflowScheduler
|
|
```
|
|
|
|
The scheduler component drives the denoising loop, and each repo resolves to its own preset (4 or 2 steps). The
|
|
distilled transformer's head holds `n_grid` predictions per latent channel, which `PiflowScheduler` integrates.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
python examples/inference/basic/basic_kandinsky6_sr.py --video-path input.mp4 --scale 2.25
|
|
python examples/inference/basic/basic_kandinsky6_sr.py --video-path input.mp4 \
|
|
--model-path kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers
|
|
```
|
|
|
|
```python
|
|
from fastvideo import VideoGenerator
|
|
|
|
generator = VideoGenerator.from_pretrained("kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers", num_gpus=1)
|
|
result = generator.generate({
|
|
"inputs": {"video_path": "input.mp4"},
|
|
"output": {"output_path": "outputs_video/sr", "return_frames": False},
|
|
"extensions": {"sr_resolution_scale": 2.25, "sr_target_resolution": "fullhd"},
|
|
})
|
|
```
|
|
|
|
The output geometry and frame rate follow the input clip; the request's `height` / `width` / `num_frames` are not used.
|
|
Clips are resampled to 24 fps by a fixed stride and only the first 121 frames (5 s, `1 + 8k`-aligned) are processed.
|
|
The source audio (mono, 44.1 kHz) is trimmed to the processed span and muxed into the output.
|
|
|
|
## Request parameters
|
|
|
|
The `sr_*` options belong only to Kandinsky6 SR. Pass them in `request.extensions` (as above), or under
|
|
`request.stage_overrides.sr`. The SR pipeline reads them from `ForwardBatch.extra`; they are not fields of the
|
|
shared `SamplingParam` or `ForwardBatch`. Other model families reject these options. Existing
|
|
`generate_video(..., sr_resolution_scale=...)` calls remain supported for Kandinsky6 SR; replace direct
|
|
`SamplingParam(sr_...=...)` construction with request extensions or these keyword arguments.
|
|
For the config-based CLI, use dotted overrides such as `--request.extensions.sr_resolution_scale 4`, rather than
|
|
shared `--sr-*` flags. The model-specific example above keeps its `--scale` and `--tiles-batch-size` flags.
|
|
|
|
| Field | Default | Meaning |
|
|
|---|---|---|
|
|
| `num_inference_steps` | 4 / 2 | Denoising steps (DiT calls) per tile. The upstream Diffusers pipeline counts grid points instead (its 5 is 4 steps here). The distilled model was trained for 2; other values run with a warning. |
|
|
| `sr_resolution_scale` | `2.25` | Total upscale: `2`, `4` or `2.25` (x1.125 pixel pre-upscale, then x2). |
|
|
| `sr_tiles_batch_size` | `1` | Tiles denoised per DiT call (raise it only if memory allows). |
|
|
| `sr_tile_min_overlap` | `0.20` | Minimum overlap between neighbouring tiles, as a fraction of the tile. |
|
|
| `sr_target_resolution` | `None` | Downscale the result to `hd`, `fullhd`, `2k` or `WxH`. |
|
|
| `sr_target_resize_mode` | `fit` | `fit` keeps the aspect ratio, `exact` uses the bucket dimensions. |
|
|
| `seed` | `42` | Tile group *k* (of `sr_tiles_batch_size` tiles) is seeded with `seed + index of its first tile`. |
|
|
|
|
Two Python-only inputs take raw tensors and are passed as keyword arguments of `generate_video()`:
|
|
|
|
- `sr_lr_latent`: an unscaled KVAE latent `[T, C, H, W]` of the source, instead of `video_path` (skips decoding and
|
|
encoding the video; scale 2 or 4 only, since 2.25 needs the pixel pre-upscale).
|
|
- `sr_audio` / `sr_audio_sample_rate`: a mono waveform in `[-1, 1]` to mux instead of the source's own audio.
|
|
|
|
## Limitations
|
|
|
|
- NABLA block-sparse attention, which both repos request for 512-pixel tiles, is not wired; the DiT runs dense attention
|
|
and logs a warning.
|
|
- No sequence or tensor parallelism: the DiT runs on one GPU. Tiles are processed one group after another.
|
|
- One clip per request.
|