nodes/aigc-apps-VideoX-Fun
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update 5b comfyui && Update Camera control && Update fun training and Readme (#284)

Browse Source 
Update 5b comfyui && Update Camera control && Update fun training and Readme
This commit is contained in:
Bubbliiiing
2025-08-13 19:20:11 +08:00
committed by GitHub
parent 24a5eed03b
commit 7157cdd48a
55 changed files with 13550 additions and 772 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1
View File
@@ -550,6 +550,7 @@ CogVideoX-Fun can be found in [Readme Train](scripts/cogvideox_fun/README_TRAIN.
|--|--|--|--|--|
| Wan2.2-Fun-A14B-InP | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-InP) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP) | Wan2.2-Fun-14B text-to-video generation weights, trained at multiple resolutions, supports start-end image prediction. |
| Wan2.2-Fun-A14B-Control | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control)| Wan2.2-Fun-14B video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc., and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction at 81 frames, trained at 16 frames per second, with multilingual prediction support. |
| Wan2.2-Fun-A14B-Control-Camera | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-Control-Camera) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera)| Wan2.2-Fun-14B camera lens control weights. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction. |
## 2. Wan2.2
README_ja-JP.md
+1
View File
@@ -550,6 +550,7 @@ CogVideoX-Funは[Readme Train](scripts/cogvideox_fun/README_TRAIN.md)と[Readme
|------|----------------|------------|-------------|------|
| Wan2.2-Fun-A14B-InP | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-InP) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP) | Wan2.2-Fun-14Bのテキスト・画像から動画を生成するモデルの重み。複数の解像度で学習されており、動画の最初と最後のフレームの予測をサポートしています。 |
| Wan2.2-Fun-A14B-Control | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control) | Wan2.2-Fun-14Bの動画制御用重み。Canny、Depth、Pose、MLSDなどのさまざまな制御条件に対応しており、軌跡制御もサポートしています。512、768、1024の複数解像度での動画生成が可能で、81フレーム、16fpsで学習されています。多言語対応の予測もサポートしています。 |
| Wan2.2-Fun-A14B-Contro-Camera | 64.0 GB | [🤗リンク](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-Control-Camera) | [😄リンク](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera)| Wan2.2-Fun-14Bのカメラレンズ制御重み。512、768、1024のマルチ解像度での動画予測をサポートし、81フレーム、毎秒16フレームで訓練されています。多言語予測に対応しています。 |
## 2. Wan2.2
README_zh-CN.md
+1
View File
@@ -540,6 +540,7 @@ CogVideoX-Fun可以查看[Readme Train](scripts/cogvideox_fun/README_TRAIN.md)
|--|--|--|--|--|
| Wan2.2-Fun-A14B-InP | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-InP) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP) | Wan2.2-Fun-14B文图生视频权重，以多分辨率训练，支持首尾图预测。 |
| Wan2.2-Fun-A14B-Control | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control)| Wan2.2-Fun-14B视频控制权重，支持不同的控制条件，如Canny、Depth、Pose、MLSD等，同时支持使用轨迹控制。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以81帧、每秒16帧进行训练，支持多语言预测 |
| Wan2.2-Fun-A14B-Control-Camera | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-Control-Camera) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera)| Wan2.2-Fun-14B相机镜头控制权重。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以81帧、每秒16帧进行训练，支持多语言预测 |
## 2. Wan2.2
comfyui/README.md
+1
View File
@@ -45,6 +45,7 @@ remote_zoe= "https://huggingface.co/lllyasviel/Annotators/resolve/main/ZoeD_M12_
|--|--|--|--|--|
| Wan2.2-Fun-A14B-InP | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-InP) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP) | Wan2.2-Fun-14B text-to-video generation weights, trained at multiple resolutions, supports start-end image prediction. |
| Wan2.2-Fun-A14B-Control | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control)| Wan2.2-Fun-14B video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc., and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction at 81 frames, trained at 16 frames per second, with multilingual prediction support. |
| Wan2.2-Fun-A14B-Control-Camera | 64.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-Control-Camera) | [😄Link](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera)| Wan2.2-Fun-14B camera lens control weights. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction. |
#### ii. Wan2.2
comfyui/comfyui_nodes.py
+4 -3
View File
@@ -77,21 +77,22 @@ class FunCompile:
for i in range(len(funmodels["pipeline"].transformer.blocks)):
funmodels["pipeline"].transformer.blocks[i] = torch.compile(funmodels["pipeline"].transformer.blocks[i])
if hasattr(funmodels["pipeline"], "transformer_2"):
if hasattr(funmodels["pipeline"], "transformer_2") and funmodels["pipeline"].transformer_2 is not None:
for i in range(len(funmodels["pipeline"].transformer_2.blocks)):
funmodels["pipeline"].transformer_2.blocks[i] = torch.compile(funmodels["pipeline"].transformer_2.blocks[i])
elif hasattr(funmodels["pipeline"].transformer, "transformer_blocks"):
for i in range(len(funmodels["pipeline"].transformer.transformer_blocks)):
funmodels["pipeline"].transformer.transformer_blocks[i] = torch.compile(funmodels["pipeline"].transformer.transformer_blocks[i])
if hasattr(funmodels["pipeline"], "transformer_2"):
if hasattr(funmodels["pipeline"], "transformer_2") and funmodels["pipeline"].transformer_2 is not None:
for i in range(len(funmodels["pipeline"].transformer_2.transformer_blocks)):
funmodels["pipeline"].transformer_2.transformer_blocks[i] = torch.compile(funmodels["pipeline"].transformer_2.transformer_blocks[i])
else:
funmodels["pipeline"].transformer.forward = torch.compile(funmodels["pipeline"].transformer.forward)
if hasattr(funmodels["pipeline"], "transformer_2"):
if hasattr(funmodels["pipeline"], "transformer_2") and funmodels["pipeline"].transformer_2 is not None:
funmodels["pipeline"].transformer_2.forward = torch.compile(funmodels["pipeline"].transformer_2.forward)
print("Add Compile")
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_control_camera.json
Executable → Regular
+184 -184
View File
@@ -1,18 +1,20 @@
{
"id": "addf6f13-291c-4b88-a80c-cd5785fa8f42",
"revision": 0,
"last_node_id": 132,
"last_link_id": 292,
"nodes": [
{
"id": 107,
"type": "Note",
"pos": {
"0": 4,
"1": 634
},
"size": {
"0": 210,
"1": 58
},
"pos": [
4,
634
],
"size": [
210,
88
],
"flags": {},
"order": 0,
"mode": 0,
@@ -30,14 +32,14 @@
{
"id": 108,
"type": "Note",
"pos": {
"0": -110,
"1": 842
},
"size": {
"0": 326.1556091308594,
"1": 145.20904541015625
},
"pos": [
-110,
842
],
"size": [
326.1556091308594,
145.20904541015625
],
"flags": {},
"order": 1,
"mode": 0,
@@ -52,54 +54,29 @@
"color": "#432",
"bgcolor": "#653"
},
{
"id": 112,
"type": "Note",
"pos": {
"0": -203,
"1": 252
},
"size": {
"0": 427.074951171875,
"1": 143.9142608642578
},
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"Due to the large size of models from EasyAnimateV5 and above, when using the 12B model, if your graphics card has 24GB or less of VRAM, please set GPU_memory_mode to model_cpu_offload_and_qfloat8. This will load the model in float8 to reduce VRAM consumption, otherwise you may receive an out-of-memory error. \n（由于EasyAnimateV5以上的模型较大，当使用12B模型时，如果使用的显卡显存为24G及以下，请将GPU_memory_mode设置为model_cpu_offload_and_qfloat8，使得模型加载在float8上减少显存消耗，否则会提示显存不足。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 122,
"type": "FunTextBox",
"pos": {
"0": 238,
"1": 805
},
"size": {
"0": 400,
"1": 200
},
"pos": [
238,
805
],
"size": [
400,
200
],
"flags": {},
"order": 3,
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
289
],
"slot_index": 0
]
}
],
"title": "Negtive Prompt（反向提示词）",
@@ -113,16 +90,16 @@
{
"id": 129,
"type": "CameraBasicFromChaoJie",
"pos": {
"0": 805.2059326171875,
"1": 1012.381103515625
},
"size": {
"0": 315,
"1": 106
},
"pos": [
805.2059326171875,
1012.381103515625
],
"size": [
315,
106
],
"flags": {},
"order": 4,
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
@@ -144,14 +121,14 @@
{
"id": 130,
"type": "CameraTrajectoryFromChaoJie",
"pos": {
"0": 1170.206298828125,
"1": 763.3814697265625
},
"size": {
"0": 367.79998779296875,
"1": 150
},
"pos": [
1170.206298828125,
763.3814697265625
],
"size": [
367.79998779296875,
150
],
"flags": {},
"order": 10,
"mode": 0,
@@ -166,10 +143,10 @@
{
"name": "camera_trajectory",
"type": "STRING",
"slot_index": 0,
"links": [
292
],
"slot_index": 0
]
},
{
"name": "video_length",
@@ -190,55 +167,53 @@
{
"id": 106,
"type": "VHS_VideoCombine",
"pos": {
"0": 1408,
"1": 68
},
"pos": [
1408,
68
],
"size": [
390,
537.4615384615385
537.4615478515625
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 291,
"slot_index": 0,
"label": "图像",
"shape": 7
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 291
},
{
"name": "audio",
"type": "AUDIO",
"link": null,
"label": "音频",
"shape": 7
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"name": "meta_batch",
"type": "VHS_BatchManager",
"link": null,
"label": "批次管理",
"shape": 7
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null,
"shape": 7
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null,
"slot_index": 0,
"shape": 3,
"label": "文件名"
"links": null
}
],
"properties": {
@@ -270,16 +245,16 @@
{
"id": 121,
"type": "FunTextBox",
"pos": {
"0": 235,
"1": 539
},
"size": {
"0": 400,
"1": 200
},
"pos": [
235,
539
],
"size": [
400,
200
],
"flags": {},
"order": 5,
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
@@ -302,35 +277,33 @@
{
"id": 100,
"type": "LoadImage",
"pos": {
"0": 237.59738159179688,
"1": 1164.597412109375
},
"size": {
"0": 378.07147216796875,
"1": 314
},
"pos": [
237.59738159179688,
1164.597412109375
],
"size": [
378.07147216796875,
314
],
"flags": {},
"order": 6,
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"label": "图像",
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
290
],
"slot_index": 0,
"shape": 3,
"label": "图像"
]
},
{
"label": "遮罩",
"name": "MASK",
"type": "MASK",
"links": null,
"shape": 3,
"label": "遮罩"
"links": null
}
],
"title": "Start Image（图片到视频的开始图片）",
@@ -345,16 +318,16 @@
{
"id": 131,
"type": "CameraCombineFromChaoJie",
"pos": {
"0": 814.2059326171875,
"1": 763.3814697265625
},
"size": {
"0": 315,
"1": 178
},
"pos": [
814.2059326171875,
763.3814697265625
],
"size": [
315,
178
],
"flags": {},
"order": 7,
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
@@ -381,16 +354,16 @@
{
"id": 110,
"type": "Note",
"pos": {
"0": 1158.206298828125,
"1": 970.381103515625
},
"size": {
"0": 608.1410522460938,
"1": 188.2682342529297
},
"pos": [
1158.206298828125,
970.381103515625
],
"size": [
608.1410522460938,
188.2682342529297
],
"flags": {},
"order": 8,
"order": 7,
"mode": 0,
"inputs": [],
"outputs": [],
@@ -406,14 +379,14 @@
{
"id": 132,
"type": "WanFunV2VSampler",
"pos": {
"0": 899,
"1": 68
},
"size": {
"0": 428.4000244140625,
"1": 486
},
"pos": [
899,
68
],
"size": [
428.4000244140625,
506
],
"flags": {},
"order": 11,
"mode": 0,
@@ -435,42 +408,39 @@
},
{
"name": "validation_video",
"shape": 7,
"type": "IMAGE",
"link": null,
"shape": 7
"link": null
},
{
"name": "control_video",
"shape": 7,
"type": "IMAGE",
"link": null,
"shape": 7
"link": null
},
{
"name": "start_image",
"shape": 7,
"type": "IMAGE",
"link": 290,
"shape": 7
"link": 290
},
{
"name": "ref_image",
"shape": 7,
"type": "IMAGE",
"link": null,
"shape": 7
},
{
"name": "riflex_k",
"type": "RIFLEXT_ARGS",
"link": null,
"shape": 7
"link": null
},
{
"name": "camera_conditions",
"shape": 7,
"type": "STRING",
"link": 292,
"widget": {
"name": "camera_conditions"
},
"shape": 7
"link": 292
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
@@ -492,7 +462,7 @@
"fixed",
50,
6,
1.0,
1,
"Flow",
0.1,
true,
@@ -504,26 +474,26 @@
{
"id": 123,
"type": "LoadWanFunModel",
"pos": {
"0": 281,
"1": 251
},
"size": {
"0": 315,
"1": 154
},
"pos": [
281,
251
],
"size": [
315,
154
],
"flags": {},
"order": 9,
"order": 8,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"slot_index": 0,
"links": [
287
],
"slot_index": 0
]
}
],
"properties": {
@@ -536,6 +506,31 @@
"wan2.1/wan_civitai.yaml",
"bf16"
]
},
{
"id": 112,
"type": "Note",
"pos": [
-203,
252
],
"size": [
427.074951171875,
143.9142608642578
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 1.3B model, you can set GPU_memory_mode to model_cpu_offload for faster generation. When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用1.3B模型时，可以设置GPU_memory_mode为model_cpu_offload进行更快速度的生成，在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
}
],
"links": [
@@ -592,12 +587,13 @@
130,
0,
132,
8,
7,
"STRING"
]
],
"groups": [
{
"id": 1,
"title": "Generate Control Video",
"bounding": [
773,
@@ -610,6 +606,7 @@
"flags": {}
},
{
"id": 2,
"title": "First Image",
"bounding": [
191,
@@ -622,6 +619,7 @@
"flags": {}
},
{
"id": 3,
"title": "Prompts",
"bounding": [
191,
@@ -634,7 +632,8 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"id": 4,
"title": "Load Model",
"bounding": [
189,
160,
@@ -649,17 +648,18 @@
"config": {},
"extra": {
"ds": {
"scale": 0.8264462809917358,
"scale": 0.6830134553650709,
"offset": [
28.17192681115923,
-3.293207324975433
236.8040025924094,
-27.611371387475245
]
},
"node_versions": {
"CogVideoX-Fun": "a7fa7028d52498f13e983eba012a81ebcae24977",
"CogVideoX-Fun": "e935795d7684aaba72be65d2ad5ef580c0526f7a",
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e",
"comfy-core": "v0.2.7-3-g8afb97c"
}
"comfy-core": "0.3.44"
},
"frontendVersion": "1.21.3"
},
"version": 0.4
}
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_control_trajectory.json
Executable → Regular
+255 -273
View File
@@ -1,18 +1,20 @@
{
"id": "34fdaa39-a397-46b5-84d1-8846799c8c09",
"revision": 0,
"last_node_id": 126,
"last_link_id": 287,
"nodes": [
{
"id": 107,
"type": "Note",
"pos": {
"0": 4,
"1": 634
},
"size": {
"0": 210,
"1": 58
},
"pos": [
4,
634
],
"size": [
210,
88
],
"flags": {},
"order": 0,
"mode": 0,
@@ -30,14 +32,14 @@
{
"id": 108,
"type": "Note",
"pos": {
"0": -110,
"1": 842
},
"size": {
"0": 326.1556091308594,
"1": 145.20904541015625
},
"pos": [
-110,
842
],
"size": [
326.1556091308594,
145.20904541015625
],
"flags": {},
"order": 1,
"mode": 0,
@@ -53,62 +55,35 @@
"bgcolor": "#653"
},
{
"id": 112,
"type": "Note",
"pos": {
"0": -203,
"1": 252
},
"size": {
"0": 427.074951171875,
"1": 143.9142608642578
},
"id": 100,
"type": "LoadImage",
"pos": [
238,
1165
],
"size": [
378.07147216796875,
314
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"Due to the large size of models from EasyAnimateV5 and above, when using the 12B model, if your graphics card has 24GB or less of VRAM, please set GPU_memory_mode to model_cpu_offload_and_qfloat8. This will load the model in float8 to reduce VRAM consumption, otherwise you may receive an out-of-memory error. \n（由于EasyAnimateV5以上的模型较大，当使用12B模型时，如果使用的显卡显存为24G及以下，请将GPU_memory_mode设置为model_cpu_offload_and_qfloat8，使得模型加载在float8上减少显存消耗，否则会提示显存不足。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 100,
"type": "LoadImage",
"pos": {
"0": 238,
"1": 1165
},
"size": {
"0": 378.07147216796875,
"1": 314
},
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"label": "图像",
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
285
],
"slot_index": 0,
"shape": 3,
"label": "图像"
]
},
{
"label": "遮罩",
"name": "MASK",
"type": "MASK",
"links": null,
"shape": 3,
"label": "遮罩"
"links": null
}
],
"title": "Start Image（图片到视频的开始图片）",
@@ -123,16 +98,16 @@
{
"id": 110,
"type": "Note",
"pos": {
"0": 847,
"1": 613
},
"size": {
"0": 608.1410522460938,
"1": 188.2682342529297
},
"pos": [
847,
613
],
"size": [
608.1410522460938,
188.2682342529297
],
"flags": {},
"order": 4,
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [],
@@ -148,14 +123,14 @@
{
"id": 118,
"type": "AppendStringsToList",
"pos": {
"0": 1140.1396484375,
"1": 909.9193115234375
},
"size": {
"0": 315,
"1": 82
},
"pos": [
1140.1396484375,
909.9193115234375
],
"size": [
315,
82
],
"flags": {
"collapsed": false
},
@@ -165,51 +140,42 @@
{
"name": "string1",
"type": "STRING",
"link": 265,
"widget": {
"name": "string1"
}
"link": 265
},
{
"name": "string2",
"type": "STRING",
"link": 266,
"widget": {
"name": "string2"
}
"link": 266
}
],
"outputs": [
{
"name": "STRING",
"type": "STRING",
"slot_index": 0,
"links": [
267
],
"slot_index": 0
]
}
],
"properties": {
"Node name for S&R": "AppendStringsToList"
},
"widgets_values": [
"",
""
]
"widgets_values": []
},
{
"id": 121,
"type": "FunTextBox",
"pos": {
"0": 235,
"1": 539
},
"size": {
"0": 400,
"1": 200
},
"pos": [
235,
539
],
"size": [
400,
200
],
"flags": {},
"order": 5,
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
@@ -232,14 +198,14 @@
{
"id": 114,
"type": "ImageMaximumNode",
"pos": {
"0": 2074,
"1": 905
},
"size": {
"0": 210,
"1": 46
},
"pos": [
2074,
905
],
"size": [
210,
46
],
"flags": {},
"order": 15,
"mode": 0,
@@ -272,74 +238,69 @@
{
"id": 95,
"type": "CreateTrajectoryBasedOnKJNodes",
"pos": {
"0": 1574.139404296875,
"1": 929.9193115234375
},
"size": {
"0": 428.4000244140625,
"1": 58
},
"pos": [
1574.139404296875,
929.9193115234375
],
"size": [
428.4000244140625,
58
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "coordinates",
"type": "STRING",
"link": 267
},
{
"name": "masks",
"type": "MASK",
"link": 249
},
{
"name": "coordinates",
"type": "STRING",
"link": 267,
"widget": {
"name": "coordinates"
}
}
],
"outputs": [
{
"name": "image",
"type": "IMAGE",
"slot_index": 0,
"links": [
237,
262,
284
],
"slot_index": 0
]
}
],
"properties": {
"Node name for S&R": "CreateTrajectoryBasedOnKJNodes"
},
"widgets_values": [
""
]
"widgets_values": []
},
{
"id": 122,
"type": "FunTextBox",
"pos": {
"0": 238,
"1": 805
},
"size": {
"0": 400,
"1": 200
},
"pos": [
238,
805
],
"size": [
400,
200
],
"flags": {},
"order": 6,
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
283
],
"slot_index": 0
]
}
],
"title": "Negtive Prompt（反向提示词）",
@@ -353,41 +314,41 @@
{
"id": 97,
"type": "SplineEditor",
"pos": {
"0": 855.1397705078125,
"1": 1058.91943359375
},
"size": {
"0": 645,
"1": 812
},
"pos": [
855.1397705078125,
1058.91943359375
],
"size": [
645,
832
],
"flags": {},
"order": 7,
"order": 6,
"mode": 0,
"inputs": [
{
"name": "bg_image",
"shape": 7,
"type": "IMAGE",
"link": null,
"shape": 7
"link": null
}
],
"outputs": [
{
"name": "mask",
"type": "MASK",
"slot_index": 0,
"links": [
249
],
"slot_index": 0
]
},
{
"name": "coord_str",
"type": "STRING",
"slot_index": 1,
"links": [
265
],
"slot_index": 1
]
},
{
"name": "float",
@@ -411,8 +372,8 @@
"imgData": null
},
"widgets_values": [
"[{\"x\":236.74497000000005,\"y\":169.10355000000004},{\"x\":263.53799999999995,\"y\":230.59575},{\"x\":321.61075881587004,\"y\":229.72197058276433},{\"x\":355.3934015486295,\"y\":168.9132136637973},{\"x\":343.23165016483614,\"y\":82.42964826793309},{\"x\":275.6663646993172,\"y\":55.40353408172552},{\"x\":197.29063355931524,\"y\":85.13225968655384},{\"x\":177.02104791965957,\"y\":148.64362802414163},{\"x\":205.39846781517753,\"y\":232.4245820013851},{\"x\":259.45069618759265,\"y\":167.56190795448694}]",
"[{\"x\":236.74496459960938,\"y\":169.10354614257812},{\"x\":238.94464111328125,\"y\":177.68576049804688},{\"x\":241.3327178955078,\"y\":186.21737670898438},{\"x\":243.95693969726562,\"y\":194.67919921875},{\"x\":246.88812255859375,\"y\":203.03933715820312},{\"x\":250.2393035888672,\"y\":211.23939514160156},{\"x\":254.20826721191406,\"y\":219.15655517578125},{\"x\":259.18768310546875,\"y\":226.46995544433594},{\"x\":265.87689208984375,\"y\":232.22171020507812},{\"x\":273.4639892578125,\"y\":236.787353515625},{\"x\":281.6380920410156,\"y\":240.17092895507812},{\"x\":290.3527526855469,\"y\":241.61793518066406},{\"x\":299.1367492675781,\"y\":240.67857360839844},{\"x\":307.49658203125,\"y\":237.7835235595703},{\"x\":315.3465576171875,\"y\":233.68600463867188},{\"x\":322.80859375,\"y\":228.9126739501953},{\"x\":330.00958251953125,\"y\":223.75323486328125},{\"x\":336.7490234375,\"y\":218.0097198486328},{\"x\":342.5494689941406,\"y\":211.32901000976562},{\"x\":346.96185302734375,\"y\":203.6618194580078},{\"x\":350.05450439453125,\"y\":195.3665771484375},{\"x\":352.2560729980469,\"y\":186.787109375},{\"x\":353.9389343261719,\"y\":178.08937072753906},{\"x\":355.33001708984375,\"y\":169.3397216796875},{\"x\":356.59344482421875,\"y\":160.57052612304688},{\"x\":357.7420959472656,\"y\":151.78565979003906},{\"x\":358.6518249511719,\"y\":142.9732208251953},{\"x\":359.1504211425781,\"y\":134.1287384033203},{\"x\":359.0207214355469,\"y\":125.2725601196289},{\"x\":358.035400390625,\"y\":116.4721908569336},{\"x\":356.0333251953125,\"y\":107.84730529785156},{\"x\":352.9960632324219,\"y\":99.53003692626953},{\"x\":349.046142578125,\"y\":91.60392761230469},{\"x\":344.37164306640625,\"y\":84.08070373535156},{\"x\":339.17694091796875,\"y\":76.9051742553711},{\"x\":333.4718322753906,\"y\":70.13098907470703},{\"x\":326.9710693359375,\"y\":64.12535858154297},{\"x\":319.4295654296875,\"y\":59.51286315917969},{\"x\":311.02655029296875,\"y\":56.7620735168457},{\"x\":302.25640869140625,\"y\":55.55453872680664},{\"x\":293.40484619140625,\"y\":55.21137619018555},{\"x\":284.5453796386719,\"y\":55.24800491333008},{\"x\":275.68701171875,\"y\":55.40314865112305},{\"x\":266.83221435546875,\"y\":55.690216064453125},{\"x\":257.9945373535156,\"y\":56.30451965332031},{\"x\":249.2041473388672,\"y\":57.398460388183594},{\"x\":240.5244140625,\"y\":59.16102981567383},{\"x\":232.06307983398438,\"y\":61.773162841796875},{\"x\":223.95306396484375,\"y\":65.32772827148438},{\"x\":216.29017639160156,\"y\":69.7664566040039},{\"x\":209.08375549316406,\"y\":74.91584777832031},{\"x\":202.27122497558594,\"y\":80.57798767089844},{\"x\":195.76087951660156,\"y\":86.58625030517578},{\"x\":189.468505859375,\"y\":92.8224105834961},{\"x\":183.6200714111328,\"y\":99.47235107421875},{\"x\":178.9163055419922,\"y\":106.95783233642578},{\"x\":176.3991241455078,\"y\":115.42044830322266},{\"x\":175.83934020996094,\"y\":124.25370788574219},{\"x\":176.12339782714844,\"y\":133.10797119140625},{\"x\":176.63381958007812,\"y\":141.95301818847656},{\"x\":177.14378356933594,\"y\":150.79808044433594},{\"x\":177.73876953125,\"y\":159.63778686523438},{\"x\":178.5083770751953,\"y\":168.46385192871094},{\"x\":179.4958038330078,\"y\":177.26815795898438},{\"x\":180.75762939453125,\"y\":186.03709411621094},{\"x\":182.37294006347656,\"y\":194.7476043701172},{\"x\":184.4583282470703,\"y\":203.35687255859375},{\"x\":187.1986083984375,\"y\":211.7789306640625},{\"x\":190.9047393798828,\"y\":219.81732177734375},{\"x\":196.108154296875,\"y\":226.9572296142578},{\"x\":203.41249084472656,\"y\":231.83749389648438},{\"x\":211.85409545898438,\"y\":230.89816284179688},{\"x\":218.7886199951172,\"y\":225.41867065429688},{\"x\":224.8160400390625,\"y\":218.92974853515625},{\"x\":230.3641357421875,\"y\":212.0236358642578},{\"x\":235.608642578125,\"y\":204.8833770751953},{\"x\":240.63941955566406,\"y\":197.59080505371094},{\"x\":245.50965881347656,\"y\":190.1898651123047},{\"x\":250.25364685058594,\"y\":182.7073516845703},{\"x\":254.8948974609375,\"y\":175.16050720214844},{\"x\":259.45068359375,\"y\":167.56190490722656}]",
"[{\"points\":[{\"x\":236.74497000000005,\"y\":169.10355000000004},{\"x\":263.53799999999995,\"y\":230.59575},{\"x\":321.61075881587004,\"y\":229.72197058276433},{\"x\":355.3934015486295,\"y\":168.9132136637973},{\"x\":343.23165016483614,\"y\":82.42964826793309},{\"x\":275.6663646993172,\"y\":55.40353408172552},{\"x\":197.29063355931524,\"y\":85.13225968655384},{\"x\":177.02104791965957,\"y\":148.64362802414163},{\"x\":205.39846781517753,\"y\":232.4245820013851},{\"x\":259.45069618759265,\"y\":167.56190795448694}],\"color\":\"#1f77b4\",\"name\":\"Spline 1\"}]",
"[[{\"x\":236.74496459960938,\"y\":169.10354614257812},{\"x\":238.94464111328125,\"y\":177.68576049804688},{\"x\":241.3327178955078,\"y\":186.21737670898438},{\"x\":243.95693969726562,\"y\":194.67919921875},{\"x\":246.88812255859375,\"y\":203.03933715820312},{\"x\":250.2393035888672,\"y\":211.23939514160156},{\"x\":254.20826721191406,\"y\":219.15655517578125},{\"x\":259.18768310546875,\"y\":226.46995544433594},{\"x\":265.87689208984375,\"y\":232.22171020507812},{\"x\":273.4639892578125,\"y\":236.787353515625},{\"x\":281.6380920410156,\"y\":240.17092895507812},{\"x\":290.3527526855469,\"y\":241.61793518066406},{\"x\":299.1367492675781,\"y\":240.67857360839844},{\"x\":307.49658203125,\"y\":237.7835235595703},{\"x\":315.3465576171875,\"y\":233.68600463867188},{\"x\":322.80859375,\"y\":228.9126739501953},{\"x\":330.00958251953125,\"y\":223.75323486328125},{\"x\":336.7490234375,\"y\":218.0097198486328},{\"x\":342.5494689941406,\"y\":211.32901000976562},{\"x\":346.96185302734375,\"y\":203.6618194580078},{\"x\":350.05450439453125,\"y\":195.3665771484375},{\"x\":352.2560729980469,\"y\":186.787109375},{\"x\":353.9389343261719,\"y\":178.08937072753906},{\"x\":355.33001708984375,\"y\":169.3397216796875},{\"x\":356.59344482421875,\"y\":160.57052612304688},{\"x\":357.7420959472656,\"y\":151.78565979003906},{\"x\":358.6518249511719,\"y\":142.9732208251953},{\"x\":359.1504211425781,\"y\":134.1287384033203},{\"x\":359.0207214355469,\"y\":125.2725601196289},{\"x\":358.035400390625,\"y\":116.4721908569336},{\"x\":356.0333251953125,\"y\":107.84730529785156},{\"x\":352.9960632324219,\"y\":99.53003692626953},{\"x\":349.046142578125,\"y\":91.60392761230469},{\"x\":344.37164306640625,\"y\":84.08070373535156},{\"x\":339.17694091796875,\"y\":76.9051742553711},{\"x\":333.4718322753906,\"y\":70.13098907470703},{\"x\":326.9710693359375,\"y\":64.12535858154297},{\"x\":319.4295654296875,\"y\":59.51286315917969},{\"x\":311.02655029296875,\"y\":56.7620735168457},{\"x\":302.25640869140625,\"y\":55.55453872680664},{\"x\":293.40484619140625,\"y\":55.21137619018555},{\"x\":284.5453796386719,\"y\":55.24800491333008},{\"x\":275.68701171875,\"y\":55.40314865112305},{\"x\":266.83221435546875,\"y\":55.690216064453125},{\"x\":257.9945373535156,\"y\":56.30451965332031},{\"x\":249.2041473388672,\"y\":57.398460388183594},{\"x\":240.5244140625,\"y\":59.16102981567383},{\"x\":232.06307983398438,\"y\":61.773162841796875},{\"x\":223.95306396484375,\"y\":65.32772827148438},{\"x\":216.29017639160156,\"y\":69.7664566040039},{\"x\":209.08375549316406,\"y\":74.91584777832031},{\"x\":202.27122497558594,\"y\":80.57798767089844},{\"x\":195.76087951660156,\"y\":86.58625030517578},{\"x\":189.468505859375,\"y\":92.8224105834961},{\"x\":183.6200714111328,\"y\":99.47235107421875},{\"x\":178.9163055419922,\"y\":106.95783233642578},{\"x\":176.3991241455078,\"y\":115.42044830322266},{\"x\":175.83934020996094,\"y\":124.25370788574219},{\"x\":176.12339782714844,\"y\":133.10797119140625},{\"x\":176.63381958007812,\"y\":141.95301818847656},{\"x\":177.14378356933594,\"y\":150.79808044433594},{\"x\":177.73876953125,\"y\":159.63778686523438},{\"x\":178.5083770751953,\"y\":168.46385192871094},{\"x\":179.4958038330078,\"y\":177.26815795898438},{\"x\":180.75762939453125,\"y\":186.03709411621094},{\"x\":182.37294006347656,\"y\":194.7476043701172},{\"x\":184.4583282470703,\"y\":203.35687255859375},{\"x\":187.1986083984375,\"y\":211.7789306640625},{\"x\":190.9047393798828,\"y\":219.81732177734375},{\"x\":196.108154296875,\"y\":226.9572296142578},{\"x\":203.41249084472656,\"y\":231.83749389648438},{\"x\":211.85409545898438,\"y\":230.89816284179688},{\"x\":218.7886199951172,\"y\":225.41867065429688},{\"x\":224.8160400390625,\"y\":218.92974853515625},{\"x\":230.3641357421875,\"y\":212.0236358642578},{\"x\":235.608642578125,\"y\":204.8833770751953},{\"x\":240.63941955566406,\"y\":197.59080505371094},{\"x\":245.50965881347656,\"y\":190.1898651123047},{\"x\":250.25364685058594,\"y\":182.7073516845703},{\"x\":254.8948974609375,\"y\":175.16050720214844},{\"x\":259.45068359375,\"y\":167.56190490722656}]]",
600,
382,
81,
@@ -423,47 +384,46 @@
"list",
0,
1,
null,
null,
"",
null
]
},
{
"id": 119,
"type": "SplineEditor",
"pos": {
"0": 1544.139404296875,
"1": 1047.919189453125
},
"size": {
"0": 645,
"1": 812
},
"pos": [
1544.139404296875,
1047.919189453125
],
"size": [
645,
832
],
"flags": {},
"order": 8,
"order": 7,
"mode": 0,
"inputs": [
{
"name": "bg_image",
"shape": 7,
"type": "IMAGE",
"link": null,
"shape": 7
"link": null
}
],
"outputs": [
{
"name": "mask",
"type": "MASK",
"links": [],
"slot_index": 0
"slot_index": 0,
"links": []
},
{
"name": "coord_str",
"type": "STRING",
"slot_index": 1,
"links": [
266
],
"slot_index": 1
]
},
{
"name": "float",
@@ -487,8 +447,8 @@
"imgData": null
},
"widgets_values": [
"[{\"x\":63.916760050380844,\"y\":114.45559357858895},{\"x\":71.34894145158792,\"y\":114.45559357858895}]",
"[{\"x\":63.9167594909668,\"y\":114.45559692382812},{\"x\":64.00965881347656,\"y\":114.45559692382812},{\"x\":64.1025619506836,\"y\":114.45559692382812},{\"x\":64.19546508789062,\"y\":114.45559692382812},{\"x\":64.28836822509766,\"y\":114.45559692382812},{\"x\":64.38127136230469,\"y\":114.45559692382812},{\"x\":64.47417449951172,\"y\":114.45559692382812},{\"x\":64.56707763671875,\"y\":114.45559692382812},{\"x\":64.65998077392578,\"y\":114.45559692382812},{\"x\":64.75287628173828,\"y\":114.45559692382812},{\"x\":64.84577941894531,\"y\":114.45559692382812},{\"x\":64.93868255615234,\"y\":114.45559692382812},{\"x\":65.03158569335938,\"y\":114.45559692382812},{\"x\":65.1244888305664,\"y\":114.45559692382812},{\"x\":65.21739196777344,\"y\":114.45559692382812},{\"x\":65.31029510498047,\"y\":114.45559692382812},{\"x\":65.4031982421875,\"y\":114.45559692382812},{\"x\":65.49609375,\"y\":114.45559692382812},{\"x\":65.58899688720703,\"y\":114.45559692382812},{\"x\":65.68190002441406,\"y\":114.45559692382812},{\"x\":65.7748031616211,\"y\":114.45559692382812},{\"x\":65.86770629882812,\"y\":114.45559692382812},{\"x\":65.96060943603516,\"y\":114.45559692382812},{\"x\":66.05351257324219,\"y\":114.45559692382812},{\"x\":66.14641571044922,\"y\":114.45559692382812},{\"x\":66.23931884765625,\"y\":114.45559692382812},{\"x\":66.33221435546875,\"y\":114.45559692382812},{\"x\":66.42511749267578,\"y\":114.45559692382812},{\"x\":66.51802062988281,\"y\":114.45559692382812},{\"x\":66.61092376708984,\"y\":114.45559692382812},{\"x\":66.70382690429688,\"y\":114.45559692382812},{\"x\":66.7967300415039,\"y\":114.45559692382812},{\"x\":66.88963317871094,\"y\":114.45559692382812},{\"x\":66.98253631591797,\"y\":114.45559692382812},{\"x\":67.075439453125,\"y\":114.45559692382812},{\"x\":67.1683349609375,\"y\":114.45559692382812},{\"x\":67.26123809814453,\"y\":114.45559692382812},{\"x\":67.35414123535156,\"y\":114.45559692382812},{\"x\":67.4470443725586,\"y\":114.45559692382812},{\"x\":67.53994750976562,\"y\":114.45559692382812},{\"x\":67.63285064697266,\"y\":114.45559692382812},{\"x\":67.72575378417969,\"y\":114.45559692382812},{\"x\":67.81864929199219,\"y\":114.45559692382812},{\"x\":67.91155242919922,\"y\":114.45559692382812},{\"x\":68.00445556640625,\"y\":114.45559692382812},{\"x\":68.09735870361328,\"y\":114.45559692382812},{\"x\":68.19026184082031,\"y\":114.45559692382812},{\"x\":68.28316497802734,\"y\":114.45559692382812},{\"x\":68.37606811523438,\"y\":114.45559692382812},{\"x\":68.4689712524414,\"y\":114.45559692382812},{\"x\":68.56187438964844,\"y\":114.45559692382812},{\"x\":68.65476989746094,\"y\":114.45559692382812},{\"x\":68.74767303466797,\"y\":114.45559692382812},{\"x\":68.840576171875,\"y\":114.45559692382812},{\"x\":68.93347930908203,\"y\":114.45559692382812},{\"x\":69.02638244628906,\"y\":114.45559692382812},{\"x\":69.1192855834961,\"y\":114.45559692382812},{\"x\":69.21218872070312,\"y\":114.45559692382812},{\"x\":69.30509185791016,\"y\":114.45559692382812},{\"x\":69.39799499511719,\"y\":114.45559692382812},{\"x\":69.49089050292969,\"y\":114.45559692382812},{\"x\":69.58379364013672,\"y\":114.45559692382812},{\"x\":69.67669677734375,\"y\":114.45559692382812},{\"x\":69.76959991455078,\"y\":114.45559692382812},{\"x\":69.86250305175781,\"y\":114.45559692382812},{\"x\":69.95540618896484,\"y\":114.45559692382812},{\"x\":70.04830932617188,\"y\":114.45559692382812},{\"x\":70.1412124633789,\"y\":114.45559692382812},{\"x\":70.2341079711914,\"y\":114.45559692382812},{\"x\":70.32701110839844,\"y\":114.45559692382812},{\"x\":70.41991424560547,\"y\":114.45559692382812},{\"x\":70.5128173828125,\"y\":114.45559692382812},{\"x\":70.60572052001953,\"y\":114.45559692382812},{\"x\":70.69862365722656,\"y\":114.45559692382812},{\"x\":70.7915267944336,\"y\":114.45559692382812},{\"x\":70.88442993164062,\"y\":114.45559692382812},{\"x\":70.97732543945312,\"y\":114.45559692382812},{\"x\":71.07022857666016,\"y\":114.45559692382812},{\"x\":71.16313171386719,\"y\":114.45559692382812},{\"x\":71.25603485107422,\"y\":114.45559692382812},{\"x\":71.34893798828125,\"y\":114.45559692382812}]",
"[{\"points\":[{\"x\":63.916760050380844,\"y\":114.45559357858895},{\"x\":71.34894145158792,\"y\":114.45559357858895}],\"color\":\"#1f77b4\",\"name\":\"Spline 1\"}]",
"[[{\"x\":63.9167594909668,\"y\":114.45559692382812},{\"x\":64.00965881347656,\"y\":114.45559692382812},{\"x\":64.1025619506836,\"y\":114.45559692382812},{\"x\":64.19546508789062,\"y\":114.45559692382812},{\"x\":64.28836822509766,\"y\":114.45559692382812},{\"x\":64.38127136230469,\"y\":114.45559692382812},{\"x\":64.47417449951172,\"y\":114.45559692382812},{\"x\":64.56707763671875,\"y\":114.45559692382812},{\"x\":64.65998077392578,\"y\":114.45559692382812},{\"x\":64.75287628173828,\"y\":114.45559692382812},{\"x\":64.84577941894531,\"y\":114.45559692382812},{\"x\":64.93868255615234,\"y\":114.45559692382812},{\"x\":65.03158569335938,\"y\":114.45559692382812},{\"x\":65.1244888305664,\"y\":114.45559692382812},{\"x\":65.21739196777344,\"y\":114.45559692382812},{\"x\":65.31029510498047,\"y\":114.45559692382812},{\"x\":65.4031982421875,\"y\":114.45559692382812},{\"x\":65.49609375,\"y\":114.45559692382812},{\"x\":65.58899688720703,\"y\":114.45559692382812},{\"x\":65.68190002441406,\"y\":114.45559692382812},{\"x\":65.7748031616211,\"y\":114.45559692382812},{\"x\":65.86770629882812,\"y\":114.45559692382812},{\"x\":65.96060943603516,\"y\":114.45559692382812},{\"x\":66.05351257324219,\"y\":114.45559692382812},{\"x\":66.14641571044922,\"y\":114.45559692382812},{\"x\":66.23931884765625,\"y\":114.45559692382812},{\"x\":66.33221435546875,\"y\":114.45559692382812},{\"x\":66.42511749267578,\"y\":114.45559692382812},{\"x\":66.51802062988281,\"y\":114.45559692382812},{\"x\":66.61092376708984,\"y\":114.45559692382812},{\"x\":66.70382690429688,\"y\":114.45559692382812},{\"x\":66.7967300415039,\"y\":114.45559692382812},{\"x\":66.88963317871094,\"y\":114.45559692382812},{\"x\":66.98253631591797,\"y\":114.45559692382812},{\"x\":67.075439453125,\"y\":114.45559692382812},{\"x\":67.1683349609375,\"y\":114.45559692382812},{\"x\":67.26123809814453,\"y\":114.45559692382812},{\"x\":67.35414123535156,\"y\":114.45559692382812},{\"x\":67.4470443725586,\"y\":114.45559692382812},{\"x\":67.53994750976562,\"y\":114.45559692382812},{\"x\":67.63285064697266,\"y\":114.45559692382812},{\"x\":67.72575378417969,\"y\":114.45559692382812},{\"x\":67.81864929199219,\"y\":114.45559692382812},{\"x\":67.91155242919922,\"y\":114.45559692382812},{\"x\":68.00445556640625,\"y\":114.45559692382812},{\"x\":68.09735870361328,\"y\":114.45559692382812},{\"x\":68.19026184082031,\"y\":114.45559692382812},{\"x\":68.28316497802734,\"y\":114.45559692382812},{\"x\":68.37606811523438,\"y\":114.45559692382812},{\"x\":68.4689712524414,\"y\":114.45559692382812},{\"x\":68.56187438964844,\"y\":114.45559692382812},{\"x\":68.65476989746094,\"y\":114.45559692382812},{\"x\":68.74767303466797,\"y\":114.45559692382812},{\"x\":68.840576171875,\"y\":114.45559692382812},{\"x\":68.93347930908203,\"y\":114.45559692382812},{\"x\":69.02638244628906,\"y\":114.45559692382812},{\"x\":69.1192855834961,\"y\":114.45559692382812},{\"x\":69.21218872070312,\"y\":114.45559692382812},{\"x\":69.30509185791016,\"y\":114.45559692382812},{\"x\":69.39799499511719,\"y\":114.45559692382812},{\"x\":69.49089050292969,\"y\":114.45559692382812},{\"x\":69.58379364013672,\"y\":114.45559692382812},{\"x\":69.67669677734375,\"y\":114.45559692382812},{\"x\":69.76959991455078,\"y\":114.45559692382812},{\"x\":69.86250305175781,\"y\":114.45559692382812},{\"x\":69.95540618896484,\"y\":114.45559692382812},{\"x\":70.04830932617188,\"y\":114.45559692382812},{\"x\":70.1412124633789,\"y\":114.45559692382812},{\"x\":70.2341079711914,\"y\":114.45559692382812},{\"x\":70.32701110839844,\"y\":114.45559692382812},{\"x\":70.41991424560547,\"y\":114.45559692382812},{\"x\":70.5128173828125,\"y\":114.45559692382812},{\"x\":70.60572052001953,\"y\":114.45559692382812},{\"x\":70.69862365722656,\"y\":114.45559692382812},{\"x\":70.7915267944336,\"y\":114.45559692382812},{\"x\":70.88442993164062,\"y\":114.45559692382812},{\"x\":70.97732543945312,\"y\":114.45559692382812},{\"x\":71.07022857666016,\"y\":114.45559692382812},{\"x\":71.16313171386719,\"y\":114.45559692382812},{\"x\":71.25603485107422,\"y\":114.45559692382812},{\"x\":71.34893798828125,\"y\":114.45559692382812}]]",
600,
382,
81,
@@ -499,21 +459,20 @@
"list",
0,
1,
null,
null,
"",
null
]
},
{
"id": 44,
"type": "VHS_VideoCombine",
"pos": {
"0": 2241.138427734375,
"1": 1051.91943359375
},
"pos": [
2241.138427734375,
1051.91943359375
],
"size": [
530,
650.4
650.4000244140625
],
"flags": {},
"order": 12,
@@ -521,36 +480,35 @@
"inputs": [
{
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 237,
"shape": 7
"link": 237
},
{
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null,
"shape": 7
"link": null
},
{
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null,
"shape": 7
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null,
"shape": 7
"link": null
}
],
"outputs": [
{
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null,
"slot_index": 0,
"shape": 3
"links": null
}
],
"title": "Trajectory Outputs",
@@ -584,10 +542,10 @@
{
"id": 115,
"type": "VHS_VideoCombine",
"pos": {
"0": 2819,
"1": 1056
},
"pos": [
2819,
1056
],
"size": [
530,
310
@@ -598,36 +556,35 @@
"inputs": [
{
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 264,
"shape": 7
"link": 264
},
{
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null,
"shape": 7
"link": null
},
{
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null,
"shape": 7
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null,
"shape": 7
"link": null
}
],
"outputs": [
{
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null,
"slot_index": 0,
"shape": 3
"links": null
}
],
"title": "Video with Trajectory Outputs",
@@ -661,13 +618,13 @@
{
"id": 126,
"type": "WanFunV2VSampler",
"pos": {
"0": 902,
"1": 60
},
"pos": [
902,
60
],
"size": [
428.4000244140625,
486
506
],
"flags": {},
"order": 13,
@@ -690,42 +647,39 @@
},
{
"name": "validation_video",
"shape": 7,
"type": "IMAGE",
"link": null,
"shape": 7
"link": null
},
{
"name": "control_video",
"shape": 7,
"type": "IMAGE",
"link": 284,
"shape": 7
"link": 284
},
{
"name": "start_image",
"shape": 7,
"type": "IMAGE",
"link": 285,
"shape": 7
"link": 285
},
{
"name": "ref_image",
"shape": 7,
"type": "IMAGE",
"link": null,
"shape": 7
},
{
"name": "riflex_k",
"type": "RIFLEXT_ARGS",
"link": null,
"shape": 7
"link": null
},
{
"name": "camera_conditions",
"shape": 7,
"type": "STRING",
"link": null,
"widget": {
"name": "camera_conditions"
},
"shape": 7
"link": null
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
@@ -748,7 +702,7 @@
"fixed",
50,
6,
1.0,
1,
"Flow",
0.1,
true,
@@ -760,10 +714,10 @@
{
"id": 106,
"type": "VHS_VideoCombine",
"pos": {
"0": 1390,
"1": 61
},
"pos": [
1390,
61
],
"size": [
390,
310
@@ -773,42 +727,40 @@
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 286,
"slot_index": 0,
"label": "图像",
"shape": 7
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 286
},
{
"name": "audio",
"type": "AUDIO",
"link": null,
"label": "音频",
"shape": 7
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"name": "meta_batch",
"type": "VHS_BatchManager",
"link": null,
"label": "批次管理",
"shape": 7
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null,
"shape": 7
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null,
"slot_index": 0,
"shape": 3,
"label": "文件名"
"links": null
}
],
"properties": {
@@ -840,26 +792,26 @@
{
"id": 123,
"type": "LoadWanFunModel",
"pos": {
"0": 281,
"1": 251
},
"size": {
"0": 315,
"1": 154
},
"pos": [
281,
251
],
"size": [
315,
154
],
"flags": {},
"order": 9,
"order": 8,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"slot_index": 0,
"links": [
281
],
"slot_index": 0
]
}
],
"properties": {
@@ -872,6 +824,31 @@
"wan2.1/wan_civitai.yaml",
"bf16"
]
},
{
"id": 112,
"type": "Note",
"pos": [
-203,
252
],
"size": [
427.074951171875,
143.9142608642578
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 1.3B model, you can set GPU_memory_mode to model_cpu_offload for faster generation. When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用1.3B模型时，可以设置GPU_memory_mode为model_cpu_offload进行更快速度的生成，在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
}
],
"links": [
@@ -888,7 +865,7 @@
97,
0,
95,
0,
1,
"MASK"
],
[
@@ -928,7 +905,7 @@
118,
0,
95,
1,
0,
"STRING"
],
[
@@ -990,7 +967,8 @@
],
"groups": [
{
"title": "Load EasyAnimate",
"id": 1,
"title": "Load Model",
"bounding": [
189,
160,
@@ -1002,6 +980,7 @@
"flags": {}
},
{
"id": 2,
"title": "Prompts",
"bounding": [
191,
@@ -1014,6 +993,7 @@
"flags": {}
},
{
"id": 3,
"title": "First Image of Trajectory",
"bounding": [
191,
@@ -1026,6 +1006,7 @@
"flags": {}
},
{
"id": 4,
"title": "Generate Control Video",
"bounding": [
786,
@@ -1041,18 +1022,19 @@
"config": {},
"extra": {
"ds": {
"scale": 0.6830134553650709,
"scale": 0.7513148009015782,
"offset": [
198.78755142053404,
143.5866129875247
119.78785424088606,
-47.62140429778734
]
},
"node_versions": {
"comfy-core": "v0.2.7-3-g8afb97c",
"ComfyUI-KJNodes": "4c5c26a2c91de356212419ac8bc7fcf9869527e9",
"CogVideoX-Fun": "717f0629175ad192927dc51ec95c4376816a4212",
"comfy-core": "0.3.44",
"ComfyUI-KJNodes": "ff49e1b01f10a14496b08e21bb89b64d2b15f333",
"CogVideoX-Fun": "e935795d7684aaba72be65d2ad5ef580c0526f7a",
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e"
}
},
"frontendVersion": "1.21.3"
},
"version": 0.4
}
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_v2v_control.json
+1 -1
View File
@@ -523,7 +523,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_v2v_control_canny.json
+1 -1
View File
@@ -653,7 +653,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_v2v_control_depth.json
+1 -1
View File
@@ -651,7 +651,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_v2v_control_depth_ref.json
+1 -1
View File
@@ -696,7 +696,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_v2v_control_pose.json
+1 -1
View File
@@ -651,7 +651,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_v2v_control_pose_ref.json
+1 -1
View File
@@ -696,7 +696,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
comfyui/wan2_1_fun/v1/wan2.1_fun_workflow_v2v_control_ref.json
+1 -1
View File
@@ -568,7 +568,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
comfyui/wan2_2/nodes.py
+138 -92
View File
@@ -18,9 +18,9 @@ from PIL import Image
from ...videox_fun.data.bucket_sampler import (ASPECT_RATIO_512,
get_closest_ratio)
from ...videox_fun.models import (AutoencoderKLWan, AutoTokenizer, CLIPModel,
from ...videox_fun.models import (AutoencoderKLWan, AutoencoderKLWan3_8, AutoTokenizer, CLIPModel,
WanT5EncoderModel, Wan2_2Transformer3DModel)
from ...videox_fun.pipeline import Wan2_2I2VPipeline, Wan2_2Pipeline
from ...videox_fun.pipeline import Wan2_2I2VPipeline, Wan2_2Pipeline, Wan2_2TI2VPipeline
from ...videox_fun.ui.controller import all_cheduler_dict
from ...videox_fun.utils.fp8_optimization import (
convert_model_weight_to_float8, convert_weight_dtype_wrapper, replace_parameters_by_name)
@@ -54,6 +54,7 @@ class LoadWan2_2Model:
[
'Wan2.2-T2V-A14B',
'Wan2.2-I2V-A14B',
'Wan2.2-TI2V-5B',
],
{
"default": 'Wan2.2-T2V-A14B',
@@ -69,6 +70,7 @@ class LoadWan2_2Model:
[
"wan2.2/wan_civitai_t2v.yaml",
"wan2.2/wan_civitai_i2v.yaml",
"wan2.2/wan_civitai_5b.yaml",
],
{
"default": "wan2.2/wan_civitai_t2v.yaml",
@@ -131,7 +133,12 @@ class LoadWan2_2Model:
print(f"- {os.path.join(eas_cache_dir, folder)}")
raise ValueError("Please download Fun model")
vae = AutoencoderKLWan.from_pretrained(
# Get Vae
Choosen_AutoencoderKL = {
"AutoencoderKLWan": AutoencoderKLWan,
"AutoencoderKLWan3_8": AutoencoderKLWan3_8
}[config['vae_kwargs'].get('vae_type', 'AutoencoderKLWan')]
vae = Choosen_AutoencoderKL.from_pretrained(
os.path.join(model_name, config['vae_kwargs'].get('vae_subpath', 'vae')),
additional_kwargs=OmegaConf.to_container(config['vae_kwargs']),
).to(weight_dtype)
@@ -153,13 +160,15 @@ class LoadWan2_2Model:
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
if config['transformer_additional_kwargs'].get('transformer_combination_type', 'single') == "moe":
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
else:
transformer_2 = None
# Update pbar
pbar.update(1)
@@ -180,46 +189,59 @@ class LoadWan2_2Model:
# Get pipeline
model_type = "Inpaint"
if model_type == "Inpaint":
if transformer.config.in_channels != vae.config.latent_channels:
pipeline = Wan2_2I2VPipeline(
transformer=transformer,
transformer_2=transformer_2,
if "wan_civitai_5b" in config_path:
pipeline = Wan2_2TI2VPipeline(
vae=vae,
tokenizer=tokenizer,
text_encoder=text_encoder,
transformer=transformer,
transformer_2=transformer_2,
scheduler=scheduler,
)
else:
pipeline = Wan2_2Pipeline(
transformer=transformer,
transformer_2=transformer_2,
vae=vae,
tokenizer=tokenizer,
text_encoder=text_encoder,
scheduler=scheduler,
)
if transformer.config.in_channels != vae.config.latent_channels:
pipeline = Wan2_2I2VPipeline(
transformer=transformer,
transformer_2=transformer_2,
vae=vae,
tokenizer=tokenizer,
text_encoder=text_encoder,
scheduler=scheduler,
)
else:
pipeline = Wan2_2Pipeline(
transformer=transformer,
transformer_2=transformer_2,
vae=vae,
tokenizer=tokenizer,
text_encoder=text_encoder,
scheduler=scheduler,
)
else:
raise ValueError(f"Model type {model_type} not supported")
if GPU_memory_mode == "sequential_cpu_offload":
replace_parameters_by_name(transformer, ["modulation",], device=device)
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer.freqs = transformer.freqs.to(device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
if transformer_2 is not None:
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
pipeline.enable_sequential_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload":
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_full_load_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.to(device=device)
else:
pipeline.to(device=device)
@@ -366,14 +388,18 @@ class Wan2_2T2VSampler:
pipeline.transformer.enable_teacache(
coefficients, steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
else:
pipeline.transformer.disable_teacache()
if pipeline.transformer_2 is not None:
pipeline.transformer_2.disable_teacache()
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, steps)
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator= torch.Generator(device).manual_seed(seed)
@@ -384,7 +410,8 @@ class Wan2_2T2VSampler:
if riflex_k > 0:
latent_frames = (video_length - 1) // pipeline.vae.config.temporal_compression_ratio + 1
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
# Apply lora
if funmodels.get("lora_cache", False):
@@ -395,13 +422,6 @@ class Wan2_2T2VSampler:
transformer_state_dict = pipeline.transformer.state_dict()
for key in transformer_state_dict:
transformer_cpu_cache[key] = transformer_state_dict[key].clone().cpu()
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_path_now = str(funmodels.get("loras", []) + funmodels.get("strength_model", []))
if lora_path_now != lora_path_before:
@@ -411,31 +431,43 @@ class Wan2_2T2VSampler:
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if pipeline.transformer_2 is not None:
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
else:
print('Merge Lora')
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_cpu_cache)
transformer_cpu_cache = {}
lora_path_before = ""
gc.collect()
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
print('Merge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
# Clear lora when switch from lora_cache=True to lora_cache=False.
if pipeline.transformer_2 is not None:
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer_2.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
sample = pipeline(
prompt,
@@ -455,8 +487,9 @@ class Wan2_2T2VSampler:
print('Unmerge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if pipeline.transformer_2 is not None:
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
return (videos,)
@@ -541,14 +574,6 @@ class Wan2_2I2VSampler:
mm.soft_empty_cache()
gc.collect()
start_img = [to_pil(_start_img) for _start_img in start_img] if start_img is not None else None
end_img = [to_pil(_end_img) for _end_img in end_img] if end_img is not None else None
# Count most suitable height and width
aspect_ratio_sample_size = {key : [x / 512 * base_resolution for x in ASPECT_RATIO_512[key]] for key in ASPECT_RATIO_512.keys()}
original_width, original_height = start_img[0].size if type(start_img) is list else Image.open(start_img).size
closest_size, closest_ratio = get_closest_ratio(original_height, original_width, ratios=aspect_ratio_sample_size)
height, width = [int(x / 16) * 16 for x in closest_size]
# Get Pipeline
pipeline = funmodels['pipeline']
@@ -556,6 +581,15 @@ class Wan2_2I2VSampler:
config = funmodels['config']
weight_dtype = funmodels['dtype']
start_img = [to_pil(_start_img) for _start_img in start_img] if start_img is not None else None
end_img = [to_pil(_end_img) for _end_img in end_img] if end_img is not None else None
# Count most suitable height and width
spatial_compression_ratio = pipeline.vae.config.spatial_compression_ratio if hasattr(pipeline.vae.config, "spatial_compression_ratio") else 8
aspect_ratio_sample_size = {key : [x / 512 * base_resolution for x in ASPECT_RATIO_512[key]] for key in ASPECT_RATIO_512.keys()}
original_width, original_height = start_img[0].size if type(start_img) is list else Image.open(start_img).size
closest_size, closest_ratio = get_closest_ratio(original_height, original_width, ratios=aspect_ratio_sample_size)
height, width = [int(x / spatial_compression_ratio / 2) * spatial_compression_ratio * 2 for x in closest_size]
# Get boundary for wan
boundary = config['transformer_additional_kwargs'].get('boundary', 0.900)
@@ -567,12 +601,18 @@ class Wan2_2I2VSampler:
pipeline.transformer.enable_teacache(
coefficients, steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
else:
pipeline.transformer.disable_teacache()
if pipeline.transformer_2 is not None:
pipeline.transformer_2.disable_teacache()
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, steps)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator= torch.Generator(device).manual_seed(seed)
@@ -583,7 +623,8 @@ class Wan2_2I2VSampler:
if riflex_k > 0:
latent_frames = (video_length - 1) // pipeline.vae.config.temporal_compression_ratio + 1
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
# Apply lora
if funmodels.get("lora_cache", False):
@@ -594,13 +635,6 @@ class Wan2_2I2VSampler:
transformer_state_dict = pipeline.transformer.state_dict()
for key in transformer_state_dict:
transformer_cpu_cache[key] = transformer_state_dict[key].clone().cpu()
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_path_now = str(funmodels.get("loras", []) + funmodels.get("strength_model", []))
if lora_path_now != lora_path_before:
@@ -610,31 +644,43 @@ class Wan2_2I2VSampler:
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if pipeline.transformer_2 is not None:
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
else:
print('Merge Lora')
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_cpu_cache)
transformer_cpu_cache = {}
lora_path_before = ""
gc.collect()
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
print('Merge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
# Clear lora when switch from lora_cache=True to lora_cache=False.
if pipeline.transformer_2 is not None:
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer_2.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
sample = pipeline(
prompt,
@@ -657,7 +703,7 @@ class Wan2_2I2VSampler:
print('Unmerge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
return (videos,)
if pipeline.transformer_2 is not None:
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
return (videos,)
comfyui/wan2_2/v1/wan2.2_workflow_i2v_5b.json
+471
View File
@@ -0,0 +1,471 @@
{
"id": "ca87b2cd-bd4a-4f31-82ec-e5e028f8848c",
"revision": 0,
"last_node_id": 103,
"last_link_id": 80,
"nodes": [
{
"id": 87,
"type": "LoadImage",
"pos": [
306,
495
],
"size": [
315,
314
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
79
]
},
{
"name": "MASK",
"type": "MASK",
"links": null
}
],
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"6.png",
"image"
]
},
{
"id": 78,
"type": "Note",
"pos": [
18,
-46
],
"size": [
210,
88
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can write prompt here\n（你可以在此填写提示词）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 95,
"type": "Note",
"pos": [
34,
550
],
"size": [
210,
88
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can upload image here\n（你可以在此上传图片）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 73,
"type": "FunTextBox",
"pos": [
250,
160
],
"size": [
383.7149963378906,
183.83506774902344
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
78
]
}
],
"title": "Negtive Prompt（反向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
]
},
{
"id": 80,
"type": "Note",
"pos": [
-75,
-297
],
"size": [
350.7127990722656,
125.54820251464844
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 17,
"type": "VHS_VideoCombine",
"pos": [
1277,
-70
],
"size": [
390,
577.7142944335938
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"label": "图像",
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 80
},
{
"label": "音频",
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"label": "批次管理",
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"slot_index": 0,
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 22,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "Fun_00107.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 16
}
}
}
},
{
"id": 99,
"type": "LoadWan2_2Model",
"pos": [
347.27996826171875,
-299.0150146484375
],
"size": [
276.705078125,
130
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"links": [
76
]
}
],
"properties": {
"Node name for S&R": "LoadWan2_2Model"
},
"widgets_values": [
"Wan2.2-TI2V-5B",
"sequential_cpu_offload",
"wan2.2/wan_civitai_5b.yaml",
"bf16"
]
},
{
"id": 103,
"type": "Wan2_2I2VSampler",
"pos": [
819.5313110351562,
-63.450443267822266
],
"size": [
325.5747985839844,
402
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "funmodels",
"type": "FunModels",
"link": 76
},
{
"name": "prompt",
"type": "STRING_PROMPT",
"link": 77
},
{
"name": "negative_prompt",
"type": "STRING_PROMPT",
"link": 78
},
{
"name": "start_img",
"shape": 7,
"type": "IMAGE",
"link": 79
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"links": [
80
]
}
],
"properties": {
"Node name for S&R": "Wan2_2I2VSampler"
},
"widgets_values": [
81,
960,
43,
"fixed",
50,
6,
"Flow",
0.1,
true,
5,
true,
0
]
},
{
"id": 75,
"type": "FunTextBox",
"pos": [
250,
-50
],
"size": [
383.54010009765625,
156.71620178222656
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
77
]
}
],
"title": "Positive Prompt（正向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"fireworks display over night city. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic."
]
}
],
"links": [
[
76,
99,
0,
103,
0,
"FunModels"
],
[
77,
75,
0,
103,
1,
"STRING_PROMPT"
],
[
78,
73,
0,
103,
2,
"STRING_PROMPT"
],
[
79,
87,
0,
103,
3,
"IMAGE"
],
[
80,
103,
0,
17,
0,
"IMAGE"
]
],
"groups": [
{
"id": 1,
"title": "Load Model",
"bounding": [
220,
-380,
472,
232
],
"color": "#b06634",
"font_size": 24,
"flags": {}
},
{
"id": 2,
"title": "Prompts",
"bounding": [
218,
-127,
450,
483
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 3,
"title": "Group",
"bounding": [
220,
409,
458,
436
],
"color": "#a1309b",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.6830134553650705,
"offset": [
282.91743113564746,
433.6498523638886
]
},
"frontendVersion": "1.21.3",
"workspace_info": {
"id": "776b62b4-bd17-4ed3-9923-b7aad000b1ea"
},
"node_versions": {
"comfy-core": "0.3.44",
"CogVideoX-Fun": "5f2a55692d0834e00d477939b818b9bb63c06535",
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e"
}
},
"version": 0.4
}
comfyui/wan2_2/v1/wan2.2_workflow_t2v_5b.json
+383
View File
@@ -0,0 +1,383 @@
{
"id": "ca87b2cd-bd4a-4f31-82ec-e5e028f8848c",
"revision": 0,
"last_node_id": 105,
"last_link_id": 84,
"nodes": [
{
"id": 78,
"type": "Note",
"pos": [
18,
-46
],
"size": [
210,
88
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can write prompt here\n（你可以在此填写提示词）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 73,
"type": "FunTextBox",
"pos": [
250,
160
],
"size": [
383.7149963378906,
183.83506774902344
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
83
]
}
],
"title": "Negtive Prompt（反向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
]
},
{
"id": 80,
"type": "Note",
"pos": [
-75,
-297
],
"size": [
350.7127990722656,
125.54820251464844
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 75,
"type": "FunTextBox",
"pos": [
250,
-50
],
"size": [
383.54010009765625,
156.71620178222656
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
82
]
}
],
"title": "Positive Prompt（正向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"fireworks display over night city. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic."
]
},
{
"id": 99,
"type": "LoadWan2_2Model",
"pos": [
347.27996826171875,
-299.0150146484375
],
"size": [
276.705078125,
130
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"links": [
81
]
}
],
"properties": {
"Node name for S&R": "LoadWan2_2Model"
},
"widgets_values": [
"Wan2.2-TI2V-5B",
"sequential_cpu_offload",
"wan2.2/wan_civitai_5b.yaml",
"bf16"
]
},
{
"id": 17,
"type": "VHS_VideoCombine",
"pos": [
1277,
-70
],
"size": [
390,
577.7142944335938
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"label": "图像",
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 84
},
{
"label": "音频",
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"label": "批次管理",
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"slot_index": 0,
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 22,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "Fun_00108.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 16
}
}
}
},
{
"id": 105,
"type": "Wan2_2FunT2VSampler",
"pos": [
827.6821899414062,
-69.17436981201172
],
"size": [
340.3540954589844,
430
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "funmodels",
"type": "FunModels",
"link": 81
},
{
"name": "prompt",
"type": "STRING_PROMPT",
"link": 82
},
{
"name": "negative_prompt",
"type": "STRING_PROMPT",
"link": 83
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"links": [
84
]
}
],
"properties": {
"Node name for S&R": "Wan2_2FunT2VSampler"
},
"widgets_values": [
81,
1280,
704,
false,
43,
"fixed",
50,
6,
"Flow",
0.1,
true,
5,
true,
0
]
}
],
"links": [
[
81,
99,
0,
105,
0,
"FunModels"
],
[
82,
75,
0,
105,
1,
"STRING_PROMPT"
],
[
83,
73,
0,
105,
2,
"STRING_PROMPT"
],
[
84,
105,
0,
17,
0,
"IMAGE"
]
],
"groups": [
{
"id": 1,
"title": "Load Model",
"bounding": [
220,
-380,
472,
232
],
"color": "#b06634",
"font_size": 24,
"flags": {}
},
{
"id": 2,
"title": "Prompts",
"bounding": [
218,
-127,
450,
483
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.6830134553650705,
"offset": [
313.88657762002225,
431.4308258013887
]
},
"frontendVersion": "1.21.3",
"workspace_info": {
"id": "776b62b4-bd17-4ed3-9923-b7aad000b1ea"
},
"node_versions": {
"CogVideoX-Fun": "5f2a55692d0834e00d477939b818b9bb63c06535",
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e"
}
},
"version": 0.4
}
comfyui/wan2_2_fun/nodes.py
+137 -95
View File
@@ -18,7 +18,7 @@ from PIL import Image
from ...videox_fun.data.bucket_sampler import (ASPECT_RATIO_512,
get_closest_ratio)
from ...videox_fun.models import (AutoencoderKLWan, AutoTokenizer, CLIPModel,
from ...videox_fun.models import (AutoencoderKLWan, AutoencoderKLWan3_8, AutoTokenizer, CLIPModel,
WanT5EncoderModel, Wan2_2Transformer3DModel)
from ...videox_fun.pipeline import Wan2_2FunInpaintPipeline, Wan2_2FunPipeline, Wan2_2FunControlPipeline
from ...videox_fun.ui.controller import all_cheduler_dict
@@ -48,6 +48,7 @@ class LoadWan2_2FunModel:
[
'Wan2.2-Fun-A14B-InP',
'Wan2.2-Fun-A14B-Control',
'Wan2.2-Fun-A14B-Control-Camera',
],
{
"default": 'Wan2.2-Fun-A14B-InP',
@@ -68,6 +69,7 @@ class LoadWan2_2FunModel:
"config": (
[
"wan2.2/wan_civitai_i2v.yaml",
"wan2.2/wan_civitai_5b.yaml",
],
{
"default": "wan2.2/wan_civitai_i2v.yaml",
@@ -129,7 +131,11 @@ class LoadWan2_2FunModel:
print(f"- {os.path.join(eas_cache_dir, folder)}")
raise ValueError("Please download Fun model")
vae = AutoencoderKLWan.from_pretrained(
Choosen_AutoencoderKL = {
"AutoencoderKLWan": AutoencoderKLWan,
"AutoencoderKLWan3_8": AutoencoderKLWan3_8
}[config['vae_kwargs'].get('vae_type', 'AutoencoderKLWan')]
vae = Choosen_AutoencoderKL.from_pretrained(
os.path.join(model_name, config['vae_kwargs'].get('vae_subpath', 'vae')),
additional_kwargs=OmegaConf.to_container(config['vae_kwargs']),
).to(weight_dtype)
@@ -357,21 +363,24 @@ class Wan2_2FunT2VSampler:
# Load Sampler
pipeline.scheduler = all_cheduler_dict[scheduler](**filter_kwargs(all_cheduler_dict[scheduler], OmegaConf.to_container(config['scheduler_kwargs'])))
coefficients = get_teacache_coefficients(model_name) if enable_teacache else None
if coefficients is not None:
print(f"Enable TeaCache with threshold {teacache_threshold} and skip the first {num_skip_start_steps} steps.")
pipeline.transformer.enable_teacache(
coefficients, steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
else:
pipeline.transformer.disable_teacache()
if pipeline.transformer_2 is not None:
pipeline.transformer_2.disable_teacache()
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, steps)
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator= torch.Generator(device).manual_seed(seed)
@@ -382,7 +391,10 @@ class Wan2_2FunT2VSampler:
if riflex_k > 0:
latent_frames = (video_length - 1) // pipeline.vae.config.temporal_compression_ratio + 1
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
input_video, input_video_mask, clip_image = get_image_to_video_latent(None, None, video_length=video_length, sample_size=(height, width))
# Apply lora
if funmodels.get("lora_cache", False):
@@ -393,13 +405,6 @@ class Wan2_2FunT2VSampler:
transformer_state_dict = pipeline.transformer.state_dict()
for key in transformer_state_dict:
transformer_cpu_cache[key] = transformer_state_dict[key].clone().cpu()
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_path_now = str(funmodels.get("loras", []) + funmodels.get("strength_model", []))
if lora_path_now != lora_path_before:
@@ -409,31 +414,43 @@ class Wan2_2FunT2VSampler:
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if pipeline.transformer_2 is not None:
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
else:
print('Merge Lora')
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_cpu_cache)
transformer_cpu_cache = {}
lora_path_before = ""
gc.collect()
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
print('Merge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
# Clear lora when switch from lora_cache=True to lora_cache=False.
if pipeline.transformer_2 is not None:
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer_2.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
sample = pipeline(
prompt,
@@ -444,6 +461,9 @@ class Wan2_2FunT2VSampler:
generator = generator,
guidance_scale = cfg,
num_inference_steps = steps,
video = input_video,
mask_video = input_video_mask,
boundary = boundary,
comfyui_progressbar = True,
).videos
@@ -453,8 +473,9 @@ class Wan2_2FunT2VSampler:
print('Unmerge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if pipeline.transformer_2 is not None:
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
return (videos,)
@@ -541,6 +562,12 @@ class Wan2_2FunInpaintSampler:
mm.soft_empty_cache()
gc.collect()
# Get Pipeline
pipeline = funmodels['pipeline']
model_name = funmodels['model_name']
config = funmodels['config']
weight_dtype = funmodels['dtype']
start_img = [to_pil(_start_img) for _start_img in start_img] if start_img is not None else None
end_img = [to_pil(_end_img) for _end_img in end_img] if end_img is not None else None
@@ -549,33 +576,30 @@ class Wan2_2FunInpaintSampler:
original_width, original_height = start_img[0].size if type(start_img) is list else Image.open(start_img).size
closest_size, closest_ratio = get_closest_ratio(original_height, original_width, ratios=aspect_ratio_sample_size)
height, width = [int(x / 16) * 16 for x in closest_size]
# Get Pipeline
pipeline = funmodels['pipeline']
model_name = funmodels['model_name']
config = funmodels['config']
weight_dtype = funmodels['dtype']
# Get boundary for wan
boundary = config['transformer_additional_kwargs'].get('boundary', 0.900)
# Load Sampler
pipeline.scheduler = all_cheduler_dict[scheduler](**filter_kwargs(all_cheduler_dict[scheduler], OmegaConf.to_container(config['scheduler_kwargs'])))
coefficients = get_teacache_coefficients(model_name) if enable_teacache else None
if coefficients is not None:
print(f"Enable TeaCache with threshold {teacache_threshold} and skip the first {num_skip_start_steps} steps.")
pipeline.transformer.enable_teacache(
coefficients, steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
else:
pipeline.transformer.disable_teacache()
if pipeline.transformer_2 is not None:
pipeline.transformer_2.disable_teacache()
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, steps)
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator= torch.Generator(device).manual_seed(seed)
@@ -585,7 +609,8 @@ class Wan2_2FunInpaintSampler:
if riflex_k > 0:
latent_frames = (video_length - 1) // pipeline.vae.config.temporal_compression_ratio + 1
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
input_video, input_video_mask, clip_image = get_image_to_video_latent(start_img, end_img, video_length=video_length, sample_size=(height, width))
@@ -598,13 +623,6 @@ class Wan2_2FunInpaintSampler:
transformer_state_dict = pipeline.transformer.state_dict()
for key in transformer_state_dict:
transformer_cpu_cache[key] = transformer_state_dict[key].clone().cpu()
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_path_now = str(funmodels.get("loras", []) + funmodels.get("strength_model", []))
if lora_path_now != lora_path_before:
@@ -614,31 +632,43 @@ class Wan2_2FunInpaintSampler:
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if pipeline.transformer_2 is not None:
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
else:
print('Merge Lora')
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_cpu_cache)
transformer_cpu_cache = {}
lora_path_before = ""
gc.collect()
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
print('Merge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
# Clear lora when switch from lora_cache=True to lora_cache=False.
if pipeline.transformer_2 is not None:
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer_2.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
sample = pipeline(
prompt,
@@ -661,8 +691,9 @@ class Wan2_2FunInpaintSampler:
print('Unmerge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if pipeline.transformer_2 is not None:
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
return (videos,)
@@ -806,14 +837,18 @@ class Wan2_2FunV2VSampler:
pipeline.transformer.enable_teacache(
coefficients, steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
else:
pipeline.transformer.disable_teacache()
if pipeline.transformer_2 is not None:
pipeline.transformer_2.disable_teacache()
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, steps)
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator= torch.Generator(device).manual_seed(seed)
@@ -823,7 +858,8 @@ class Wan2_2FunV2VSampler:
if riflex_k > 0:
latent_frames = (video_length - 1) // pipeline.vae.config.temporal_compression_ratio + 1
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
if pipeline.transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
if model_type == "Inpaint":
input_video, input_video_mask, ref_image, clip_image = get_video_to_video_latent(validation_video, video_length=video_length, sample_size=(height, width), fps=16, ref_image=ref_image[0] if ref_image is not None else ref_image)
@@ -860,13 +896,6 @@ class Wan2_2FunV2VSampler:
transformer_state_dict = pipeline.transformer.state_dict()
for key in transformer_state_dict:
transformer_cpu_cache[key] = transformer_state_dict[key].clone().cpu()
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_path_now = str(funmodels.get("loras", []) + funmodels.get("strength_model", []))
if lora_path_now != lora_path_before:
@@ -876,31 +905,43 @@ class Wan2_2FunV2VSampler:
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if pipeline.transformer_2 is not None:
# Save the original weights to cpu
if len(transformer_high_cpu_cache) == 0:
print('Save transformer high state_dict to cpu memory')
transformer_high_state_dict = pipeline.transformer_2.state_dict()
for key in transformer_high_state_dict:
transformer_high_cpu_cache[key] = transformer_high_state_dict[key].clone().cpu()
lora_high_path_now = str(funmodels.get("loras_high", []) + funmodels.get("strength_model", []))
if lora_high_path_now != lora_high_path_before:
print('Merge Lora High with Cache')
lora_high_path_before = copy.deepcopy(lora_high_path_now)
pipeline.transformer_2.load_state_dict(transformer_cpu_cache)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
else:
print('Merge Lora')
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_cpu_cache)
transformer_cpu_cache = {}
lora_path_before = ""
gc.collect()
# Clear lora when switch from lora_cache=True to lora_cache=False.
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
print('Merge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
# Clear lora when switch from lora_cache=True to lora_cache=False.
if pipeline.transformer_2 is not None:
if len(transformer_high_cpu_cache) != 0:
pipeline.transformer_2.load_state_dict(transformer_high_cpu_cache)
transformer_high_cpu_cache = {}
lora_high_path_before = ""
gc.collect()
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = merge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
if model_type == "Inpaint":
sample = pipeline(
@@ -944,6 +985,7 @@ class Wan2_2FunV2VSampler:
print('Unmerge Lora')
for _lora_path, _lora_weight in zip(funmodels.get("loras", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype)
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
return (videos,)
if pipeline.transformer_2 is not None:
for _lora_path, _lora_weight in zip(funmodels.get("loras_high", []), funmodels.get("strength_model", [])):
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight, device="cuda", dtype=weight_dtype, sub_transformer_name="transformer_2")
return (videos,)
comfyui/wan2_2_fun/v1/wan2.2_fun_workflow_control_camera.json
+670
View File
@@ -0,0 +1,670 @@
{
"id": "f2736b24-5147-427c-a9dd-036fd3053b5d",
"revision": 0,
"last_node_id": 135,
"last_link_id": 300,
"nodes": [
{
"id": 107,
"type": "Note",
"pos": [
4,
634
],
"size": [
210,
88
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can write prompt here\n（你可以在此填写提示词）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 108,
"type": "Note",
"pos": [
-110,
842
],
"size": [
326.1556091308594,
145.20904541015625
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"Using longer neg prompt such as \"Blurring, mutation, deformation, distortion, dark and solid, comics.\" can increase stability. Adding words such as \"quiet, solid\" to the neg prompt can increase dynamism.\n(使用更长的neg prompt如\"模糊，突变，变形，失真，画面暗，画面固定，连环画，漫画，线稿，没有主体。\"，可以增加稳定性。在neg prompt中添加\"安静，固定\"等词语可以增加动态性。)"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 122,
"type": "FunTextBox",
"pos": [
238,
805
],
"size": [
400,
200
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
297
]
}
],
"title": "Negtive Prompt（反向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
]
},
{
"id": 129,
"type": "CameraBasicFromChaoJie",
"pos": [
805.2059326171875,
1012.381103515625
],
"size": [
315,
106
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CameraPose",
"type": "CameraPose",
"links": null
}
],
"properties": {
"Node name for S&R": "CameraBasicFromChaoJie"
},
"widgets_values": [
"Static",
1,
16
]
},
{
"id": 130,
"type": "CameraTrajectoryFromChaoJie",
"pos": [
1170.206298828125,
763.3814697265625
],
"size": [
367.79998779296875,
150
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "camera_pose",
"type": "CameraPose",
"link": 285
}
],
"outputs": [
{
"name": "camera_trajectory",
"type": "STRING",
"slot_index": 0,
"links": [
299
]
},
{
"name": "video_length",
"type": "INT",
"links": null
}
],
"properties": {
"Node name for S&R": "CameraTrajectoryFromChaoJie"
},
"widgets_values": [
0.532139961,
0.946026558,
0.5,
0.5
]
},
{
"id": 106,
"type": "VHS_VideoCombine",
"pos": [
1408,
68
],
"size": [
390,
310
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"label": "图像",
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 300
},
{
"label": "音频",
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"label": "批次管理",
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"slot_index": 0,
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 22,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "Fun_00112.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 16
}
}
}
},
{
"id": 121,
"type": "FunTextBox",
"pos": [
235,
539
],
"size": [
400,
200
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"links": [
296
]
}
],
"title": "Positive Prompt（正向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"Fireworks light up the evening sky over a sprawling cityscape with gothic-style buildings featuring pointed towers and clock faces. The city is lit by both artificial lights from the buildings and the colorful bursts of the fireworks. The scene is viewed from an elevated angle, showcasing a vibrant urban environment set against a backdrop of a dramatic, partially cloudy sky at dusk."
]
},
{
"id": 100,
"type": "LoadImage",
"pos": [
237.59738159179688,
1164.597412109375
],
"size": [
378.07147216796875,
314
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"label": "图像",
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
298
]
},
{
"label": "遮罩",
"name": "MASK",
"type": "MASK",
"links": null
}
],
"title": "Start Image（图片到视频的开始图片）",
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"5.png",
"image"
]
},
{
"id": 131,
"type": "CameraCombineFromChaoJie",
"pos": [
814.2059326171875,
763.3814697265625
],
"size": [
315,
178
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CameraPose",
"type": "CameraPose",
"links": [
285
]
}
],
"properties": {
"Node name for S&R": "CameraCombineFromChaoJie"
},
"widgets_values": [
"Pan Right",
"Pan Up",
"Static",
"Static",
1,
81
]
},
{
"id": 110,
"type": "Note",
"pos": [
1158.206298828125,
970.381103515625
],
"size": [
608.1410522460938,
188.2682342529297
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"CameraCombine is used to combine multiple camera movements, while CameraBasic produces a single camera movement. The nodes come from https://github.com/chaojie/ComfyUI-CameraCtrl-Wrapper/. Since ComfyUI-CameraCtrl-Wrapper requires a specific version of diffusers, the code has been copied into the current repository.\n(CameraCombine用于组合多个镜头运动，CameraBasic产出单个镜头运动；节点来自于https://github.com/chaojie/ComfyUI-CameraCtrl-Wrapper/，由于ComfyUI-CameraCtrl-Wrapper有具体diffusers版本要求，故复制代码到当前库中。)"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 112,
"type": "Note",
"pos": [
-203,
252
],
"size": [
427.074951171875,
143.9142608642578
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 134,
"type": "LoadWan2_2FunModel",
"pos": [
273.49224853515625,
249.6970672607422
],
"size": [
350.6878662109375,
155.20101928710938
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"links": [
295
]
}
],
"properties": {
"Node name for S&R": "LoadWan2_2FunModel"
},
"widgets_values": [
"Wan2.2-Fun-A14B-Control-Camera",
"Control",
"sequential_cpu_offload",
"wan2.2/wan_civitai_i2v.yaml",
"bf16"
]
},
{
"id": 135,
"type": "Wan2_2FunV2VSampler",
"pos": [
923.0663452148438,
107.64505767822266
],
"size": [
350.2320251464844,
526
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "funmodels",
"type": "FunModels",
"link": 295
},
{
"name": "prompt",
"type": "STRING_PROMPT",
"link": 296
},
{
"name": "negative_prompt",
"type": "STRING_PROMPT",
"link": 297
},
{
"name": "validation_video",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "control_video",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "start_image",
"shape": 7,
"type": "IMAGE",
"link": 298
},
{
"name": "end_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "ref_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "camera_conditions",
"shape": 7,
"type": "STRING",
"link": 299
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"links": [
300
]
}
],
"properties": {
"Node name for S&R": "Wan2_2FunV2VSampler"
},
"widgets_values": [
81,
640,
43,
"fixed",
50,
6.000000000000001,
1,
"Flow",
0.1,
true,
5,
true,
0
]
}
],
"links": [
[
285,
131,
0,
130,
0,
"CameraPose"
],
[
295,
134,
0,
135,
0,
"FunModels"
],
[
296,
121,
0,
135,
1,
"STRING_PROMPT"
],
[
297,
122,
0,
135,
2,
"STRING_PROMPT"
],
[
298,
100,
0,
135,
5,
"IMAGE"
],
[
299,
130,
0,
135,
8,
"STRING"
],
[
300,
135,
0,
106,
0,
"IMAGE"
]
],
"groups": [
{
"id": 1,
"title": "Generate Control Video",
"bounding": [
773,
666,
1025,
531
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 2,
"title": "First Image",
"bounding": [
191,
1068,
475,
456
],
"color": "#a1309b",
"font_size": 24,
"flags": {}
},
{
"id": 3,
"title": "Prompts",
"bounding": [
191,
456,
475,
587
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 4,
"title": "Load Model",
"bounding": [
189,
160,
475,
269
],
"color": "#b06634",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.6830134553650709,
"offset": [
246.58945220178464,
51.36424150314978
]
},
"node_versions": {
"CogVideoX-Fun": "e935795d7684aaba72be65d2ad5ef580c0526f7a",
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e",
"comfy-core": "0.3.44"
},
"frontendVersion": "1.21.3"
},
"version": 0.4
}
comfyui/wan2_2_fun/v1/wan2.2_fun_workflow_control_trajectory.json
+1045
View File
File diff suppressed because it is too large Load Diff
comfyui/wan2_2_fun/v1/wan2.2_fun_workflow_t2v.json
+409
View File
@@ -0,0 +1,409 @@
{
"id": "8d9a378f-1cac-4610-8858-351d5982a6ab",
"revision": 0,
"last_node_id": 103,
"last_link_id": 73,
"nodes": [
{
"id": 75,
"type": "FunTextBox",
"pos": [
250,
-50
],
"size": [
383.54010009765625,
156.71620178222656
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
72
]
}
],
"title": "Positive Prompt（正向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"fireworks display over night city. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic."
]
},
{
"id": 78,
"type": "Note",
"pos": [
18,
-46
],
"size": [
210,
88
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can write prompt here\n（你可以在此填写提示词）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 94,
"type": "Note",
"pos": [
17,
-35
],
"size": [
210,
88
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can write prompt here\n（你可以在此填写提示词）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 80,
"type": "Note",
"pos": [
-95.01139068603516,
-334.30706787109375
],
"size": [
355.636474609375,
132.4238739013672
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 73,
"type": "FunTextBox",
"pos": [
250,
160
],
"size": [
383.7149963378906,
183.83506774902344
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
73
]
}
],
"title": "Negtive Prompt（反向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
]
},
{
"id": 17,
"type": "VHS_VideoCombine",
"pos": [
1257.373291015625,
-146.444091796875
],
"size": [
390,
537.4615478515625
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"label": "图像",
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 70
},
{
"label": "音频",
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"label": "批次管理",
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"slot_index": 0,
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 22,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "Fun_00008.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 16
}
}
}
},
{
"id": 99,
"type": "LoadWan2_2FunModel",
"pos": [
290,
-334
],
"size": [
315,
154
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"links": [
71
]
}
],
"properties": {
"Node name for S&R": "LoadWan2_2FunModel"
},
"widgets_values": [
"Wan2.2-Fun-A14B-InP",
"Inpaint",
"sequential_cpu_offload",
"wan2.2/wan_civitai_i2v.yaml",
"bf16"
]
},
{
"id": 103,
"type": "Wan2_2FunT2VSampler",
"pos": [
819.128173828125,
-145.90399169921875
],
"size": [
340.3540954589844,
430
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "funmodels",
"type": "FunModels",
"link": 71
},
{
"name": "prompt",
"type": "STRING_PROMPT",
"link": 72
},
{
"name": "negative_prompt",
"type": "STRING_PROMPT",
"link": 73
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"links": [
70
]
}
],
"properties": {
"Node name for S&R": "Wan2_2FunT2VSampler"
},
"widgets_values": [
81,
832,
480,
false,
43,
"fixed",
50,
6,
"Flow",
0.1,
true,
5,
true,
0
]
}
],
"links": [
[
70,
103,
0,
17,
0,
"IMAGE"
],
[
71,
99,
0,
103,
0,
"FunModels"
],
[
72,
75,
0,
103,
1,
"STRING_PROMPT"
],
[
73,
73,
0,
103,
2,
"STRING_PROMPT"
]
],
"groups": [
{
"id": 2,
"title": "Prompts",
"bounding": [
218,
-127,
450,
483
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 3,
"title": "Load Model",
"bounding": [
227,
-416,
469,
256
],
"color": "#b06634",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.8264462809917354,
"offset": [
175.2685031365507,
463.74646895767313
]
},
"workspace_info": {
"id": "776b62b4-bd17-4ed3-9923-b7aad000b1ea"
},
"node_versions": {
"CogVideoX-Fun": "5f2a55692d0834e00d477939b818b9bb63c06535",
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e"
},
"frontendVersion": "1.21.3"
},
"version": 0.4
}
comfyui/wan2_2_fun/v1/wan2.2_fun_workflow_v2v_control.json
+1 -1
View File
@@ -529,7 +529,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
comfyui/wan2_2_fun/v1/wan2.2_fun_workflow_v2v_control_canny.json
+700
View File
@@ -0,0 +1,700 @@
{
"id": "90329caf-a94f-48a6-80d7-d6e167a8b1e3",
"revision": 0,
"last_node_id": 102,
"last_link_id": 78,
"nodes": [
{
"id": 78,
"type": "Note",
"pos": [
18,
-46
],
"size": [
210,
88
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can write prompt here\n（你可以在此填写提示词）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 79,
"type": "Note",
"pos": [
15.739953994750977,
462.38665771484375
],
"size": [
210,
88
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can upload video here\n（在此上传视频）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 88,
"type": "Note",
"pos": [
-99,
197
],
"size": [
326.1556091308594,
145.20904541015625
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"Using longer neg prompt such as \"Blurring, mutation, deformation, distortion, dark and solid, comics.\" can increase stability. Adding words such as \"quiet, solid\" to the neg prompt can increase dynamism.\n(使用更长的neg prompt如\"模糊，突变，变形，失真，画面暗，画面固定，连环画，漫画，线稿，没有主体。\"，可以增加稳定性。在neg prompt中添加\"安静，固定\"等词语可以增加动态性。)"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 89,
"type": "Note",
"pos": [
-192,
-293
],
"size": [
427.074951171875,
143.9142608642578
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 92,
"type": "FunTextBox",
"pos": [
254,
-46
],
"size": [
380.845703125,
157.68350219726562
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
73
]
}
],
"title": "Positive Prompt（正向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"在这个阳光明媚的户外花园里，美女身穿一袭及膝的白色无袖连衣裙，裙摆在她轻盈的舞姿中轻柔地摆动，宛如一只翩翩起舞的蝴蝶。阳光透过树叶间洒下斑驳的光影，映衬出她柔和的脸庞和清澈的眼眸，显得格外优雅。仿佛每一个动作都在诉说着青春与活力，她在草地上旋转，裙摆随之飞扬，仿佛整个花园都因她的舞动而欢愉。周围五彩缤纷的花朵在微风中摇曳，玫瑰、菊花、百合，各自释放出阵阵香气，营造出一种轻松而愉快的氛围。"
]
},
{
"id": 99,
"type": "VHS_VideoCombine",
"pos": [
1094,
559
],
"size": [
315,
851.4834594726562
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 65
},
{
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun-Preprocess-Video",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 19,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "AnimateDiff_00001.gif",
"subfolder": "",
"type": "output",
"format": "image/gif",
"frame_rate": 8
}
}
}
},
{
"id": 94,
"type": "FunTextBox",
"pos": [
258,
178
],
"size": [
368.5529479980469,
159.4075927734375
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
74
]
}
],
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
]
},
{
"id": 97,
"type": "VideoToCanny",
"pos": [
729,
566
],
"size": [
315,
106
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "input_video",
"type": "IMAGE",
"link": 66
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"slot_index": 0,
"links": [
65,
75
]
}
],
"properties": {
"Node name for S&R": "VideoToCanny"
},
"widgets_values": [
100,
200,
81
]
},
{
"id": 101,
"type": "LoadWan2_2FunModel",
"pos": [
306.2298583984375,
-313.5478515625
],
"size": [
397.9565734863281,
154
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"links": [
78
]
}
],
"properties": {
"Node name for S&R": "LoadWan2_2FunModel"
},
"widgets_values": [
"Wan2.2-Fun-A14B-Control",
"Control",
"sequential_cpu_offload",
"wan2.2/wan_civitai_i2v.yaml",
"bf16"
]
},
{
"id": 17,
"type": "VHS_VideoCombine",
"pos": [
1465.6671142578125,
-174.95152282714844
],
"size": [
390.9534912109375,
966.9860229492188
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"label": "图像",
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 77
},
{
"label": "音频",
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"label": "批次管理",
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"slot_index": 0,
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 22,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "Fun_00004.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 16
}
}
}
},
{
"id": 85,
"type": "VHS_LoadVideo",
"pos": [
335,
476
],
"size": [
252.056640625,
409.87884521484375
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
66
]
},
{
"name": "frame_count",
"type": "INT",
"links": null
},
{
"name": "audio",
"type": "AUDIO",
"links": null
},
{
"name": "video_info",
"type": "VHS_VIDEOINFO",
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_LoadVideo"
},
"widgets_values": {
"video": "00000005.mp4",
"force_rate": 0,
"force_size": "Disabled",
"custom_width": 512,
"custom_height": 512,
"frame_load_cap": 0,
"skip_first_frames": 0,
"select_every_nth": 1,
"choose video to upload": "image",
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"frame_load_cap": 0,
"skip_first_frames": 0,
"force_rate": 0,
"filename": "00000005.mp4",
"type": "input",
"format": "video/mp4",
"select_every_nth": 1
}
}
}
},
{
"id": 102,
"type": "Wan2_2FunV2VSampler",
"pos": [
848.4100341796875,
-180.23484802246094
],
"size": [
458.36383056640625,
526
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "funmodels",
"type": "FunModels",
"link": 78
},
{
"name": "prompt",
"type": "STRING_PROMPT",
"link": 73
},
{
"name": "negative_prompt",
"type": "STRING_PROMPT",
"link": 74
},
{
"name": "validation_video",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "control_video",
"shape": 7,
"type": "IMAGE",
"link": 75
},
{
"name": "start_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "end_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "ref_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "camera_conditions",
"shape": 7,
"type": "STRING",
"link": null
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"links": [
77
]
}
],
"properties": {
"Node name for S&R": "Wan2_2FunV2VSampler"
},
"widgets_values": [
81,
640,
43,
"fixed",
50,
6.000000000000001,
1,
"Flow",
0.1,
true,
5,
true,
0
]
}
],
"links": [
[
65,
97,
0,
99,
0,
"IMAGE"
],
[
66,
85,
0,
97,
0,
"IMAGE"
],
[
73,
92,
0,
102,
1,
"STRING_PROMPT"
],
[
74,
94,
0,
102,
2,
"STRING_PROMPT"
],
[
75,
97,
0,
102,
4,
"IMAGE"
],
[
77,
102,
0,
17,
0,
"IMAGE"
],
[
78,
101,
0,
102,
0,
"FunModels"
]
],
"groups": [
{
"id": 1,
"title": "Upload Your Video",
"bounding": [
218,
385,
487,
789
],
"color": "#a1309b",
"font_size": 24,
"flags": {}
},
{
"id": 2,
"title": "Load Model",
"bounding": [
218,
-387,
542,
248
],
"color": "#b06634",
"font_size": 24,
"flags": {}
},
{
"id": 3,
"title": "Prompts",
"bounding": [
218,
-127,
450,
483
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.6830134553650709,
"offset": [
192.62803696741028,
413.1926211906501
]
},
"workspace_info": {
"id": "776b62b4-bd17-4ed3-9923-b7aad000b1ea"
},
"node_versions": {
"CogVideoX-Fun": "e935795d7684aaba72be65d2ad5ef580c0526f7a",
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e"
},
"frontendVersion": "1.21.3"
},
"version": 0.4
}
comfyui/wan2_2_fun/v1/wan2.2_fun_workflow_v2v_control_depth.json
+697
View File
@@ -0,0 +1,697 @@
{
"id": "90329caf-a94f-48a6-80d7-d6e167a8b1e3",
"revision": 0,
"last_node_id": 103,
"last_link_id": 83,
"nodes": [
{
"id": 78,
"type": "Note",
"pos": [
18,
-46
],
"size": [
210,
88
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can write prompt here\n（你可以在此填写提示词）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 79,
"type": "Note",
"pos": [
15.739953994750977,
462.38665771484375
],
"size": [
210,
88
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can upload video here\n（在此上传视频）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 88,
"type": "Note",
"pos": [
-99,
197
],
"size": [
326.1556091308594,
145.20904541015625
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"Using longer neg prompt such as \"Blurring, mutation, deformation, distortion, dark and solid, comics.\" can increase stability. Adding words such as \"quiet, solid\" to the neg prompt can increase dynamism.\n(使用更长的neg prompt如\"模糊，突变，变形，失真，画面暗，画面固定，连环画，漫画，线稿，没有主体。\"，可以增加稳定性。在neg prompt中添加\"安静，固定\"等词语可以增加动态性。)"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 89,
"type": "Note",
"pos": [
-192,
-293
],
"size": [
427.074951171875,
143.9142608642578
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 92,
"type": "FunTextBox",
"pos": [
254,
-46
],
"size": [
380.845703125,
157.68350219726562
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
73
]
}
],
"title": "Positive Prompt（正向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"在这个阳光明媚的户外花园里，美女身穿一袭及膝的白色无袖连衣裙，裙摆在她轻盈的舞姿中轻柔地摆动，宛如一只翩翩起舞的蝴蝶。阳光透过树叶间洒下斑驳的光影，映衬出她柔和的脸庞和清澈的眼眸，显得格外优雅。仿佛每一个动作都在诉说着青春与活力，她在草地上旋转，裙摆随之飞扬，仿佛整个花园都因她的舞动而欢愉。周围五彩缤纷的花朵在微风中摇曳，玫瑰、菊花、百合，各自释放出阵阵香气，营造出一种轻松而愉快的氛围。"
]
},
{
"id": 99,
"type": "VHS_VideoCombine",
"pos": [
1094,
559
],
"size": [
315,
851.4834594726562
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 81
},
{
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun-Preprocess-Video",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 19,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "AnimateDiff_00001.gif",
"subfolder": "",
"type": "output",
"format": "image/gif",
"frame_rate": 8
}
}
}
},
{
"id": 94,
"type": "FunTextBox",
"pos": [
258,
178
],
"size": [
368.5529479980469,
159.4075927734375
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
74
]
}
],
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
]
},
{
"id": 101,
"type": "LoadWan2_2FunModel",
"pos": [
306.2298583984375,
-313.5478515625
],
"size": [
397.9565734863281,
154
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"links": [
78
]
}
],
"properties": {
"Node name for S&R": "LoadWan2_2FunModel"
},
"widgets_values": [
"Wan2.2-Fun-A14B-Control",
"Control",
"sequential_cpu_offload",
"wan2.2/wan_civitai_i2v.yaml",
"bf16"
]
},
{
"id": 17,
"type": "VHS_VideoCombine",
"pos": [
1465.6671142578125,
-174.95152282714844
],
"size": [
390.9534912109375,
966.9860229492188
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"label": "图像",
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 77
},
{
"label": "音频",
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"label": "批次管理",
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"slot_index": 0,
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 22,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "Fun_00004.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 16
}
}
}
},
{
"id": 85,
"type": "VHS_LoadVideo",
"pos": [
335,
476
],
"size": [
252.056640625,
409.87884521484375
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
82
]
},
{
"name": "frame_count",
"type": "INT",
"links": null
},
{
"name": "audio",
"type": "AUDIO",
"links": null
},
{
"name": "video_info",
"type": "VHS_VIDEOINFO",
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_LoadVideo"
},
"widgets_values": {
"video": "00000005.mp4",
"force_rate": 0,
"force_size": "Disabled",
"custom_width": 512,
"custom_height": 512,
"frame_load_cap": 0,
"skip_first_frames": 0,
"select_every_nth": 1,
"choose video to upload": "image",
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"frame_load_cap": 0,
"skip_first_frames": 0,
"force_rate": 0,
"filename": "00000005.mp4",
"type": "input",
"format": "video/mp4",
"select_every_nth": 1
}
}
}
},
{
"id": 102,
"type": "Wan2_2FunV2VSampler",
"pos": [
848.4100341796875,
-180.23484802246094
],
"size": [
458.36383056640625,
526
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "funmodels",
"type": "FunModels",
"link": 78
},
{
"name": "prompt",
"type": "STRING_PROMPT",
"link": 73
},
{
"name": "negative_prompt",
"type": "STRING_PROMPT",
"link": 74
},
{
"name": "validation_video",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "control_video",
"shape": 7,
"type": "IMAGE",
"link": 83
},
{
"name": "start_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "end_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "ref_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "camera_conditions",
"shape": 7,
"type": "STRING",
"link": null
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"links": [
77
]
}
],
"properties": {
"Node name for S&R": "Wan2_2FunV2VSampler"
},
"widgets_values": [
81,
640,
43,
"fixed",
50,
6.000000000000001,
1,
"Flow",
0.1,
true,
5,
true,
0
]
},
{
"id": 103,
"type": "VideoToDepth",
"pos": [
734.0958862304688,
474.7727355957031
],
"size": [
270,
58
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "input_video",
"type": "IMAGE",
"link": 82
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"links": [
81,
83
]
}
],
"properties": {
"Node name for S&R": "VideoToDepth"
},
"widgets_values": [
81
]
}
],
"links": [
[
73,
92,
0,
102,
1,
"STRING_PROMPT"
],
[
74,
94,
0,
102,
2,
"STRING_PROMPT"
],
[
77,
102,
0,
17,
0,
"IMAGE"
],
[
78,
101,
0,
102,
0,
"FunModels"
],
[
81,
103,
0,
99,
0,
"IMAGE"
],
[
82,
85,
0,
103,
0,
"IMAGE"
],
[
83,
103,
0,
102,
4,
"IMAGE"
]
],
"groups": [
{
"id": 1,
"title": "Upload Your Video",
"bounding": [
218,
385,
487,
789
],
"color": "#a1309b",
"font_size": 24,
"flags": {}
},
{
"id": 2,
"title": "Load Model",
"bounding": [
218,
-387,
542,
248
],
"color": "#b06634",
"font_size": 24,
"flags": {}
},
{
"id": 3,
"title": "Prompts",
"bounding": [
218,
-127,
450,
483
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.6830134553650709,
"offset": [
192.62803696741028,
413.1926211906501
]
},
"workspace_info": {
"id": "776b62b4-bd17-4ed3-9923-b7aad000b1ea"
},
"node_versions": {
"CogVideoX-Fun": "e935795d7684aaba72be65d2ad5ef580c0526f7a",
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e"
},
"frontendVersion": "1.21.3"
},
"version": 0.4
}
comfyui/wan2_2_fun/v1/wan2.2_fun_workflow_v2v_control_pose_ref.json
+753
View File
@@ -0,0 +1,753 @@
{
"id": "c5a53a91-88dc-41d4-b414-12e402c59e4c",
"revision": 0,
"last_node_id": 105,
"last_link_id": 91,
"nodes": [
{
"id": 78,
"type": "Note",
"pos": [
18,
-46
],
"size": [
210,
88
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can write prompt here\n（你可以在此填写提示词）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 79,
"type": "Note",
"pos": [
-111.46612548828125,
460.2178955078125
],
"size": [
210,
88
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"You can upload video here\n（在此上传视频）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 88,
"type": "Note",
"pos": [
-99,
197
],
"size": [
326.1556091308594,
145.20904541015625
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"Using longer neg prompt such as \"Blurring, mutation, deformation, distortion, dark and solid, comics.\" can increase stability. Adding words such as \"quiet, solid\" to the neg prompt can increase dynamism.\n(使用更长的neg prompt如\"模糊，突变，变形，失真，画面暗，画面固定，连环画，漫画，线稿，没有主体。\"，可以增加稳定性。在neg prompt中添加\"安静，固定\"等词语可以增加动态性。)"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 89,
"type": "Note",
"pos": [
-192,
-293
],
"size": [
427.074951171875,
143.9142608642578
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {
"text": ""
},
"widgets_values": [
"When using the 14B model, you can use sequential_cpu_offload to save GPU memory during generation.\n（在使用14B模型时，可以使用sequential_cpu_offload节省显存，进行生成。）"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 99,
"type": "VHS_VideoCombine",
"pos": [
1005,
554
],
"size": [
315,
849.46875
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 73
},
{
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"name": "Filenames",
"type": "VHS_FILENAMES",
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun-Preprocess-Video",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 19,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "Fun-Preprocess-Video_00007.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 16
}
}
}
},
{
"id": 85,
"type": "VHS_LoadVideo",
"pos": [
207.79391479492188,
473.83123779296875
],
"size": [
252.056640625,
688.545166015625
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
72
]
},
{
"name": "frame_count",
"type": "INT",
"links": null
},
{
"name": "audio",
"type": "AUDIO",
"links": null
},
{
"name": "video_info",
"type": "VHS_VIDEOINFO",
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_LoadVideo"
},
"widgets_values": {
"video": "000007.mp4",
"force_rate": 16,
"force_size": "Disabled",
"custom_width": 512,
"custom_height": 512,
"frame_load_cap": 0,
"skip_first_frames": 0,
"select_every_nth": 1,
"choose video to upload": "image",
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"frame_load_cap": 0,
"skip_first_frames": 0,
"force_rate": 16,
"filename": "000007.mp4",
"type": "input",
"format": "video/mp4",
"select_every_nth": 1
}
}
}
},
{
"id": 17,
"type": "VHS_VideoCombine",
"pos": [
1488,
8
],
"size": [
390.9534912109375,
942.2557983398438
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"label": "图像",
"name": "images",
"shape": 7,
"type": "IMAGE",
"link": 84
},
{
"label": "音频",
"name": "audio",
"shape": 7,
"type": "AUDIO",
"link": null
},
{
"label": "批次管理",
"name": "meta_batch",
"shape": 7,
"type": "VHS_BatchManager",
"link": null
},
{
"name": "vae",
"shape": 7,
"type": "VAE",
"link": null
}
],
"outputs": [
{
"label": "文件名",
"name": "Filenames",
"type": "VHS_FILENAMES",
"slot_index": 0,
"links": null
}
],
"properties": {
"Node name for S&R": "VHS_VideoCombine"
},
"widgets_values": {
"frame_rate": 16,
"loop_count": 0,
"filename_prefix": "Fun",
"format": "video/h264-mp4",
"pix_fmt": "yuv420p",
"crf": 22,
"save_metadata": true,
"pingpong": false,
"save_output": true,
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"filename": "Fun_00049.mp4",
"subfolder": "",
"type": "output",
"format": "video/h264-mp4",
"frame_rate": 16
}
}
}
},
{
"id": 102,
"type": "LoadImage",
"pos": [
553,
598
],
"size": [
315,
314
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
86
]
},
{
"name": "MASK",
"type": "MASK",
"links": null
}
],
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"9.png",
"image"
]
},
{
"id": 101,
"type": "VideoToOpenpose",
"pos": [
558,
474
],
"size": [
315,
58
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "input_video",
"type": "IMAGE",
"link": 72
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"slot_index": 0,
"links": [
73,
87,
91
]
}
],
"properties": {
"Node name for S&R": "VideoToOpenpose"
},
"widgets_values": [
81
]
},
{
"id": 104,
"type": "LoadWan2_2FunModel",
"pos": [
303.9479064941406,
-309.996337890625
],
"size": [
390.1385192871094,
154
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "funmodels",
"type": "FunModels",
"links": [
88
]
}
],
"properties": {
"Node name for S&R": "LoadWan2_2FunModel"
},
"widgets_values": [
"Wan2.2-Fun-A14B-Control",
"Control",
"sequential_cpu_offload",
"wan2.2/wan_civitai_i2v.yaml",
"bf16"
]
},
{
"id": 92,
"type": "FunTextBox",
"pos": [
254,
-46
],
"size": [
380.845703125,
157.68350219726562
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
89
]
}
],
"title": "Positive Prompt（正向提示词）",
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"一位动漫风格的女孩。她有着紫色的短发，头上戴着一个黑色和金色相间的蝴蝶结。她的表情显得有些严肃或沉思，眼睛大而有神。女孩穿着一件白色衬衫，外面搭配了一件深蓝色的背心，背心上有一个粉色的蝴蝶结装饰。她的裙子是白色的，裙摆蓬松，整体造型非常可爱且精致。背景是一个简单的圆形图案，颜色为粉红色和灰色相间，给人一种柔和的感觉。整个画面色调柔和，人物形象生动鲜明。"
]
},
{
"id": 94,
"type": "FunTextBox",
"pos": [
258,
178
],
"size": [
368.5529479980469,
159.4075927734375
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "prompt",
"type": "STRING_PROMPT",
"slot_index": 0,
"links": [
90
]
}
],
"properties": {
"Node name for S&R": "FunTextBox"
},
"widgets_values": [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
]
},
{
"id": 105,
"type": "Wan2_2FunV2VSampler",
"pos": [
987.7456665039062,
-76.86135864257812
],
"size": [
434.938232421875,
526
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "funmodels",
"type": "FunModels",
"link": 88
},
{
"name": "prompt",
"type": "STRING_PROMPT",
"link": 89
},
{
"name": "negative_prompt",
"type": "STRING_PROMPT",
"link": 90
},
{
"name": "validation_video",
"shape": 7,
"type": "IMAGE",
"link": 91
},
{
"name": "control_video",
"shape": 7,
"type": "IMAGE",
"link": 87
},
{
"name": "start_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "end_image",
"shape": 7,
"type": "IMAGE",
"link": null
},
{
"name": "ref_image",
"shape": 7,
"type": "IMAGE",
"link": 86
},
{
"name": "camera_conditions",
"shape": 7,
"type": "STRING",
"link": null
},
{
"name": "riflex_k",
"shape": 7,
"type": "RIFLEXT_ARGS",
"link": null
}
],
"outputs": [
{
"name": "images",
"type": "IMAGE",
"links": [
84
]
}
],
"properties": {
"Node name for S&R": "Wan2_2FunV2VSampler"
},
"widgets_values": [
81,
640,
43,
"fixed",
50,
6.000000000000001,
1,
"Flow",
0.1,
true,
5,
true,
0
]
}
],
"links": [
[
72,
85,
0,
101,
0,
"IMAGE"
],
[
73,
101,
0,
99,
0,
"IMAGE"
],
[
84,
105,
0,
17,
0,
"IMAGE"
],
[
86,
102,
0,
105,
7,
"IMAGE"
],
[
87,
101,
0,
105,
4,
"IMAGE"
],
[
88,
104,
0,
105,
0,
"FunModels"
],
[
89,
92,
0,
105,
1,
"STRING_PROMPT"
],
[
90,
94,
0,
105,
2,
"STRING_PROMPT"
],
[
91,
101,
0,
105,
3,
"IMAGE"
]
],
"groups": [
{
"id": 1,
"title": "Upload Your Video And Reference Image",
"bounding": [
91,
383,
859,
841
],
"color": "#a1309b",
"font_size": 24,
"flags": {}
},
{
"id": 2,
"title": "Load Model",
"bounding": [
218,
-387,
542,
248
],
"color": "#b06634",
"font_size": 24,
"flags": {}
},
{
"id": 3,
"title": "Prompts",
"bounding": [
218,
-127,
450,
483
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.7513148009015781,
"offset": [
235.41928184428536,
445.7655228500244
]
},
"workspace_info": {
"id": "776b62b4-bd17-4ed3-9923-b7aad000b1ea"
},
"node_versions": {
"ComfyUI-VideoHelperSuite": "70faa9bcef65932ab72e7404d6373fb300013a2e",
"comfy-core": "0.3.44",
"CogVideoX-Fun": "e935795d7684aaba72be65d2ad5ef580c0526f7a"
},
"frontendVersion": "1.21.3"
},
"version": 0.4
}
comfyui/wan2_2_fun/v1/wan2.2_fun_workflow_v2v_control_ref.json
+1 -1
View File
@@ -574,7 +574,7 @@
"flags": {}
},
{
"title": "Load EasyAnimate",
"title": "Load Model",
"bounding": [
218,
-387,
examples/wan2.2/post_infer_queue.py
+2 -3
View File
@@ -106,9 +106,8 @@ if __name__ == '__main__':
# Recommended to be set between 0.05 and 0.30. A larger threshold can cache more steps, speeding up the inference process,
# but it may cause slight differences between the generated content and the original content.
# # --------------------------------------------------------------------------------------------------- #
# | Model Name | threshold | Model Name | threshold | Model Name | threshold |
# | Wan2.1-T2V-1.3B | 0.05~0.10 | Wan2.1-T2V-14B | 0.10~0.15 | Wan2.1-I2V-14B-720P | 0.20~0.30 |
# | Wan2.1-I2V-14B-480P | 0.20~0.25 | Wan2.1-Fun-*-1.3B-* | 0.05~0.10 | Wan2.1-Fun-*-14B-* | 0.20~0.30 |
# | Model Name | threshold | Model Name | threshold |
# | Wan2.2-T2V-A14B | 0.10~0.15 | Wan2.2-I2V-A14B | 0.15~0.20 |
# # --------------------------------------------------------------------------------------------------- #
teacache_threshold = 0.10
# The number of steps to skip TeaCache at the beginning of the inference process, which can
examples/wan2.2/post_infer_queue_i2v.py
+2 -3
View File
@@ -123,9 +123,8 @@ if __name__ == '__main__':
# Recommended to be set between 0.05 and 0.30. A larger threshold can cache more steps, speeding up the inference process,
# but it may cause slight differences between the generated content and the original content.
# # --------------------------------------------------------------------------------------------------- #
# | Model Name | threshold | Model Name | threshold | Model Name | threshold |
# | Wan2.1-T2V-1.3B | 0.05~0.10 | Wan2.1-T2V-14B | 0.10~0.15 | Wan2.1-I2V-14B-720P | 0.20~0.30 |
# | Wan2.1-I2V-14B-480P | 0.20~0.25 | Wan2.1-Fun-*-1.3B-* | 0.05~0.10 | Wan2.1-Fun-*-14B-* | 0.20~0.30 |
# | Model Name | threshold | Model Name | threshold |
# | Wan2.2-T2V-A14B | 0.10~0.15 | Wan2.2-I2V-A14B | 0.15~0.20 |
# # --------------------------------------------------------------------------------------------------- #
teacache_threshold = 0.10
# The number of steps to skip TeaCache at the beginning of the inference process, which can
examples/wan2.2/predict_ti2v.py
+1 -1
View File
@@ -216,7 +216,7 @@ scheduler = Choosen_Scheduler(
# Get Pipeline
pipeline = Wan2_2TI2VPipeline(
transformer=transformer,
transformer_2=transformer_2 ,
transformer_2=transformer_2,
vae=vae,
tokenizer=tokenizer,
text_encoder=text_encoder,
examples/wan2.2_fun/app.py
Executable
+79
View File
@@ -0,0 +1,79 @@
import os
import sys
import time
import torch
current_file_path = os.path.abspath(__file__)
project_roots = [os.path.dirname(current_file_path), os.path.dirname(os.path.dirname(current_file_path)), os.path.dirname(os.path.dirname(os.path.dirname(current_file_path)))]
for project_root in project_roots:
sys.path.insert(0, project_root) if project_root not in sys.path else None
from videox_fun.api.api import (infer_forward_api,
update_diffusion_transformer_api)
from videox_fun.ui.controller import flow_scheduler_dict
from videox_fun.ui.wan2_2_fun_ui import ui, ui_client, ui_host
if __name__ == "__main__":
# Choose the ui mode
# "normal" refers to the standard UI, which allows users to click to switch models, change model types, and more.
# "host" represents the hosting mode, where the model is loaded directly at startup and can be accessed via
# the API to return generation results.
# "client" represents the client mode, offering a simple UI that sends requests to a remote API for generation.
ui_mode = "normal"
# GPU memory mode, which can be choosen in [model_full_load, model_cpu_offload, model_cpu_offload_and_qfloat8, sequential_cpu_offload].
# model_full_load means that the entire model will be moved to the GPU.
#
# model_cpu_offload means that the entire model will be moved to the CPU after use, which can save some GPU memory.
#
# model_cpu_offload_and_qfloat8 indicates that the entire model will be moved to the CPU after use,
# and the transformer model has been quantized to float8, which can save more GPU memory.
#
# sequential_cpu_offload means that each layer of the model will be moved to the CPU after use,
# resulting in slower speeds but saving a large amount of GPU memory.
GPU_memory_mode = "sequential_cpu_offload"
# Compile will give a speedup in fixed resolution and need a little GPU memory.
# The compile_dit is not compatible with the fsdp_dit and sequential_cpu_offload.
compile_dit = False
# Use torch.float16 if GPU does not support torch.bfloat16
# ome graphics cards, such as v100, 2080ti, do not support torch.bfloat16
weight_dtype = torch.bfloat16
# Server ip
server_name = "0.0.0.0"
server_port = 7860
# Config path
config_path = "config/wan2.2/wan_civitai_i2v.yaml"
# Params below is used when ui_mode = "host"
# Model path of the pretrained model
model_name = "models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
# "Inpaint" or "Control"
model_type = "Inpaint"
if ui_mode == "host":
demo, controller = ui_host(GPU_memory_mode, flow_scheduler_dict, model_name, model_type, config_path, compile_dit, weight_dtype)
elif ui_mode == "client":
demo, controller = ui_client(flow_scheduler_dict, model_name)
else:
demo, controller = ui(GPU_memory_mode, flow_scheduler_dict, config_path, compile_dit, weight_dtype)
def gr_launch():
# launch gradio
app, _, _ = demo.queue(status_update_rate=1).launch(
server_name=server_name,
server_port=server_port,
prevent_thread_lock=True
)
# launch api
infer_forward_api(None, app, controller)
update_diffusion_transformer_api(None, app, controller)
gr_launch()
# not close the python
while True:
time.sleep(5)
examples/wan2.2_fun/launch_api.py
Executable
+91
View File
@@ -0,0 +1,91 @@
import argparse
import os
import sys
import time
import gradio as gr
import ray
import torch
current_file_path = os.path.abspath(__file__)
project_roots = [os.path.dirname(current_file_path), os.path.dirname(os.path.dirname(current_file_path)), os.path.dirname(os.path.dirname(os.path.dirname(current_file_path)))]
for project_root in project_roots:
sys.path.insert(0, project_root) if project_root not in sys.path else None
from videox_fun.api.api_multi_nodes import (MultiNodesEngine,
multi_nodes_infer_forward_api)
from videox_fun.ui.controller import flow_scheduler_dict
from videox_fun.ui.wan2_2_fun_ui import Wan2_2_Fun_Controller
def main():
parser = argparse.ArgumentParser(description='xDiT HTTP Service')
parser.add_argument('--world_size', type=int, default=8, help='Number of parallel workers')
parser.add_argument(
'--gpu_memory_mode', type=str, default="model_full_load", help='''
GPU memory mode, which can be choosen in [model_full_load, model_full_load_and_qfloat8, model_cpu_offload, model_cpu_offload_and_qfloat8].
model_full_load means that the entire model will be moved to the GPU.
model_full_load_and_qfloat8 means that the entire model will be moved to the GPU,
and the transformer model has been quantized to float8, which can save more GPU memory.
model_cpu_offload means that the entire model will be moved to the CPU after use, which can save some GPU memory.
model_cpu_offload_and_qfloat8 indicates that the entire model will be moved to the CPU after use,
and the transformer model has been quantized to float8, which can save more GPU memory.
'''
)
parser.add_argument('--ulysses_degree', type=int, default=4, help='Degree of Ulysses configuration')
parser.add_argument('--ring_degree', type=int, default=2, help='Degree of Ring configuration')
parser.add_argument(
'--compile_dit', action='store_true', help='''
Enable compile dit.
Compile will give a speedup in fixed resolution and need a little GPU memory.
The compile_dit is not compatible with the fsdp_dit and sequential_cpu_offload.
'''
)
parser.add_argument('--fsdp_dit', action='store_true', help="Use DIT FSDP to save more GPU memory in multi gpus.")
parser.add_argument('--fsdp_text_encoder', action='store_true', help="Use Text Encoder FSDP to save more GPU memory in multi gpus.")
parser.add_argument('--weight_dtype', type=str, default='bf16', help='Weight data type')
parser.add_argument('--server_name', type=str, default="0.0.0.0", help='Server IP address')
parser.add_argument('--server_port', type=int, default=7860, help='Server Port')
parser.add_argument('--config_path', type=str, default="config/wan2.1/wan_civitai.yaml", help='Path to config file')
parser.add_argument('--model_name', type=str, default="models/Diffusion_Transformer/Wan2.1-Fun-V1.1-1.3B-InP", help='Model path')
parser.add_argument('--model_type', type=str, default="Inpaint", help='Model type (Inpaint/Control)')
parser.add_argument('--savedir_sample', type=str, default=None, help='The save directory for samples')
args = parser.parse_args()
weight_dtype = torch.float32
if args.weight_dtype == "bf16":
weight_dtype = torch.bfloat16
elif args.weight_dtype == "fp16":
weight_dtype = torch.float16
engine = MultiNodesEngine(
world_size=args.world_size, Controller=Wan2_2_Fun_Controller,
GPU_memory_mode=args.gpu_memory_mode, scheduler_dict=flow_scheduler_dict, model_name=args.model_name, model_type=args.model_type, config_path=args.config_path,
ulysses_degree=args.ulysses_degree, ring_degree=args.ring_degree,
fsdp_dit=args.fsdp_dit, fsdp_text_encoder=args.fsdp_text_encoder, compile_dit=args.compile_dit,
weight_dtype=weight_dtype, savedir_sample=args.savedir_sample,
)
def gr_launch():
# launch gradio
with gr.Blocks() as demo:
gr.Markdown("")
app, _, _ = demo.queue(status_update_rate=1).launch(
server_name=args.server_name,
server_port=args.server_port,
prevent_thread_lock=True
)
# launch api
multi_nodes_infer_forward_api(None, app, engine)
gr_launch()
# not close the python
while True:
time.sleep(5)
if __name__ == "__main__":
main()
examples/wan2.2_fun/post_infer.py
Executable
+150
View File
@@ -0,0 +1,150 @@
import base64
import json
import time
from datetime import datetime
import requests
import base64
def post_diffusion_transformer(diffusion_transformer_path, url='http://127.0.0.1:7860'):
datas = json.dumps({
"diffusion_transformer_path": diffusion_transformer_path
})
r = requests.post(f'{url}/videox_fun/update_diffusion_transformer', data=datas, timeout=1500)
data = r.content.decode('utf-8')
return data
def post_update_edition(edition, url='http://0.0.0.0:7860'):
datas = json.dumps({
"edition": edition
})
r = requests.post(f'{url}/videox_fun/update_edition', data=datas, timeout=1500)
data = r.content.decode('utf-8')
return data
def post_infer(
generation_method,
length_slider,
url='http://127.0.0.1:7860',
POST_TOKEN="",
timeout=5000,
base_model_path="none",
lora_model_path="none",
lora_alpha_slider=0.55,
prompt_textbox="A young woman with beautiful and clear eyes and blonde hair standing and white dress in a forest wearing a crown. She seems to be lost in thought, and the camera focuses on her face. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic.",
negative_prompt_textbox="The video is not of a high quality, it has a low resolution. Watermark present in each frame. The background is solid. Strange body and strange trajectory. Distortion.",
sampler_dropdown="Flow",
sample_step_slider=50,
width_slider=672,
height_slider=384,
cfg_scale_slider=6,
seed_textbox=43
):
# Prepare the data payload
datas = json.dumps({
"base_model_path": base_model_path,
"lora_model_path": lora_model_path,
"lora_alpha_slider": lora_alpha_slider,
"prompt_textbox": prompt_textbox,
"negative_prompt_textbox": negative_prompt_textbox,
"sampler_dropdown": sampler_dropdown,
"sample_step_slider": sample_step_slider,
"width_slider": width_slider,
"height_slider": height_slider,
"generation_method": generation_method,
"length_slider": length_slider,
"cfg_scale_slider": cfg_scale_slider,
"seed_textbox": seed_textbox,
})
# Initialize session and set headers
session = requests.session()
session.headers.update({"Authorization": POST_TOKEN})
# Send POST request
if url[-1] == "/":
url = url[:-1]
post_r = session.post(f'{url}/videox_fun/infer_forward', data=datas, timeout=timeout)
data = post_r.content.decode('utf-8')
return data
if __name__ == '__main__':
# initiate time
time_start = time.time()
# The Url you want to post
POST_URL = 'http://0.0.0.0:7860'
# Used in EAS. If you don't need Authorization, please set it to empty string.
TOKEN = ''
# -------------------------- #
# Step 1: update edition
# -------------------------- #
# diffusion_transformer_path = "models/Diffusion_Transformer/Wan2.1-Fun-1.3B-InP"
# outputs = post_diffusion_transformer(diffusion_transformer_path)
# print('Output update edition: ', outputs)
# -------------------------- #
# Step 2: infer
# -------------------------- #
# "Video Generation" and "Image Generation"
generation_method = "Video Generation"
# Video length
length_slider = 49
# Used in Lora models
lora_model_path = "none"
lora_alpha_slider = 0.55
# Prompts
prompt_textbox = "A young woman with beautiful and clear eyes and blonde hair standing and white dress in a forest wearing a crown. She seems to be lost in thought, and the camera focuses on her face. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic."
negative_prompt_textbox = "The video is not of a high quality, it has a low resolution. Watermark present in each frame. The background is solid. Strange body and strange trajectory. Distortion."
# Sampler name
sampler_dropdown = "Flow"
# Sampler steps
sample_step_slider = 50
# height and width
width_slider = 832
height_slider = 480
# cfg scale
cfg_scale_slider = 6
seed_textbox = 43
outputs = post_infer(
generation_method,
length_slider,
lora_model_path=lora_model_path,
lora_alpha_slider=lora_alpha_slider,
prompt_textbox=prompt_textbox,
negative_prompt_textbox=negative_prompt_textbox,
sampler_dropdown=sampler_dropdown,
sample_step_slider=sample_step_slider,
width_slider=width_slider,
height_slider=height_slider,
cfg_scale_slider=cfg_scale_slider,
seed_textbox=seed_textbox,
url=POST_URL,
POST_TOKEN=TOKEN
)
# Get decoded data
outputs = json.loads(outputs)
base64_encoding = outputs["base64_encoding"]
decoded_data = base64.b64decode(base64_encoding)
is_image = True if generation_method == "Image Generation" else False
if is_image or length_slider == 1:
file_path = "1.png"
else:
file_path = "1.mp4"
with open(file_path, "wb") as file:
file.write(decoded_data)
# End of record time
# The calculated time difference is the execution time of the program, expressed in seconds / s
time_end = time.time()
time_sum = (time_end - time_start)
print('# --------------------------------------------------------- #')
print(f'# Total expenditure: {time_sum}s')
print('# --------------------------------------------------------- #')
examples/wan2.2_fun/post_infer_queue.py
Executable
+192
View File
@@ -0,0 +1,192 @@
import base64
import json
import time
import urllib.parse
import requests
def post_infer(
generation_method,
length_slider,
url='http://127.0.0.1:7860',
POST_TOKEN="",
timeout=5,
base_model_path="none",
lora_model_path="none",
lora_alpha_slider=0.55,
prompt_textbox="A young woman with beautiful and clear eyes and blonde hair standing and white dress in a forest wearing a crown. She seems to be lost in thought, and the camera focuses on her face. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic.",
negative_prompt_textbox="色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走",
sampler_dropdown="Flow",
sample_step_slider=50,
width_slider=672,
height_slider=384,
cfg_scale_slider=6,
seed_textbox=43,
enable_teacache = None,
teacache_threshold = None,
num_skip_start_steps = None,
teacache_offload = None,
cfg_skip_ratio = None,
enable_riflex = None,
riflex_k = None,
):
# Prepare the data payload
datas = json.dumps({
"base_model_path": base_model_path,
"lora_model_path": lora_model_path,
"lora_alpha_slider": lora_alpha_slider,
"prompt_textbox": prompt_textbox,
"negative_prompt_textbox": negative_prompt_textbox,
"sampler_dropdown": sampler_dropdown,
"sample_step_slider": sample_step_slider,
"width_slider": width_slider,
"height_slider": height_slider,
"generation_method": generation_method,
"length_slider": length_slider,
"cfg_scale_slider": cfg_scale_slider,
"seed_textbox": seed_textbox,
"enable_teacache": enable_teacache,
"teacache_threshold": teacache_threshold,
"num_skip_start_steps": num_skip_start_steps,
"teacache_offload": teacache_offload,
"cfg_skip_ratio": cfg_skip_ratio,
"enable_riflex": enable_riflex,
"riflex_k": riflex_k,
})
# Initialize session and set headers
session = requests.session()
session.headers.update({"Authorization": POST_TOKEN})
# Send POST request
if url[-1] == "/":
url = url[:-1]
post_r = session.post(f'{url}/videox_fun/infer_forward', data=datas, timeout=timeout)
# Extract request ID from POST response headers
request_id = post_r.headers.get("X-Eas-Queueservice-Request-Id")
# Prepare query parameters for GET request
query = {
'_index_': '0',
'_length_': '1',
'_timeout_': str(timeout),
'_raw_': 'false',
'_auto_delete_': 'true',
}
if request_id:
query['requestId'] = request_id
query_str = urllib.parse.urlencode(query)
# Polling GET request until status code is not 204
status_code = 204
while status_code == 204:
if query_str:
get_r = session.get(f'{url}/sink?{query_str}', timeout=timeout)
else:
get_r = session.get(f'{url}/sink', timeout=timeout)
status_code = get_r.status_code
# Decode and return the response content
data = get_r.content.decode('utf-8')
return data
if __name__ == '__main__':
# initiate time
time_start = time.time()
# EAS队列配置
EAS_URL = 'http://17xxxxxxxxx.pai-eas.aliyuncs.com/api/predict/xxxxxxxx'
# Use in EAS Queue
TOKEN = 'xxxxxxxx'
# Support TeaCache.
enable_teacache = True
# Recommended to be set between 0.05 and 0.30. A larger threshold can cache more steps, speeding up the inference process,
# but it may cause slight differences between the generated content and the original content.
# # --------------------------------------------------------------------------------------------------- #
# | Model Name | threshold | Model Name | threshold |
# | Wan2.2-T2V-A14B | 0.10~0.15 | Wan2.2-I2V-A14B | 0.15~0.20 |
# | Wan2.2-Fun-A14B-* | 0.15~0.20 |
# # --------------------------------------------------------------------------------------------------- #
teacache_threshold = 0.10
# The number of steps to skip TeaCache at the beginning of the inference process, which can
# reduce the impact of TeaCache on generated video quality.
num_skip_start_steps = 5
# Whether to offload TeaCache tensors to cpu to save a little bit of GPU memory.
teacache_offload = False
# Skip some cfg steps in inference
# Recommended to be set between 0.00 and 0.25
cfg_skip_ratio = 0
# Riflex config
enable_riflex = False
# Index of intrinsic frequency
riflex_k = 6
# "Video Generation" and "Image Generation"
generation_method = "Video Generation"
# Video length
length_slider = 81
# Used in Lora models
lora_model_path = "none"
lora_alpha_slider = 0.55
# Prompts
prompt_textbox = "A young woman with beautiful and clear eyes and blonde hair standing and white dress in a forest wearing a crown. She seems to be lost in thought, and the camera focuses on her face. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic."
negative_prompt_textbox = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
# Sampler name
sampler_dropdown = "Flow"
# Sampler steps
sample_step_slider = 50
# height and width
width_slider = 832
height_slider = 480
# cfg scale
cfg_scale_slider = 6
seed_textbox = 43
outputs = post_infer(
generation_method,
length_slider,
lora_model_path=lora_model_path,
lora_alpha_slider=lora_alpha_slider,
prompt_textbox=prompt_textbox,
negative_prompt_textbox=negative_prompt_textbox,
sampler_dropdown=sampler_dropdown,
sample_step_slider=sample_step_slider,
width_slider=width_slider,
height_slider=height_slider,
cfg_scale_slider=cfg_scale_slider,
seed_textbox=seed_textbox,
enable_teacache = enable_teacache,
teacache_threshold = teacache_threshold,
num_skip_start_steps = num_skip_start_steps,
teacache_offload = teacache_offload,
cfg_skip_ratio = cfg_skip_ratio,
enable_riflex = enable_riflex,
riflex_k = riflex_k,
url=EAS_URL,
POST_TOKEN=TOKEN
)
# Get decoded data
outputs = json.loads(base64.b64decode(json.loads(outputs)[0]['data']))
base64_encoding = outputs["base64_encoding"]
decoded_data = base64.b64decode(base64_encoding)
is_image = True if generation_method == "Image Generation" else False
if is_image or length_slider == 1:
file_path = "1.png"
else:
file_path = "1.mp4"
with open(file_path, "wb") as file:
file.write(decoded_data)
# End of record time
# The calculated time difference is the execution time of the program, expressed in seconds / s
time_end = time.time()
time_sum = (time_end - time_start)
print('# --------------------------------------------------------- #')
print(f'# Total expenditure: {time_sum}s')
print('# --------------------------------------------------------- #')
examples/wan2.2_fun/post_infer_queue_i2v.py
Executable
+213
View File
@@ -0,0 +1,213 @@
import base64
import json
import time
import urllib.parse
import requests
from PIL import Image
from io import BytesIO
def post_infer(
generation_method,
length_slider,
url='http://127.0.0.1:7860',
POST_TOKEN="",
timeout=5,
base_model_path="none",
lora_model_path="none",
lora_alpha_slider=0.55,
prompt_textbox="A young woman with beautiful and clear eyes and blonde hair standing and white dress in a forest wearing a crown. She seems to be lost in thought, and the camera focuses on her face. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic.",
negative_prompt_textbox="色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走",
sampler_dropdown="Flow",
sample_step_slider=50,
width_slider=672,
height_slider=384,
cfg_scale_slider=6,
seed_textbox=43,
enable_teacache = None,
teacache_threshold = None,
num_skip_start_steps = None,
teacache_offload = None,
cfg_skip_ratio = None,
enable_riflex = None,
riflex_k = None,
start_image = None
):
if start_image:
try:
if not start_image.startswith("http"):
image = Image.open(start_image).convert("RGB")
# 将图片转换为 Base64 编码
buffered = BytesIO()
image.save(buffered, format="JPEG")
start_image = base64.b64encode(buffered.getvalue()).decode('utf-8')
except Exception as e:
print(f"Error processing start_image: {e}")
raise
# Prepare the data payload
datas = json.dumps({
"base_model_path": base_model_path,
"lora_model_path": lora_model_path,
"lora_alpha_slider": lora_alpha_slider,
"prompt_textbox": prompt_textbox,
"negative_prompt_textbox": negative_prompt_textbox,
"sampler_dropdown": sampler_dropdown,
"sample_step_slider": sample_step_slider,
"width_slider": width_slider,
"height_slider": height_slider,
"generation_method": generation_method,
"length_slider": length_slider,
"cfg_scale_slider": cfg_scale_slider,
"seed_textbox": seed_textbox,
"enable_teacache": enable_teacache,
"teacache_threshold": teacache_threshold,
"num_skip_start_steps": num_skip_start_steps,
"teacache_offload": teacache_offload,
"cfg_skip_ratio": cfg_skip_ratio,
"enable_riflex": enable_riflex,
"riflex_k": riflex_k,
"start_image": start_image
})
# Initialize session and set headers
session = requests.session()
session.headers.update({"Authorization": POST_TOKEN})
# Send POST request
if url[-1] == "/":
url = url[:-1]
post_r = session.post(f'{url}/videox_fun/infer_forward', data=datas, timeout=timeout)
# Extract request ID from POST response headers
request_id = post_r.headers.get("X-Eas-Queueservice-Request-Id")
# Prepare query parameters for GET request
query = {
'_index_': '0',
'_length_': '1',
'_timeout_': str(timeout),
'_raw_': 'false',
'_auto_delete_': 'true',
}
if request_id:
query['requestId'] = request_id
query_str = urllib.parse.urlencode(query)
# Polling GET request until status code is not 204
status_code = 204
while status_code == 204:
if query_str:
get_r = session.get(f'{url}/sink?{query_str}', timeout=timeout)
else:
get_r = session.get(f'{url}/sink', timeout=timeout)
status_code = get_r.status_code
# Decode and return the response content
data = get_r.content.decode('utf-8')
return data
if __name__ == '__main__':
# initiate time
time_start = time.time()
# EAS队列配置
EAS_URL = 'http://17xxxxxxxxx.pai-eas.aliyuncs.com/api/predict/xxxxxxxx'
# Use in EAS Queue
TOKEN = 'xxxxxxxx'
# Support TeaCache.
enable_teacache = True
# Recommended to be set between 0.05 and 0.30. A larger threshold can cache more steps, speeding up the inference process,
# but it may cause slight differences between the generated content and the original content.
# # --------------------------------------------------------------------------------------------------- #
# | Model Name | threshold | Model Name | threshold |
# | Wan2.2-T2V-A14B | 0.10~0.15 | Wan2.2-I2V-A14B | 0.15~0.20 |
# | Wan2.2-Fun-A14B-* | 0.15~0.20 |
# # --------------------------------------------------------------------------------------------------- #
teacache_threshold = 0.10
# The number of steps to skip TeaCache at the beginning of the inference process, which can
# reduce the impact of TeaCache on generated video quality.
num_skip_start_steps = 5
# Whether to offload TeaCache tensors to cpu to save a little bit of GPU memory.
teacache_offload = False
# Skip some cfg steps in inference
# Recommended to be set between 0.00 and 0.25
cfg_skip_ratio = 0
# Riflex config
enable_riflex = False
# Index of intrinsic frequency
riflex_k = 6
# "Video Generation" and "Image Generation"
generation_method = "Video Generation"
# Video length
length_slider = 81
# Used in Lora models
lora_model_path = "none"
lora_alpha_slider = 0.55
# Prompts
prompt_textbox = "一只棕色的狗摇着头，坐在舒适房间里的浅色沙发上。在狗的后面，架子上有一幅镶框的画，周围是粉红色的花朵。房间里柔和温暖的灯光营造出舒适的氛围。"
negative_prompt_textbox = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
# Sampler name
sampler_dropdown = "Flow"
# Sampler steps
sample_step_slider = 50
# height and width
width_slider = 832
height_slider = 480
# cfg scale
cfg_scale_slider = 6
seed_textbox = 43
# 起始图片路径
start_image_path = "asset/1.png" # 替换为实际的图片路径
outputs = post_infer(
generation_method,
length_slider,
lora_model_path=lora_model_path,
lora_alpha_slider=lora_alpha_slider,
prompt_textbox=prompt_textbox,
negative_prompt_textbox=negative_prompt_textbox,
sampler_dropdown=sampler_dropdown,
sample_step_slider=sample_step_slider,
width_slider=width_slider,
height_slider=height_slider,
cfg_scale_slider=cfg_scale_slider,
seed_textbox=seed_textbox,
enable_teacache = enable_teacache,
teacache_threshold = teacache_threshold,
num_skip_start_steps = num_skip_start_steps,
teacache_offload = teacache_offload,
cfg_skip_ratio = cfg_skip_ratio,
enable_riflex = enable_riflex,
riflex_k = riflex_k,
url=EAS_URL,
POST_TOKEN=TOKEN,
start_image=start_image_path # 传递起始图片路径
)
# Get decoded data
outputs = json.loads(base64.b64decode(json.loads(outputs)[0]['data']))
base64_encoding = outputs["base64_encoding"]
decoded_data = base64.b64decode(base64_encoding)
is_image = True if generation_method == "Image Generation" else False
if is_image or length_slider == 1:
file_path = "1.png"
else:
file_path = "1.mp4"
with open(file_path, "wb") as file:
file.write(decoded_data)
# End of record time
# The calculated time difference is the execution time of the program, expressed in seconds / s
time_end = time.time()
time_sum = (time_end - time_start)
print('# --------------------------------------------------------- #')
print(f'# Total expenditure: {time_sum}s')
print('# --------------------------------------------------------- #')
examples/wan2.2_fun/post_infer_queue_v2v_control.py
Executable
+230
View File
@@ -0,0 +1,230 @@
import base64
import json
import time
import urllib.parse
from io import BytesIO
import requests
from PIL import Image
def post_infer(
generation_method,
length_slider,
url='http://127.0.0.1:7860',
POST_TOKEN="",
timeout=5,
base_model_path="none",
lora_model_path="none",
lora_alpha_slider=0.55,
prompt_textbox="A young woman with beautiful and clear eyes and blonde hair standing and white dress in a forest wearing a crown. She seems to be lost in thought, and the camera focuses on her face. The video is of high quality, and the view is very clear. High quality, masterpiece, best quality, highres, ultra-detailed, fantastic.",
negative_prompt_textbox="色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走",
sampler_dropdown="Flow",
sample_step_slider=50,
width_slider=672,
height_slider=384,
cfg_scale_slider=6,
seed_textbox=43,
enable_teacache = None,
teacache_threshold = None,
num_skip_start_steps = None,
teacache_offload = None,
cfg_skip_ratio = None,
enable_riflex = None,
riflex_k = None,
control_video = None,
ref_image = None
):
if control_video:
try:
if not control_video.startswith("http"):
with open(control_video, "rb") as file:
video_data = file.read()
control_video = base64.b64encode(video_data).decode('utf-8')
except Exception as e:
print(f"Error processing control_video: {e}")
raise
if ref_image:
try:
if not ref_image.startswith("http"):
image = Image.open(ref_image).convert("RGB")
# 将图片转换为 Base64 编码
buffered = BytesIO()
image.save(buffered, format="JPEG")
ref_image = base64.b64encode(buffered.getvalue()).decode('utf-8')
except Exception as e:
print(f"Error processing ref_image: {e}")
raise
# Prepare the data payload
datas = json.dumps({
"base_model_path": base_model_path,
"lora_model_path": lora_model_path,
"lora_alpha_slider": lora_alpha_slider,
"prompt_textbox": prompt_textbox,
"negative_prompt_textbox": negative_prompt_textbox,
"sampler_dropdown": sampler_dropdown,
"sample_step_slider": sample_step_slider,
"width_slider": width_slider,
"height_slider": height_slider,
"generation_method": generation_method,
"length_slider": length_slider,
"cfg_scale_slider": cfg_scale_slider,
"seed_textbox": seed_textbox,
"ref_image": ref_image,
"enable_teacache": enable_teacache,
"teacache_threshold": teacache_threshold,
"num_skip_start_steps": num_skip_start_steps,
"teacache_offload": teacache_offload,
"cfg_skip_ratio": cfg_skip_ratio,
"enable_riflex": enable_riflex,
"riflex_k": riflex_k,
"control_video": control_video
})
# Initialize session and set headers
session = requests.session()
session.headers.update({"Authorization": POST_TOKEN})
# Send POST request
if url[-1] == "/":
url = url[:-1]
post_r = session.post(f'{url}/videox_fun/infer_forward', data=datas, timeout=timeout)
# Extract request ID from POST response headers
request_id = post_r.headers.get("X-Eas-Queueservice-Request-Id")
# Prepare query parameters for GET request
query = {
'_index_': '0',
'_length_': '1',
'_timeout_': str(timeout),
'_raw_': 'false',
'_auto_delete_': 'true',
}
if request_id:
query['requestId'] = request_id
query_str = urllib.parse.urlencode(query)
# Polling GET request until status code is not 204
status_code = 204
while status_code == 204:
if query_str:
get_r = session.get(f'{url}/sink?{query_str}', timeout=timeout)
else:
get_r = session.get(f'{url}/sink', timeout=timeout)
status_code = get_r.status_code
# Decode and return the response content
data = get_r.content.decode('utf-8')
return data
if __name__ == '__main__':
# initiate time
time_start = time.time()
# EAS队列配置
EAS_URL = 'http://17xxxxxxxxx.pai-eas.aliyuncs.com/api/predict/xxxxxxxx'
# Use in EAS Queue
TOKEN = 'xxxxxxxx'
# Support TeaCache.
enable_teacache = True
# Recommended to be set between 0.05 and 0.30. A larger threshold can cache more steps, speeding up the inference process,
# but it may cause slight differences between the generated content and the original content.
# # --------------------------------------------------------------------------------------------------- #
# | Model Name | threshold | Model Name | threshold |
# | Wan2.2-T2V-A14B | 0.10~0.15 | Wan2.2-I2V-A14B | 0.15~0.20 |
# | Wan2.2-Fun-A14B-* | 0.15~0.20 |
# # --------------------------------------------------------------------------------------------------- #
teacache_threshold = 0.10
# The number of steps to skip TeaCache at the beginning of the inference process, which can
# reduce the impact of TeaCache on generated video quality.
num_skip_start_steps = 5
# Whether to offload TeaCache tensors to cpu to save a little bit of GPU memory.
teacache_offload = False
# Skip some cfg steps in inference
# Recommended to be set between 0.00 and 0.25
cfg_skip_ratio = 0
# Riflex config
enable_riflex = False
# Index of intrinsic frequency
riflex_k = 6
# "Video Generation" and "Image Generation"
generation_method = "Video Generation"
# Video length
length_slider = 81
# Used in Lora models
lora_model_path = "none"
lora_alpha_slider = 0.55
# Prompts
prompt_textbox = "在这个阳光明媚的户外花园里，美女身穿一袭及膝的白色无袖连衣裙，裙摆在她轻盈的舞姿中轻柔地摆动，宛如一只翩翩起舞的蝴蝶。阳光透过树叶间洒下斑驳的光影，映衬出她柔和的脸庞和清澈的眼眸，显得格外优雅。仿佛每一个动作都在诉说着青春与活力，她在草地上旋转，裙摆随之飞扬，仿佛整个花园都因她的舞动而欢愉。周围五彩缤纷的花朵在微风中摇曳，玫瑰、菊花、百合，各自释放出阵阵香气，营造出一种轻松而愉快的氛围。"
negative_prompt_textbox = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
# Sampler name
sampler_dropdown = "Flow"
# Sampler steps
sample_step_slider = 50
# height and width
width_slider = 480
height_slider = 832
# cfg scale
cfg_scale_slider = 6
seed_textbox = 43
# 控制视频路径（可以是本地路径或 URL）
control_video_path = "asset/000000.mp4" # 替换为实际的视频路径
# 参考图片路径
ref_image_path = None # 替换为实际的图片路径
outputs = post_infer(
generation_method,
length_slider,
lora_model_path=lora_model_path,
lora_alpha_slider=lora_alpha_slider,
prompt_textbox=prompt_textbox,
negative_prompt_textbox=negative_prompt_textbox,
sampler_dropdown=sampler_dropdown,
sample_step_slider=sample_step_slider,
width_slider=width_slider,
height_slider=height_slider,
cfg_scale_slider=cfg_scale_slider,
seed_textbox=seed_textbox,
enable_teacache = enable_teacache,
teacache_threshold = teacache_threshold,
num_skip_start_steps = num_skip_start_steps,
teacache_offload = teacache_offload,
cfg_skip_ratio = cfg_skip_ratio,
enable_riflex = enable_riflex,
riflex_k = riflex_k,
url=EAS_URL,
POST_TOKEN=TOKEN,
control_video=control_video_path, # 传递控制视频路径
ref_image=ref_image_path # 传递参考图片路径
)
# Get decoded data
outputs = json.loads(base64.b64decode(json.loads(outputs)[0]['data']))
base64_encoding = outputs["base64_encoding"]
decoded_data = base64.b64decode(base64_encoding)
is_image = True if generation_method == "Image Generation" else False
if is_image or length_slider == 1:
file_path = "1.png"
else:
file_path = "1.mp4"
with open(file_path, "wb") as file:
file.write(decoded_data)
# End of record time
# The calculated time difference is the execution time of the program, expressed in seconds / s
time_end = time.time()
time_sum = (time_end - time_start)
print('# --------------------------------------------------------- #')
print(f'# Total expenditure: {time_sum}s')
print('# --------------------------------------------------------- #')
examples/wan2.2_fun/predict_i2v.py
+49 -34
View File
@@ -13,7 +13,7 @@ for project_root in project_roots:
sys.path.insert(0, project_root) if project_root not in sys.path else None
from videox_fun.dist import set_multi_gpus_devices, shard_model
from videox_fun.models import (AutoencoderKLWan, AutoTokenizer, CLIPModel,
from videox_fun.models import (AutoencoderKLWan, AutoencoderKLWan3_8, AutoTokenizer, CLIPModel,
WanT5EncoderModel, Wan2_2Transformer3DModel)
from videox_fun.models.cache_utils import get_teacache_coefficients
from videox_fun.pipeline import Wan2_2I2VPipeline
@@ -120,7 +120,7 @@ num_inference_steps = 50
# The lora_weight is used for low noise model, the lora_high_weight is used for high noise model.
lora_weight = 0.55
lora_high_weight = 0.55
save_path = "samples/wan-fun-videos-i2v"
save_path = "samples/wan-videos-fun-i2v"
device = set_multi_gpus_devices(ulysses_degree, ring_degree)
config = OmegaConf.load(config_path)
@@ -132,13 +132,15 @@ transformer = Wan2_2Transformer3DModel.from_pretrained(
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
if config['transformer_additional_kwargs'].get('transformer_combination_type', 'single') == "moe":
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
else:
transformer_2 = None
if transformer_path is not None:
print(f"From checkpoint: {transformer_path}")
@@ -152,21 +154,23 @@ if transformer_path is not None:
m, u = transformer.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
if transformer_high_path is not None:
print(f"From checkpoint: {transformer_high_path}")
if transformer_high_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_high_path)
else:
state_dict = torch.load(transformer_high_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
if transformer_2 is not None:
if transformer_high_path is not None:
print(f"From checkpoint: {transformer_high_path}")
if transformer_high_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_high_path)
else:
state_dict = torch.load(transformer_high_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = transformer_2.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
m, u = transformer_2.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
# Get Vae
Choosen_AutoencoderKL = {
"AutoencoderKLWan": AutoencoderKLWan,
"AutoencoderKLWan3_8": AutoencoderKLWan3_8
}[config['vae_kwargs'].get('vae_type', 'AutoencoderKLWan')]
vae = Choosen_AutoencoderKL.from_pretrained(
os.path.join(model_name, config['vae_kwargs'].get('vae_subpath', 'vae')),
@@ -223,11 +227,13 @@ pipeline = Wan2_2I2VPipeline(
if ulysses_degree > 1 or ring_degree > 1:
from functools import partial
transformer.enable_multi_gpus_inference()
transformer_2.enable_multi_gpus_inference()
if transformer_2 is not None:
transformer_2.enable_multi_gpus_inference()
if fsdp_dit:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
pipeline.transformer = shard_fn(pipeline.transformer)
pipeline.transformer_2 = shard_fn(pipeline.transformer_2)
if transformer_2 is not None:
pipeline.transformer_2 = shard_fn(pipeline.transformer_2)
print("Add FSDP DIT")
if fsdp_text_encoder:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
@@ -237,29 +243,33 @@ if ulysses_degree > 1 or ring_degree > 1:
if compile_dit:
for i in range(len(pipeline.transformer.blocks)):
pipeline.transformer.blocks[i] = torch.compile(pipeline.transformer.blocks[i])
for i in range(len(pipeline.transformer_2.blocks)):
pipeline.transformer_2.blocks[i] = torch.compile(pipeline.transformer_2.blocks[i])
if transformer_2 is not None:
for i in range(len(pipeline.transformer_2.blocks)):
pipeline.transformer_2.blocks[i] = torch.compile(pipeline.transformer_2.blocks[i])
print("Add Compile")
if GPU_memory_mode == "sequential_cpu_offload":
replace_parameters_by_name(transformer, ["modulation",], device=device)
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer.freqs = transformer.freqs.to(device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
if transformer_2 is not None:
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
pipeline.enable_sequential_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload":
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_full_load_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.to(device=device)
else:
pipeline.to(device=device)
@@ -270,18 +280,21 @@ if coefficients is not None:
pipeline.transformer.enable_teacache(
coefficients, num_inference_steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, num_inference_steps)
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
if transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator = torch.Generator(device=device).manual_seed(seed)
if lora_path is not None:
pipeline = merge_lora(pipeline, lora_path, lora_weight, device=device)
pipeline = merge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
if transformer_2 is not None:
pipeline = merge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
with torch.no_grad():
video_length = int((video_length - 1) // vae.config.temporal_compression_ratio * vae.config.temporal_compression_ratio) + 1 if video_length != 1 else 1
@@ -289,7 +302,8 @@ with torch.no_grad():
if enable_riflex:
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
if transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
input_video, input_video_mask, clip_image = get_image_to_video_latent(validation_image_start, validation_image_end, video_length=video_length, sample_size=sample_size)
@@ -311,7 +325,8 @@ with torch.no_grad():
if lora_path is not None:
pipeline = unmerge_lora(pipeline, lora_path, lora_weight, device=device)
pipeline = unmerge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
if transformer_2 is not None:
pipeline = unmerge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
def save_results():
if not os.path.exists(save_path):
examples/wan2.2_fun/predict_t2v.py
+336
View File
@@ -0,0 +1,336 @@
import os
import sys
import numpy as np
import torch
from diffusers import FlowMatchEulerDiscreteScheduler
from omegaconf import OmegaConf
from PIL import Image
current_file_path = os.path.abspath(__file__)
project_roots = [os.path.dirname(current_file_path), os.path.dirname(os.path.dirname(current_file_path)), os.path.dirname(os.path.dirname(os.path.dirname(current_file_path)))]
for project_root in project_roots:
sys.path.insert(0, project_root) if project_root not in sys.path else None
from videox_fun.dist import set_multi_gpus_devices, shard_model
from videox_fun.models import (AutoencoderKLWan, AutoTokenizer, CLIPModel,
WanT5EncoderModel, Wan2_2Transformer3DModel)
from videox_fun.models.cache_utils import get_teacache_coefficients
from videox_fun.pipeline import Wan2_2I2VPipeline
from videox_fun.utils.fp8_optimization import (convert_model_weight_to_float8, replace_parameters_by_name,
convert_weight_dtype_wrapper)
from videox_fun.utils.lora_utils import merge_lora, unmerge_lora
from videox_fun.utils.utils import (filter_kwargs, get_image_to_video_latent,
save_videos_grid)
from videox_fun.utils.fm_solvers import FlowDPMSolverMultistepScheduler
from videox_fun.utils.fm_solvers_unipc import FlowUniPCMultistepScheduler
# GPU memory mode, which can be choosen in [model_full_load, model_full_load_and_qfloat8, model_cpu_offload, model_cpu_offload_and_qfloat8, sequential_cpu_offload].
# model_full_load means that the entire model will be moved to the GPU.
#
# model_full_load_and_qfloat8 means that the entire model will be moved to the GPU,
# and the transformer model has been quantized to float8, which can save more GPU memory.
#
# model_cpu_offload means that the entire model will be moved to the CPU after use, which can save some GPU memory.
#
# model_cpu_offload_and_qfloat8 indicates that the entire model will be moved to the CPU after use,
# and the transformer model has been quantized to float8, which can save more GPU memory.
#
# sequential_cpu_offload means that each layer of the model will be moved to the CPU after use,
# resulting in slower speeds but saving a large amount of GPU memory.
GPU_memory_mode = "sequential_cpu_offload"
# Multi GPUs config
# Please ensure that the product of ulysses_degree and ring_degree equals the number of GPUs used.
# For example, if you are using 8 GPUs, you can set ulysses_degree = 2 and ring_degree = 4.
# If you are using 1 GPU, you can set ulysses_degree = 1 and ring_degree = 1.
ulysses_degree = 1
ring_degree = 1
# Use FSDP to save more GPU memory in multi gpus.
fsdp_dit = False
fsdp_text_encoder = True
# Compile will give a speedup in fixed resolution and need a little GPU memory.
# The compile_dit is not compatible with the fsdp_dit and sequential_cpu_offload.
compile_dit = False
# TeaCache config
enable_teacache = True
# Recommended to be set between 0.05 and 0.30. A larger threshold can cache more steps, speeding up the inference process,
# but it may cause slight differences between the generated content and the original content.
# # --------------------------------------------------------------------------------------------------- #
# | Model Name | threshold | Model Name | threshold |
# | Wan2.2-T2V-A14B | 0.10~0.15 | Wan2.2-I2V-A14B | 0.15~0.20 |
# | Wan2.2-Fun-A14B-* | 0.15~0.20 |
# # --------------------------------------------------------------------------------------------------- #
teacache_threshold = 0.10
# The number of steps to skip TeaCache at the beginning of the inference process, which can
# reduce the impact of TeaCache on generated video quality.
num_skip_start_steps = 5
# Whether to offload TeaCache tensors to cpu to save a little bit of GPU memory.
teacache_offload = False
# Skip some cfg steps in inference
# Recommended to be set between 0.00 and 0.25
cfg_skip_ratio = 0
# Riflex config
enable_riflex = False
# Index of intrinsic frequency
riflex_k = 6
# Config and model path
config_path = "config/wan2.2/wan_civitai_i2v.yaml"
# model path
model_name = "models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
# Choose the sampler in "Flow", "Flow_Unipc", "Flow_DPM++"
sampler_name = "Flow"
# [NOTE]: Noise schedule shift parameter. Affects temporal dynamics.
# Used when the sampler is in "Flow_Unipc", "Flow_DPM++".
shift = 5
# Load pretrained model if need
# The transformer_path is used for low noise model, the transformer_high_path is used for high noise model.
transformer_path = None
transformer_high_path = None
vae_path = None
# Load lora model if need
# The lora_path is used for low noise model, the lora_high_path is used for high noise model.
lora_path = None
lora_high_path = None
# Other params
sample_size = [480, 832]
video_length = 81
fps = 16
# Use torch.float16 if GPU does not support torch.bfloat16
# ome graphics cards, such as v100, 2080ti, do not support torch.bfloat16
weight_dtype = torch.bfloat16
# 使用更长的neg prompt如"模糊，突变，变形，失真，画面暗，文本字幕，画面固定，连环画，漫画，线稿，没有主体。"，可以增加稳定性
# 在neg prompt中添加"安静，固定"等词语可以增加动态性。
prompt = "一只棕色的狗摇着头，坐在舒适房间里的浅色沙发上。在狗的后面，架子上有一幅镶框的画，周围是粉红色的花朵。房间里柔和温暖的灯光营造出舒适的氛围。"
negative_prompt = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
guidance_scale = 6.0
seed = 43
num_inference_steps = 50
# The lora_weight is used for low noise model, the lora_high_weight is used for high noise model.
lora_weight = 0.55
lora_high_weight = 0.55
save_path = "samples/wan-fun-videos-i2v"
device = set_multi_gpus_devices(ulysses_degree, ring_degree)
config = OmegaConf.load(config_path)
boundary = config['transformer_additional_kwargs'].get('boundary', 0.900)
transformer = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_low_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
if transformer_path is not None:
print(f"From checkpoint: {transformer_path}")
if transformer_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_path)
else:
state_dict = torch.load(transformer_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = transformer.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
if transformer_high_path is not None:
print(f"From checkpoint: {transformer_high_path}")
if transformer_high_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_high_path)
else:
state_dict = torch.load(transformer_high_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = transformer_2.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
# Get Vae
Choosen_AutoencoderKL = {
"AutoencoderKLWan": AutoencoderKLWan,
"AutoencoderKLWan3_8": AutoencoderKLWan3_8
}[config['vae_kwargs'].get('vae_type', 'AutoencoderKLWan')]
vae = Choosen_AutoencoderKL.from_pretrained(
os.path.join(model_name, config['vae_kwargs'].get('vae_subpath', 'vae')),
additional_kwargs=OmegaConf.to_container(config['vae_kwargs']),
).to(weight_dtype)
if vae_path is not None:
print(f"From checkpoint: {vae_path}")
if vae_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(vae_path)
else:
state_dict = torch.load(vae_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = vae.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
# Get Tokenizer
tokenizer = AutoTokenizer.from_pretrained(
os.path.join(model_name, config['text_encoder_kwargs'].get('tokenizer_subpath', 'tokenizer')),
)
# Get Text encoder
text_encoder = WanT5EncoderModel.from_pretrained(
os.path.join(model_name, config['text_encoder_kwargs'].get('text_encoder_subpath', 'text_encoder')),
additional_kwargs=OmegaConf.to_container(config['text_encoder_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
text_encoder = text_encoder.eval()
# Get Scheduler
Choosen_Scheduler = scheduler_dict = {
"Flow": FlowMatchEulerDiscreteScheduler,
"Flow_Unipc": FlowUniPCMultistepScheduler,
"Flow_DPM++": FlowDPMSolverMultistepScheduler,
}[sampler_name]
if sampler_name == "Flow_Unipc" or sampler_name == "Flow_DPM++":
config['scheduler_kwargs']['shift'] = 1
scheduler = Choosen_Scheduler(
**filter_kwargs(Choosen_Scheduler, OmegaConf.to_container(config['scheduler_kwargs']))
)
# Get Pipeline
pipeline = Wan2_2I2VPipeline(
transformer=transformer,
transformer_2=transformer_2,
vae=vae,
tokenizer=tokenizer,
text_encoder=text_encoder,
scheduler=scheduler,
)
if ulysses_degree > 1 or ring_degree > 1:
from functools import partial
transformer.enable_multi_gpus_inference()
transformer_2.enable_multi_gpus_inference()
if fsdp_dit:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
pipeline.transformer = shard_fn(pipeline.transformer)
pipeline.transformer_2 = shard_fn(pipeline.transformer_2)
print("Add FSDP DIT")
if fsdp_text_encoder:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
pipeline.text_encoder = shard_fn(pipeline.text_encoder)
print("Add FSDP TEXT ENCODER")
if compile_dit:
for i in range(len(pipeline.transformer.blocks)):
pipeline.transformer.blocks[i] = torch.compile(pipeline.transformer.blocks[i])
for i in range(len(pipeline.transformer_2.blocks)):
pipeline.transformer_2.blocks[i] = torch.compile(pipeline.transformer_2.blocks[i])
print("Add Compile")
if GPU_memory_mode == "sequential_cpu_offload":
replace_parameters_by_name(transformer, ["modulation",], device=device)
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer.freqs = transformer.freqs.to(device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
pipeline.enable_sequential_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload":
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_full_load_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.to(device=device)
else:
pipeline.to(device=device)
coefficients = get_teacache_coefficients(model_name) if enable_teacache else None
if coefficients is not None:
print(f"Enable TeaCache with threshold {teacache_threshold} and skip the first {num_skip_start_steps} steps.")
pipeline.transformer.enable_teacache(
coefficients, num_inference_steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, num_inference_steps)
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator = torch.Generator(device=device).manual_seed(seed)
if lora_path is not None:
pipeline = merge_lora(pipeline, lora_path, lora_weight, device=device)
pipeline = merge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
with torch.no_grad():
video_length = int((video_length - 1) // vae.config.temporal_compression_ratio * vae.config.temporal_compression_ratio) + 1 if video_length != 1 else 1
latent_frames = (video_length - 1) // vae.config.temporal_compression_ratio + 1
if enable_riflex:
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
input_video, input_video_mask, _ = get_image_to_video_latent(None, None, video_length=video_length, sample_size=sample_size)
sample = pipeline(
prompt,
num_frames = video_length,
negative_prompt = negative_prompt,
height = sample_size[0],
width = sample_size[1],
generator = generator,
guidance_scale = guidance_scale,
num_inference_steps = num_inference_steps,
video = input_video,
mask_video = input_video_mask,
boundary = boundary,
shift = shift,
).videos
if lora_path is not None:
pipeline = unmerge_lora(pipeline, lora_path, lora_weight, device=device)
pipeline = unmerge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
def save_results():
if not os.path.exists(save_path):
os.makedirs(save_path, exist_ok=True)
index = len([path for path in os.listdir(save_path)]) + 1
prefix = str(index).zfill(8)
if video_length == 1:
video_path = os.path.join(save_path, prefix + ".png")
image = sample[0, :, 0]
image = image.transpose(0, 1).transpose(1, 2)
image = (image * 255).numpy().astype(np.uint8)
image = Image.fromarray(image)
image.save(video_path)
else:
video_path = os.path.join(save_path, prefix + ".mp4")
save_videos_grid(sample, video_path, fps=fps)
if ulysses_degree * ring_degree > 1:
import torch.distributed as dist
if dist.get_rank() == 0:
save_results()
else:
save_results()
examples/wan2.2_fun/predict_v2v_control.py
+50 -34
View File
@@ -14,7 +14,7 @@ for project_root in project_roots:
sys.path.insert(0, project_root) if project_root not in sys.path else None
from videox_fun.dist import set_multi_gpus_devices, shard_model
from videox_fun.models import (AutoencoderKLWan, AutoTokenizer, CLIPModel,
from videox_fun.models import (AutoencoderKLWan, AutoencoderKLWan3_8, AutoTokenizer, CLIPModel,
WanT5EncoderModel, Wan2_2Transformer3DModel)
from videox_fun.data.dataset_image_video import process_pose_file
from videox_fun.models.cache_utils import get_teacache_coefficients
@@ -128,7 +128,7 @@ negative_prompt = "色调艳丽，过曝，静态，细节模糊不清，字
# prompt = "A young woman with beautiful, clear eyes and blonde hair stands in the forest, wearing a white dress and a crown. Her expression is serene, reminiscent of a movie star, with fair and youthful skin. Her brown long hair flows in the wind. The video quality is very high, with a clear view. High quality, masterpiece, best quality, high resolution, ultra-fine, fantastical."
# negative_prompt = "Twisted body, limb deformities, text captions, comic, static, ugly, error, messy code."
guidance_scale = 6.0
seed = 42
seed = 43
num_inference_steps = 50
# The lora_weight is used for low noise model, the lora_high_weight is used for high noise model.
lora_weight = 0.55
@@ -145,13 +145,15 @@ transformer = Wan2_2Transformer3DModel.from_pretrained(
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
if config['transformer_additional_kwargs'].get('transformer_combination_type', 'single') == "moe":
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
else:
transformer_2 = None
if transformer_path is not None:
print(f"From checkpoint: {transformer_path}")
@@ -165,21 +167,23 @@ if transformer_path is not None:
m, u = transformer.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
if transformer_high_path is not None:
print(f"From checkpoint: {transformer_high_path}")
if transformer_high_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_high_path)
else:
state_dict = torch.load(transformer_high_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
if transformer_2 is not None:
if transformer_high_path is not None:
print(f"From checkpoint: {transformer_high_path}")
if transformer_high_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_high_path)
else:
state_dict = torch.load(transformer_high_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = transformer_2.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
m, u = transformer_2.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
# Get Vae
Choosen_AutoencoderKL = {
"AutoencoderKLWan": AutoencoderKLWan,
"AutoencoderKLWan3_8": AutoencoderKLWan3_8
}[config['vae_kwargs'].get('vae_type', 'AutoencoderKLWan')]
vae = Choosen_AutoencoderKL.from_pretrained(
os.path.join(model_name, config['vae_kwargs'].get('vae_subpath', 'vae')),
@@ -236,11 +240,13 @@ pipeline = Wan2_2FunControlPipeline(
if ulysses_degree > 1 or ring_degree > 1:
from functools import partial
transformer.enable_multi_gpus_inference()
transformer_2.enable_multi_gpus_inference()
if transformer_2 is not None:
transformer_2.enable_multi_gpus_inference()
if fsdp_dit:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
pipeline.transformer = shard_fn(pipeline.transformer)
pipeline.transformer_2 = shard_fn(pipeline.transformer_2)
if transformer_2 is not None:
pipeline.transformer_2 = shard_fn(pipeline.transformer_2)
print("Add FSDP DIT")
if fsdp_text_encoder:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
@@ -250,29 +256,33 @@ if ulysses_degree > 1 or ring_degree > 1:
if compile_dit:
for i in range(len(pipeline.transformer.blocks)):
pipeline.transformer.blocks[i] = torch.compile(pipeline.transformer.blocks[i])
for i in range(len(pipeline.transformer_2.blocks)):
pipeline.transformer_2.blocks[i] = torch.compile(pipeline.transformer_2.blocks[i])
if transformer_2 is not None:
for i in range(len(pipeline.transformer_2.blocks)):
pipeline.transformer_2.blocks[i] = torch.compile(pipeline.transformer_2.blocks[i])
print("Add Compile")
if GPU_memory_mode == "sequential_cpu_offload":
replace_parameters_by_name(transformer, ["modulation",], device=device)
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer.freqs = transformer.freqs.to(device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
if transformer_2 is not None:
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
pipeline.enable_sequential_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload":
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload_and_qfloat8":
elif GPU_memory_mode == "model_full_load_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.to(device=device)
else:
pipeline.to(device=device)
@@ -283,18 +293,21 @@ if coefficients is not None:
pipeline.transformer.enable_teacache(
coefficients, num_inference_steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, num_inference_steps)
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
if transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator = torch.Generator(device=device).manual_seed(seed)
if lora_path is not None:
pipeline = merge_lora(pipeline, lora_path, lora_weight, device=device)
pipeline = merge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
if transformer_2 is not None:
pipeline = merge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
with torch.no_grad():
video_length = int((video_length - 1) // vae.config.temporal_compression_ratio * vae.config.temporal_compression_ratio) + 1 if video_length != 1 else 1
@@ -302,6 +315,8 @@ with torch.no_grad():
if enable_riflex:
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
if transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
inpaint_video, inpaint_video_mask, clip_image = get_image_to_video_latent(start_image, end_image, video_length=video_length, sample_size=sample_size)
@@ -337,7 +352,8 @@ with torch.no_grad():
if lora_path is not None:
pipeline = unmerge_lora(pipeline, lora_path, lora_weight, device=device)
pipeline = unmerge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
if transformer_2 is not None:
pipeline = unmerge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
def save_results():
if not os.path.exists(save_path):
examples/wan2.2_fun/predict_v2v_control_camera.py
+381
View File
@@ -0,0 +1,381 @@
import os
import sys
import numpy as np
import torch
from diffusers import FlowMatchEulerDiscreteScheduler
from omegaconf import OmegaConf
from PIL import Image
from transformers import AutoTokenizer
current_file_path = os.path.abspath(__file__)
project_roots = [os.path.dirname(current_file_path), os.path.dirname(os.path.dirname(current_file_path)), os.path.dirname(os.path.dirname(os.path.dirname(current_file_path)))]
for project_root in project_roots:
sys.path.insert(0, project_root) if project_root not in sys.path else None
from videox_fun.dist import set_multi_gpus_devices, shard_model
from videox_fun.models import (AutoencoderKLWan, AutoencoderKLWan3_8, AutoTokenizer, CLIPModel,
WanT5EncoderModel, Wan2_2Transformer3DModel)
from videox_fun.data.dataset_image_video import process_pose_file
from videox_fun.models.cache_utils import get_teacache_coefficients
from videox_fun.pipeline import Wan2_2FunControlPipeline, WanPipeline
from videox_fun.utils.fp8_optimization import (convert_model_weight_to_float8,
convert_weight_dtype_wrapper,
replace_parameters_by_name)
from videox_fun.utils.lora_utils import merge_lora, unmerge_lora
from videox_fun.utils.utils import (filter_kwargs, get_image_latent, get_image_to_video_latent,
get_video_to_video_latent,
save_videos_grid)
from videox_fun.utils.fm_solvers import FlowDPMSolverMultistepScheduler
from videox_fun.utils.fm_solvers_unipc import FlowUniPCMultistepScheduler
# GPU memory mode, which can be choosen in [model_full_load, model_cpu_offload_and_qfloat8, model_cpu_offload, model_cpu_offload_and_qfloat8, sequential_cpu_offload].
# model_full_load means that the entire model will be moved to the GPU.
#
# model_full_load_and_qfloat8 means that the entire model will be moved to the GPU,
# and the transformer model has been quantized to float8, which can save more GPU memory.
#
# model_cpu_offload means that the entire model will be moved to the CPU after use, which can save some GPU memory.
#
# model_cpu_offload_and_qfloat8 indicates that the entire model will be moved to the CPU after use,
# and the transformer model has been quantized to float8, which can save more GPU memory.
#
# sequential_cpu_offload means that each layer of the model will be moved to the CPU after use,
# resulting in slower speeds but saving a large amount of GPU memory.
GPU_memory_mode = "sequential_cpu_offload"
# Multi GPUs config
# Please ensure that the product of ulysses_degree and ring_degree equals the number of GPUs used.
# For example, if you are using 8 GPUs, you can set ulysses_degree = 2 and ring_degree = 4.
# If you are using 1 GPU, you can set ulysses_degree = 1 and ring_degree = 1.
ulysses_degree = 1
ring_degree = 1
# Use FSDP to save more GPU memory in multi gpus.
fsdp_dit = False
fsdp_text_encoder = True
# Compile will give a speedup in fixed resolution and need a little GPU memory.
# The compile_dit is not compatible with the fsdp_dit and sequential_cpu_offload.
compile_dit = False
# Support TeaCache.
enable_teacache = True
# Recommended to be set between 0.05 and 0.30. A larger threshold can cache more steps, speeding up the inference process,
# but it may cause slight differences between the generated content and the original content.
# # --------------------------------------------------------------------------------------------------- #
# | Model Name | threshold | Model Name | threshold |
# | Wan2.2-T2V-A14B | 0.10~0.15 | Wan2.2-I2V-A14B | 0.15~0.20 |
# | Wan2.2-Fun-A14B-* | 0.15~0.20 |
# # --------------------------------------------------------------------------------------------------- #
teacache_threshold = 0.10
# The number of steps to skip TeaCache at the beginning of the inference process, which can
# reduce the impact of TeaCache on generated video quality.
num_skip_start_steps = 5
# Whether to offload TeaCache tensors to cpu to save a little bit of GPU memory.
teacache_offload = False
# Skip some cfg steps in inference
# Recommended to be set between 0.00 and 0.25
cfg_skip_ratio = 0
# Riflex config
enable_riflex = False
# Index of intrinsic frequency
riflex_k = 6
# Config and model path
config_path = "config/wan2.2/wan_civitai_i2v.yaml"
# model path
model_name = "models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control-Camera"
# Choose the sampler in "Flow", "Flow_Unipc", "Flow_DPM++"
sampler_name = "Flow"
# [NOTE]: Noise schedule shift parameter. Affects temporal dynamics.
# Used when the sampler is in "Flow_Unipc", "Flow_DPM++".
# If you want to generate a 480p video, it is recommended to set the shift value to 3.0.
# If you want to generate a 720p video, it is recommended to set the shift value to 5.0.
shift = 5
# Load pretrained model if need
# The transformer_path is used for low noise model, the transformer_high_path is used for high noise model.
transformer_path = None
transformer_high_path = None
vae_path = None
# Load lora model if need
# The lora_path is used for low noise model, the lora_high_path is used for high noise model.
lora_path = None
lora_high_path = None
# Other params
sample_size = [480, 832]
video_length = 81
fps = 16
# Use torch.float16 if GPU does not support torch.bfloat16
# ome graphics cards, such as v100, 2080ti, do not support torch.bfloat16
weight_dtype = torch.bfloat16
control_video = None
control_camera_txt = "asset/Zoom_In.txt"
start_image = "asset/7.png"
end_image = None
ref_image = None
# 使用更长的neg prompt如"模糊，突变，变形，失真，画面暗，文本字幕，画面固定，连环画，漫画，线稿，没有主体。"，可以增加稳定性
# 在neg prompt中添加"安静，固定"等词语可以增加动态性。
prompt = "一个小女孩正在户外玩耍。她穿着一件蓝色的短袖上衣和粉色的短裤，头发扎成一个可爱的辫子。她的脚上没有穿鞋，显得非常自然和随意。她正用一把红色的小铲子在泥土里挖土，似乎在进行某种有趣的活动，可能是种花或是挖掘宝藏。地上有一根长长的水管，可能是用来浇水的。背景是一片草地和一些绿色植物，阳光明媚，整个场景充满了童趣和生机。小女孩专注的表情和认真的动作让人感受到她的快乐和好奇心。"
negative_prompt = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
# Using longer neg prompt such as "Blurring, mutation, deformation, distortion, dark and solid, comics, text subtitles, line art." can increase stability
# Adding words such as "quiet, solid" to the neg prompt can increase dynamism.
# prompt = "A young woman with beautiful, clear eyes and blonde hair stands in the forest, wearing a white dress and a crown. Her expression is serene, reminiscent of a movie star, with fair and youthful skin. Her brown long hair flows in the wind. The video quality is very high, with a clear view. High quality, masterpiece, best quality, high resolution, ultra-fine, fantastical."
# negative_prompt = "Twisted body, limb deformities, text captions, comic, static, ugly, error, messy code."
guidance_scale = 6.0
seed = 42
num_inference_steps = 50
# The lora_weight is used for low noise model, the lora_high_weight is used for high noise model.
lora_weight = 0.55
lora_high_weight = 0.55
save_path = "samples/wan-videos-fun-control"
device = set_multi_gpus_devices(ulysses_degree, ring_degree)
config = OmegaConf.load(config_path)
boundary = config['transformer_additional_kwargs'].get('boundary', 0.875)
transformer = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_low_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
if config['transformer_additional_kwargs'].get('transformer_combination_type', 'single') == "moe":
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
else:
transformer_2 = None
if transformer_path is not None:
print(f"From checkpoint: {transformer_path}")
if transformer_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_path)
else:
state_dict = torch.load(transformer_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = transformer.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
if transformer_2 is not None:
if transformer_high_path is not None:
print(f"From checkpoint: {transformer_high_path}")
if transformer_high_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_high_path)
else:
state_dict = torch.load(transformer_high_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = transformer_2.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
# Get Vae
Choosen_AutoencoderKL = {
"AutoencoderKLWan": AutoencoderKLWan,
"AutoencoderKLWan3_8": AutoencoderKLWan3_8
}[config['vae_kwargs'].get('vae_type', 'AutoencoderKLWan')]
vae = Choosen_AutoencoderKL.from_pretrained(
os.path.join(model_name, config['vae_kwargs'].get('vae_subpath', 'vae')),
additional_kwargs=OmegaConf.to_container(config['vae_kwargs']),
).to(weight_dtype)
if vae_path is not None:
print(f"From checkpoint: {vae_path}")
if vae_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(vae_path)
else:
state_dict = torch.load(vae_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = vae.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
# Get Tokenizer
tokenizer = AutoTokenizer.from_pretrained(
os.path.join(model_name, config['text_encoder_kwargs'].get('tokenizer_subpath', 'tokenizer')),
)
# Get Text encoder
text_encoder = WanT5EncoderModel.from_pretrained(
os.path.join(model_name, config['text_encoder_kwargs'].get('text_encoder_subpath', 'text_encoder')),
additional_kwargs=OmegaConf.to_container(config['text_encoder_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
text_encoder = text_encoder.eval()
# Get Scheduler
Choosen_Scheduler = scheduler_dict = {
"Flow": FlowMatchEulerDiscreteScheduler,
"Flow_Unipc": FlowUniPCMultistepScheduler,
"Flow_DPM++": FlowDPMSolverMultistepScheduler,
}[sampler_name]
if sampler_name == "Flow_Unipc" or sampler_name == "Flow_DPM++":
config['scheduler_kwargs']['shift'] = 1
scheduler = Choosen_Scheduler(
**filter_kwargs(Choosen_Scheduler, OmegaConf.to_container(config['scheduler_kwargs']))
)
# Get Pipeline
pipeline = Wan2_2FunControlPipeline(
transformer=transformer,
transformer_2=transformer_2,
vae=vae,
tokenizer=tokenizer,
text_encoder=text_encoder,
scheduler=scheduler,
)
if ulysses_degree > 1 or ring_degree > 1:
from functools import partial
transformer.enable_multi_gpus_inference()
if transformer_2 is not None:
transformer_2.enable_multi_gpus_inference()
if fsdp_dit:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
pipeline.transformer = shard_fn(pipeline.transformer)
if transformer_2 is not None:
pipeline.transformer_2 = shard_fn(pipeline.transformer_2)
print("Add FSDP DIT")
if fsdp_text_encoder:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
pipeline.text_encoder = shard_fn(pipeline.text_encoder)
print("Add FSDP TEXT ENCODER")
if compile_dit:
for i in range(len(pipeline.transformer.blocks)):
pipeline.transformer.blocks[i] = torch.compile(pipeline.transformer.blocks[i])
if transformer_2 is not None:
for i in range(len(pipeline.transformer_2.blocks)):
pipeline.transformer_2.blocks[i] = torch.compile(pipeline.transformer_2.blocks[i])
print("Add Compile")
if GPU_memory_mode == "sequential_cpu_offload":
replace_parameters_by_name(transformer, ["modulation",], device=device)
transformer.freqs = transformer.freqs.to(device=device)
if transformer_2 is not None:
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
pipeline.enable_sequential_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload":
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_full_load_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.to(device=device)
else:
pipeline.to(device=device)
coefficients = get_teacache_coefficients(model_name) if enable_teacache else None
if coefficients is not None:
print(f"Enable TeaCache with threshold {teacache_threshold} and skip the first {num_skip_start_steps} steps.")
pipeline.transformer.enable_teacache(
coefficients, num_inference_steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
if transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, num_inference_steps)
if transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator = torch.Generator(device=device).manual_seed(seed)
if lora_path is not None:
pipeline = merge_lora(pipeline, lora_path, lora_weight, device=device)
if transformer_2 is not None:
pipeline = merge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
with torch.no_grad():
video_length = int((video_length - 1) // vae.config.temporal_compression_ratio * vae.config.temporal_compression_ratio) + 1 if video_length != 1 else 1
latent_frames = (video_length - 1) // vae.config.temporal_compression_ratio + 1
if enable_riflex:
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
if transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
inpaint_video, inpaint_video_mask, clip_image = get_image_to_video_latent(start_image, end_image, video_length=video_length, sample_size=sample_size)
if ref_image is not None:
ref_image = get_image_latent(ref_image, sample_size=sample_size)
if control_camera_txt is not None:
input_video, input_video_mask = None, None
control_camera_video = process_pose_file(control_camera_txt, sample_size[1], sample_size[0])
control_camera_video = control_camera_video[:video_length].permute([3, 0, 1, 2]).unsqueeze(0)
else:
input_video, input_video_mask, _, _ = get_video_to_video_latent(control_video, video_length=video_length, sample_size=sample_size, fps=fps, ref_image=None)
control_camera_video = None
sample = pipeline(
prompt,
num_frames = video_length,
negative_prompt = negative_prompt,
height = sample_size[0],
width = sample_size[1],
generator = generator,
guidance_scale = guidance_scale,
num_inference_steps = num_inference_steps,
video = inpaint_video,
mask_video = inpaint_video_mask,
control_video = input_video,
control_camera_video = control_camera_video,
ref_image = ref_image,
boundary = boundary,
shift = shift,
).videos
if lora_path is not None:
pipeline = unmerge_lora(pipeline, lora_path, lora_weight, device=device)
if transformer_2 is not None:
pipeline = unmerge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
def save_results():
if not os.path.exists(save_path):
os.makedirs(save_path, exist_ok=True)
index = len([path for path in os.listdir(save_path)]) + 1
prefix = str(index).zfill(8)
if video_length == 1:
video_path = os.path.join(save_path, prefix + ".png")
image = sample[0, :, 0]
image = image.transpose(0, 1).transpose(1, 2)
image = (image * 255).numpy().astype(np.uint8)
image = Image.fromarray(image)
image.save(video_path)
else:
video_path = os.path.join(save_path, prefix + ".mp4")
save_videos_grid(sample, video_path, fps=fps)
if ulysses_degree * ring_degree > 1:
import torch.distributed as dist
if dist.get_rank() == 0:
save_results()
else:
save_results()
examples/wan2.2_fun/predict_v2v_control_ref.py
+51 -35
View File
@@ -14,7 +14,7 @@ for project_root in project_roots:
sys.path.insert(0, project_root) if project_root not in sys.path else None
from videox_fun.dist import set_multi_gpus_devices, shard_model
from videox_fun.models import (AutoencoderKLWan, AutoTokenizer, CLIPModel,
from videox_fun.models import (AutoencoderKLWan, AutoencoderKLWan3_8, AutoTokenizer, CLIPModel,
WanT5EncoderModel, Wan2_2Transformer3DModel)
from videox_fun.data.dataset_image_video import process_pose_file
from videox_fun.models.cache_utils import get_teacache_coefficients
@@ -29,7 +29,7 @@ from videox_fun.utils.utils import (filter_kwargs, get_image_latent, get_image_t
from videox_fun.utils.fm_solvers import FlowDPMSolverMultistepScheduler
from videox_fun.utils.fm_solvers_unipc import FlowUniPCMultistepScheduler
# GPU memory mode, which can be choosen in [model_full_load, model_cpu_offload_and_qfloat8, model_cpu_offload, model_cpu_offload_and_qfloat8, sequential_cpu_offload].
# GPU memory mode, which can be choosen in [model_full_load, model_full_load_and_qfloat8, model_cpu_offload, model_cpu_offload_and_qfloat8, sequential_cpu_offload].
# model_full_load means that the entire model will be moved to the GPU.
#
# model_full_load_and_qfloat8 means that the entire model will be moved to the GPU,
@@ -128,7 +128,7 @@ negative_prompt = "色调艳丽，过曝，静态，细节模糊不清，字
# prompt = "A young woman with beautiful, clear eyes and blonde hair stands in the forest, wearing a white dress and a crown. Her expression is serene, reminiscent of a movie star, with fair and youthful skin. Her brown long hair flows in the wind. The video quality is very high, with a clear view. High quality, masterpiece, best quality, high resolution, ultra-fine, fantastical."
# negative_prompt = "Twisted body, limb deformities, text captions, comic, static, ugly, error, messy code."
guidance_scale = 6.0
seed = 42
seed = 43
num_inference_steps = 50
# The lora_weight is used for low noise model, the lora_high_weight is used for high noise model.
lora_weight = 0.55
@@ -145,13 +145,15 @@ transformer = Wan2_2Transformer3DModel.from_pretrained(
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
if config['transformer_additional_kwargs'].get('transformer_combination_type', 'single') == "moe":
transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(model_name, config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=weight_dtype,
)
else:
transformer_2 = None
if transformer_path is not None:
print(f"From checkpoint: {transformer_path}")
@@ -165,21 +167,23 @@ if transformer_path is not None:
m, u = transformer.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
if transformer_high_path is not None:
print(f"From checkpoint: {transformer_high_path}")
if transformer_high_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_high_path)
else:
state_dict = torch.load(transformer_high_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
if transformer_2 is not None:
if transformer_high_path is not None:
print(f"From checkpoint: {transformer_high_path}")
if transformer_high_path.endswith("safetensors"):
from safetensors.torch import load_file, safe_open
state_dict = load_file(transformer_high_path)
else:
state_dict = torch.load(transformer_high_path, map_location="cpu")
state_dict = state_dict["state_dict"] if "state_dict" in state_dict else state_dict
m, u = transformer_2.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
m, u = transformer_2.load_state_dict(state_dict, strict=False)
print(f"missing keys: {len(m)}, unexpected keys: {len(u)}")
# Get Vae
Choosen_AutoencoderKL = {
"AutoencoderKLWan": AutoencoderKLWan,
"AutoencoderKLWan3_8": AutoencoderKLWan3_8
}[config['vae_kwargs'].get('vae_type', 'AutoencoderKLWan')]
vae = Choosen_AutoencoderKL.from_pretrained(
os.path.join(model_name, config['vae_kwargs'].get('vae_subpath', 'vae')),
@@ -236,11 +240,13 @@ pipeline = Wan2_2FunControlPipeline(
if ulysses_degree > 1 or ring_degree > 1:
from functools import partial
transformer.enable_multi_gpus_inference()
transformer_2.enable_multi_gpus_inference()
if transformer_2 is not None:
transformer_2.enable_multi_gpus_inference()
if fsdp_dit:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
pipeline.transformer = shard_fn(pipeline.transformer)
pipeline.transformer_2 = shard_fn(pipeline.transformer_2)
if transformer_2 is not None:
pipeline.transformer_2 = shard_fn(pipeline.transformer_2)
print("Add FSDP DIT")
if fsdp_text_encoder:
shard_fn = partial(shard_model, device_id=device, param_dtype=weight_dtype)
@@ -250,29 +256,33 @@ if ulysses_degree > 1 or ring_degree > 1:
if compile_dit:
for i in range(len(pipeline.transformer.blocks)):
pipeline.transformer.blocks[i] = torch.compile(pipeline.transformer.blocks[i])
for i in range(len(pipeline.transformer_2.blocks)):
pipeline.transformer_2.blocks[i] = torch.compile(pipeline.transformer_2.blocks[i])
if transformer_2 is not None:
for i in range(len(pipeline.transformer_2.blocks)):
pipeline.transformer_2.blocks[i] = torch.compile(pipeline.transformer_2.blocks[i])
print("Add Compile")
if GPU_memory_mode == "sequential_cpu_offload":
replace_parameters_by_name(transformer, ["modulation",], device=device)
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer.freqs = transformer.freqs.to(device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
if transformer_2 is not None:
replace_parameters_by_name(transformer_2, ["modulation",], device=device)
transformer_2.freqs = transformer_2.freqs.to(device=device)
pipeline.enable_sequential_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload":
pipeline.enable_model_cpu_offload(device=device)
elif GPU_memory_mode == "model_cpu_offload_and_qfloat8":
elif GPU_memory_mode == "model_full_load_and_qfloat8":
convert_model_weight_to_float8(transformer, exclude_module_name=["modulation",], device=device)
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer, weight_dtype)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
if transformer_2 is not None:
convert_model_weight_to_float8(transformer_2, exclude_module_name=["modulation",], device=device)
convert_weight_dtype_wrapper(transformer_2, weight_dtype)
pipeline.to(device=device)
else:
pipeline.to(device=device)
@@ -283,18 +293,21 @@ if coefficients is not None:
pipeline.transformer.enable_teacache(
coefficients, num_inference_steps, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if transformer_2 is not None:
pipeline.transformer_2.share_teacache(transformer=pipeline.transformer)
if cfg_skip_ratio is not None:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, num_inference_steps)
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
if transformer_2 is not None:
pipeline.transformer_2.share_cfg_skip(transformer=pipeline.transformer)
generator = torch.Generator(device=device).manual_seed(seed)
if lora_path is not None:
pipeline = merge_lora(pipeline, lora_path, lora_weight, device=device)
pipeline = merge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
if transformer_2 is not None:
pipeline = merge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
with torch.no_grad():
video_length = int((video_length - 1) // vae.config.temporal_compression_ratio * vae.config.temporal_compression_ratio) + 1 if video_length != 1 else 1
@@ -302,6 +315,8 @@ with torch.no_grad():
if enable_riflex:
pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames)
if transformer_2 is not None:
pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames)
inpaint_video, inpaint_video_mask, clip_image = get_image_to_video_latent(start_image, end_image, video_length=video_length, sample_size=sample_size)
@@ -337,7 +352,8 @@ with torch.no_grad():
if lora_path is not None:
pipeline = unmerge_lora(pipeline, lora_path, lora_weight, device=device)
pipeline = unmerge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
if transformer_2 is not None:
pipeline = unmerge_lora(pipeline, lora_high_path, lora_high_weight, device=device, sub_transformer_name="transformer_2")
def save_results():
if not os.path.exists(save_path):
scripts/wan2.1_fun/README_TRAIN.md
+2 -2
View File
@@ -135,7 +135,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag true --use_deepspeed --deepspeed_config_file config/zero_stage3_config.json --deepspeed_multinode_launcher standard scripts/wan2.1/train.py \
accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag true --use_deepspeed --deepspeed_config_file config/zero_stage3_config.json --deepspeed_multinode_launcher standard scripts/wan2.1_fun/train.py \
--config_path="config/wan2.1/wan_civitai.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
@@ -184,7 +184,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP --fsdp_transformer_layer_cls_to_wrap=WanAttentionBlock --fsdp_sharding_strategy "FULL_SHARD" --fsdp_state_dict_type=SHARDED_STATE_DICT --fsdp_backward_prefetch "BACKWARD_PRE" --fsdp_cpu_ram_efficient_loading False scripts/wan2.1/train.py \
accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP --fsdp_transformer_layer_cls_to_wrap=WanAttentionBlock --fsdp_sharding_strategy "FULL_SHARD" --fsdp_state_dict_type=SHARDED_STATE_DICT --fsdp_backward_prefetch "BACKWARD_PRE" --fsdp_cpu_ram_efficient_loading False scripts/wan2.1_fun/train.py \
--config_path="config/wan2.1/wan_civitai.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
scripts/wan2.2_fun/README_TRAIN.md
Executable
+227
View File
@@ -0,0 +1,227 @@
## Training Code
We can choose whether to use deep speed in Wan, which can save a lot of video memory.
Some parameters in the sh file can be confusing, and they are explained in this document:
- `enable_bucket` is used to enable bucket training. When enabled, the model does not crop the images and videos at the center, but instead, it trains the entire images and videos after grouping them into buckets based on resolution.
- `random_frame_crop` is used for random cropping on video frames to simulate videos with different frame counts.
- `random_hw_adapt` is used to enable automatic height and width scaling for images and videos. When `random_hw_adapt` is enabled, the training images will have their height and width set to `image_sample_size` as the maximum and `min(video_sample_size, 512)` as the minimum. For training videos, the height and width will be set to `image_sample_size` as the maximum and `min(video_sample_size, 512)` as the minimum.
- For example, when `random_hw_adapt` is enabled, with `video_sample_n_frames=49`, `video_sample_size=1024`, and `image_sample_size=1024`, the resolution of image inputs for training is `512x512` to `1024x1024`, and the resolution of video inputs for training is `512x512x49` to `1024x1024x49`.
- For example, when `random_hw_adapt` is enabled, with `video_sample_n_frames=49`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49`.
- `training_with_video_token_length` specifies training the model according to token length. For training images and videos, the height and width will be set to `image_sample_size` as the maximum and `video_sample_size` as the minimum.
- For example, when `training_with_video_token_length` is enabled, with `video_sample_n_frames=49`, `token_sample_size=1024`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49` to `1024x1024x49`.
- For example, when `training_with_video_token_length` is enabled, with `video_sample_n_frames=49`, `token_sample_size=512`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49` to `1024x1024x9`.
- The token length for a video with dimensions 512x512 and 49 frames is 13,312. We need to set the `token_sample_size = 512`.
- At 512x512 resolution, the number of video frames is 49 (~= 512 * 512 * 49 / 512 / 512).
- At 768x768 resolution, the number of video frames is 21 (~= 512 * 512 * 49 / 768 / 768).
- At 1024x1024 resolution, the number of video frames is 9 (~= 512 * 512 * 49 / 1024 / 1024).
- These resolutions combined with their corresponding lengths allow the model to generate videos of different sizes.
- `train_mode` is used to specify the training mode, which can be either normal or inpaint. Since Wan uses the inpaint model to achieve image-to-video generation, the default is set to inpaint mode. If you only wish to achieve text-to-video generation, you can remove this line, and it will default to the text-to-video mode.
- `resume_from_checkpoint` is used to set the training should be resumed from a previous checkpoint. Use a path or `"latest"` to automatically select the last available checkpoint.
- `boundary_type`: The Wan2.2 series includes two distinct models that handle different noise levels, specified via the `boundary_type` parameter. `low`: Corresponds to the **low noise model** (low_noise_model). `high`: Corresponds to the **high noise model**. (high_noise_model). `full`: Corresponds to the ti2v 5B model (single mode).
Wan T2V without deepspeed:
Training 14B Wan2.2 without DeepSpeed may result in insufficient GPU memory.
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" scripts/wan2.2_fun/train.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--low_vram \
--train_mode="normal" \
--trainable_modules "."
```
Wan T2V with deepspeed zero-2:
Wan with DeepSpeed Zero-2 is suitable for training 14B Wan at low resolutions, but training 14B Wan at high resolutions may still result in insufficient GPU memory.
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_config.json --deepspeed_multinode_launcher standard scripts/wan2.2_fun/train.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--low_vram \
--use_deepspeed \
--train_mode="inpaint" \
--trainable_modules "."
```
Wan T2V with deepspeed zero-3:
Wan with DeepSpeed Zero-3 is suitable for 14B Wan at high resolutions. After training, you can use the following command to get the final model:
```sh
python scripts/zero_to_bf16.py output_dir/checkpoint-{our-num-steps} output_dir/checkpoint-{your-num-steps}-outputs --max_shard_size 80GB --safe_serialization
```
Training shell command is as follows:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag true --use_deepspeed --deepspeed_config_file config/zero_stage3_config.json --deepspeed_multinode_launcher standard scripts/wan2.2_fun/train.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--low_vram \
--use_deepspeed \
--train_mode="inpaint" \
--trainable_modules "."
```
Wan T2V with FSDP:
Wan with FSDP is suitable for 14B Wan at high resolutions. Training shell command is as follows:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP --fsdp_transformer_layer_cls_to_wrap=WanAttentionBlock --fsdp_sharding_strategy "FULL_SHARD" --fsdp_state_dict_type=SHARDED_STATE_DICT --fsdp_backward_prefetch "BACKWARD_PRE" --fsdp_cpu_ram_efficient_loading False scripts/wan2.2_fun/train.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--low_vram \
--use_deepspeed \
--train_mode="inpaint" \
--trainable_modules "."
```
scripts/wan2.2_fun/README_TRAIN_CONTROL.md
Executable
+272
View File
@@ -0,0 +1,272 @@
## Training Code
We can choose whether to use deep speed in Wan-Fun, which can save a lot of video memory.
The metadata_control.json is a little different from normal json in Wan-Fun, you need to add a control_file_path, and [DWPose](https://github.com/IDEA-Research/DWPose) is suggested as tool to generate control file.
```json
[
{
"file_path": "train/00000001.mp4",
"control_file_path": "control/00000001.mp4",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"type": "video"
},
{
"file_path": "train/00000002.jpg",
"control_file_path": "control/00000002.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"type": "image"
},
.....
]
```
Some parameters in the sh file can be confusing, and they are explained in this document:
- `enable_bucket` is used to enable bucket training. When enabled, the model does not crop the images and videos at the center, but instead, it trains the entire images and videos after grouping them into buckets based on resolution.
- `random_frame_crop` is used for random cropping on video frames to simulate videos with different frame counts.
- `random_hw_adapt` is used to enable automatic height and width scaling for images and videos. When `random_hw_adapt` is enabled, the training images will have their height and width set to `image_sample_size` as the maximum and `min(video_sample_size, 512)` as the minimum. For training videos, the height and width will be set to `image_sample_size` as the maximum and `min(video_sample_size, 512)` as the minimum.
- For example, when `random_hw_adapt` is enabled, with `video_sample_n_frames=49`, `video_sample_size=1024`, and `image_sample_size=1024`, the resolution of image inputs for training is `512x512` to `1024x1024`, and the resolution of video inputs for training is `512x512x49` to `1024x1024x49`.
- For example, when `random_hw_adapt` is enabled, with `video_sample_n_frames=49`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49`.
- `training_with_video_token_length` specifies training the model according to token length. For training images and videos, the height and width will be set to `image_sample_size` as the maximum and `video_sample_size` as the minimum.
- For example, when `training_with_video_token_length` is enabled, with `video_sample_n_frames=49`, `token_sample_size=1024`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49` to `1024x1024x49`.
- For example, when `training_with_video_token_length` is enabled, with `video_sample_n_frames=49`, `token_sample_size=512`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49` to `1024x1024x9`.
- The token length for a video with dimensions 512x512 and 49 frames is 13,312. We need to set the `token_sample_size = 512`.
- At 512x512 resolution, the number of video frames is 49 (~= 512 * 512 * 49 / 512 / 512).
- At 768x768 resolution, the number of video frames is 21 (~= 512 * 512 * 49 / 768 / 768).
- At 1024x1024 resolution, the number of video frames is 9 (~= 512 * 512 * 49 / 1024 / 1024).
- These resolutions combined with their corresponding lengths allow the model to generate videos of different sizes.
- `resume_from_checkpoint` is used to set the training should be resumed from a previous checkpoint. Use a path or `"latest"` to automatically select the last available checkpoint.
- `train_mode` is used to set the training mode.
- The models named `Wan2.1-Fun-*-Control` are trained in the `control_ref` mode.
- The models named `Wan2.1-Fun-*-Control-Camera` are trained in the `control_ref_camera` mode.
- `control_ref_image` is used to specify the type of control image. The available options are `first_frame` and `random`.
- `first_frame` is used in V1.0 because V1.0 supports using a specified start frame as the control image. The Control-Camera models use the first frame as the control image.
- `random` is used in V1.1 because V1.1 supports both using a specified start frame and a reference image as the control image.
- `add_full_ref_image_in_self_attention` determines whether to include the reference image in self-attention. This option is used in V1.1, as it supports using a reference image as the control image. It should not be used in V1.0 and Control-Camera models.
- `add_inpaint_info` determines whether to incorporate inpaint information into the model training. When enabled, this allows the model to support specifying starting and ending images in the controls during generation.
- `boundary_type`: The Wan2.2 series includes two distinct models that handle different noise levels, specified via the `boundary_type` parameter. `low`: Corresponds to the **low noise model** (low_noise_model). `high`: Corresponds to the **high noise model**. (high_noise_model). `full`: Corresponds to the ti2v 5B model (single mode).
When train model with multi machines, please set the params as follows:
```sh
export MASTER_ADDR="your master address"
export MASTER_PORT=10086
export WORLD_SIZE=1 # The number of machines
export NUM_PROCESS=8 # The number of processes, such as WORLD_SIZE * 8
export RANK=0 # The rank of this machine
accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main_process_port=$MASTER_PORT --num_machines=$WORLD_SIZE --num_processes=$NUM_PROCESS --machine_rank=$RANK scripts/wan2.2_fun/xxx.py
```
Wan-Fun-Control without deepspeed:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" scripts/wan2.2_fun/train_control.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--low_vram \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--trainable_modules "."
```
Wan-Fun-Control with deepspeed:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_config.json --deepspeed_multinode_launcher standard scripts/wan2.2_fun/train_control.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--low_vram \
--use_deepspeed \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--trainable_modules "."
```
Wan-Fun-Control with deepspeed zero-3:
Wan with DeepSpeed Zero-3 is suitable for 14B Wan at high resolutions. After training, you can use the following command to get the final model:
```sh
python scripts/zero_to_bf16.py output_dir/checkpoint-{our-num-steps} output_dir/checkpoint-{your-num-steps}-outputs --max_shard_size 80GB --safe_serialization
```
Training shell command is as follows:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag true --use_deepspeed --deepspeed_config_file config/zero_stage3_config.json --deepspeed_multinode_launcher standard scripts/wan2.2_fun/train_control.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--low_vram \
--use_deepspeed \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--trainable_modules "."
```
Wan-Fun-Control with FSDP:
Wan with FSDP is suitable for 14B Wan at high resolutions. Training shell command is as follows:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP --fsdp_transformer_layer_cls_to_wrap=WanAttentionBlock --fsdp_sharding_strategy "FULL_SHARD" --fsdp_state_dict_type=SHARDED_STATE_DICT --fsdp_backward_prefetch "BACKWARD_PRE" --fsdp_cpu_ram_efficient_loading False scripts/wan2.2_fun/train_control.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--low_vram \
--use_deepspeed \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--trainable_modules "."
```
scripts/wan2.2_fun/README_TRAIN_CONTROL_LORA.md
Executable
+262
View File
@@ -0,0 +1,262 @@
## Training Code
We can choose whether to use deep speed in Wan-Fun, which can save a lot of video memory.
The metadata_control.json is a little different from normal json in Wan-Fun, you need to add a control_file_path, and [DWPose](https://github.com/IDEA-Research/DWPose) is suggested as tool to generate control file.
```json
[
{
"file_path": "train/00000001.mp4",
"control_file_path": "control/00000001.mp4",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"type": "video"
},
{
"file_path": "train/00000002.jpg",
"control_file_path": "control/00000002.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"type": "image"
},
.....
]
```
Some parameters in the sh file can be confusing, and they are explained in this document:
- `enable_bucket` is used to enable bucket training. When enabled, the model does not crop the images and videos at the center, but instead, it trains the entire images and videos after grouping them into buckets based on resolution.
- `random_frame_crop` is used for random cropping on video frames to simulate videos with different frame counts.
- `random_hw_adapt` is used to enable automatic height and width scaling for images and videos. When `random_hw_adapt` is enabled, the training images will have their height and width set to `image_sample_size` as the maximum and `min(video_sample_size, 512)` as the minimum. For training videos, the height and width will be set to `image_sample_size` as the maximum and `min(video_sample_size, 512)` as the minimum.
- For example, when `random_hw_adapt` is enabled, with `video_sample_n_frames=49`, `video_sample_size=1024`, and `image_sample_size=1024`, the resolution of image inputs for training is `512x512` to `1024x1024`, and the resolution of video inputs for training is `512x512x49` to `1024x1024x49`.
- For example, when `random_hw_adapt` is enabled, with `video_sample_n_frames=49`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49`.
- `training_with_video_token_length` specifies training the model according to token length. For training images and videos, the height and width will be set to `image_sample_size` as the maximum and `video_sample_size` as the minimum.
- For example, when `training_with_video_token_length` is enabled, with `video_sample_n_frames=49`, `token_sample_size=1024`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49` to `1024x1024x49`.
- For example, when `training_with_video_token_length` is enabled, with `video_sample_n_frames=49`, `token_sample_size=512`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49` to `1024x1024x9`.
- The token length for a video with dimensions 512x512 and 49 frames is 13,312. We need to set the `token_sample_size = 512`.
- At 512x512 resolution, the number of video frames is 49 (~= 512 * 512 * 49 / 512 / 512).
- At 768x768 resolution, the number of video frames is 21 (~= 512 * 512 * 49 / 768 / 768).
- At 1024x1024 resolution, the number of video frames is 9 (~= 512 * 512 * 49 / 1024 / 1024).
- These resolutions combined with their corresponding lengths allow the model to generate videos of different sizes.
- `resume_from_checkpoint` is used to set the training should be resumed from a previous checkpoint. Use a path or `"latest"` to automatically select the last available checkpoint and set the `save_state` to `True`.
- `train_mode` is used to set the training mode.
- The models named `Wan2.1-Fun-*-Control` are trained in the `control_ref` mode.
- The models named `Wan2.1-Fun-*-Control-Camera` are trained in the `control_ref_camera` mode.
- `control_ref_image` is used to specify the type of control image. The available options are `first_frame` and `random`.
- `first_frame` is used in V1.0 because V1.0 supports using a specified start frame as the control image. The Control-Camera models use the first frame as the control image.
- `random` is used in V1.1 because V1.1 supports both using a specified start frame and a reference image as the control image.
- `add_full_ref_image_in_self_attention` determines whether to include the reference image in self-attention. This option is used in V1.1, as it supports using a reference image as the control image. It should not be used in V1.0 and Control-Camera models.
- `add_inpaint_info` determines whether to incorporate inpaint information into the model training. When enabled, this allows the model to support specifying starting and ending images in the controls during generation.
- `boundary_type`: The Wan2.2 series includes two distinct models that handle different noise levels, specified via the `boundary_type` parameter. `low`: Corresponds to the **low noise model** (low_noise_model). `high`: Corresponds to the **high noise model**. (high_noise_model). `full`: Corresponds to the ti2v 5B model (single mode).
When train model with multi machines, please set the params as follows:
```sh
export MASTER_ADDR="your master address"
export MASTER_PORT=10086
export WORLD_SIZE=1 # The number of machines
export NUM_PROCESS=8 # The number of processes, such as WORLD_SIZE * 8
export RANK=0 # The rank of this machine
accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main_process_port=$MASTER_PORT --num_machines=$WORLD_SIZE --num_processes=$NUM_PROCESS --machine_rank=$RANK scripts/wan2.2_fun/xxx.py
```
Wan-Fun-Control without deepspeed:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" scripts/wan2.2_fun/train_control_lora.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-04 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--low_vram
```
Wan-Fun-Control with deepspeed:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_config.json --deepspeed_multinode_launcher standard scripts/wan2.2_fun/train_control_lora.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-04 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--use_deepspeed \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--low_vram
```
Wan-Fun-Control with deepspeed zero-3:
Wan with DeepSpeed Zero-3 is suitable for 14B Wan at high resolutions. You must set save_state to True to save the model. After training, you can use the following command to get the final model:
```sh
python scripts/zero_to_bf16.py output_dir/checkpoint-{our-num-steps} output_dir/checkpoint-{your-num-steps}-outputs --max_shard_size 80GB --safe_serialization
```
Training shell command is as follows:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag true --use_deepspeed --deepspeed_config_file config/zero_stage3_config.json --deepspeed_multinode_launcher standard scripts/wan2.2_fun/train_control_lora.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-04 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--save_state \
--use_deepspeed \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--low_vram
```
Wan-Fun-Control with FSDP:
Wan with FSDP is suitable for 14B Wan at high resolutions. Training shell command is as follows:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP --fsdp_transformer_layer_cls_to_wrap=WanAttentionBlock --fsdp_sharding_strategy "FULL_SHARD" --fsdp_state_dict_type=SHARDED_STATE_DICT --fsdp_backward_prefetch "BACKWARD_PRE" --fsdp_cpu_ram_efficient_loading False scripts/wan2.2_fun/train_control_lora.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-04 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--save_state \
--use_fsdp \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--low_vram
```
scripts/wan2.2_fun/README_TRAIN_LORA.md
Executable
+217
View File
@@ -0,0 +1,217 @@
## Lora Training Code
We can choose whether to use deep speed in Wan, which can save a lot of video memory.
Some parameters in the sh file can be confusing, and they are explained in this document:
- `enable_bucket` is used to enable bucket training. When enabled, the model does not crop the images and videos at the center, but instead, it trains the entire images and videos after grouping them into buckets based on resolution.
- `random_frame_crop` is used for random cropping on video frames to simulate videos with different frame counts.
- `random_hw_adapt` is used to enable automatic height and width scaling for images and videos. When `random_hw_adapt` is enabled, the training images will have their height and width set to `image_sample_size` as the maximum and `min(video_sample_size, 512)` as the minimum. For training videos, the height and width will be set to `image_sample_size` as the maximum and `min(video_sample_size, 512)` as the minimum.
- For example, when `random_hw_adapt` is enabled, with `video_sample_n_frames=49`, `video_sample_size=1024`, and `image_sample_size=1024`, the resolution of image inputs for training is `512x512` to `1024x1024`, and the resolution of video inputs for training is `512x512x49` to `1024x1024x49`.
- For example, when `random_hw_adapt` is enabled, with `video_sample_n_frames=49`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49`.
- `training_with_video_token_length` specifies training the model according to token length. For training images and videos, the height and width will be set to `image_sample_size` as the maximum and `video_sample_size` as the minimum.
- For example, when `training_with_video_token_length` is enabled, with `video_sample_n_frames=49`, `token_sample_size=1024`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49` to `1024x1024x49`.
- For example, when `training_with_video_token_length` is enabled, with `video_sample_n_frames=49`, `token_sample_size=512`, `video_sample_size=1024`, and `image_sample_size=256`, the resolution of image inputs for training is `256x256` to `1024x1024`, and the resolution of video inputs for training is `256x256x49` to `1024x1024x9`.
- The token length for a video with dimensions 512x512 and 49 frames is 13,312. We need to set the `token_sample_size = 512`.
- At 512x512 resolution, the number of video frames is 49 (~= 512 * 512 * 49 / 512 / 512).
- At 768x768 resolution, the number of video frames is 21 (~= 512 * 512 * 49 / 768 / 768).
- At 1024x1024 resolution, the number of video frames is 9 (~= 512 * 512 * 49 / 1024 / 1024).
- These resolutions combined with their corresponding lengths allow the model to generate videos of different sizes.
- `train_mode` is used to specify the training mode, which can be either normal or inpaint. Since Wan uses the inpaint model to achieve image-to-video generation, the default is set to inpaint mode. If you only wish to achieve text-to-video generation, you can remove this line, and it will default to the text-to-video mode.
- `resume_from_checkpoint` is used to set the training should be resumed from a previous checkpoint. Use a path or `"latest"` to automatically select the last available checkpoint.
- `boundary_type`: The Wan2.2 series includes two distinct models that handle different noise levels, specified via the `boundary_type` parameter. `low`: Corresponds to the **low noise model** (low_noise_model). `high`: Corresponds to the **high noise model**. (high_noise_model). `full`: Corresponds to the ti2v 5B model (single mode).
Wan T2V without deepspeed:
Training 14B Wan2.2 without DeepSpeed may result in insufficient GPU memory.
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" scripts/wan2.2_fun/train_lora.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-04 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--train_mode="inpaint" \
--low_vram
```
Wan T2V with deepspeed zero-2:
Wan with DeepSpeed Zero-2 is suitable for training 14B Wan at low resolutions, but training 14B Wan at high resolutions may still result in insufficient GPU memory.
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_config.json --deepspeed_multinode_launcher standard scripts/wan2.2_fun/train_lora.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-04 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--use_deepspeed \
--train_mode="inpaint" \
--low_vram
```
Wan T2V with deepspeed zero-3:
Wan with DeepSpeed Zero-3 is suitable for 14B Wan at high resolutions. You must set save_state to True to save the model. After training, you can use the following command to get the final model:
```sh
python scripts/zero_to_bf16.py output_dir/checkpoint-{our-num-steps} output_dir/checkpoint-{your-num-steps}-outputs --max_shard_size 80GB --safe_serialization
```
Training shell command is as follows:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag true --use_deepspeed --deepspeed_config_file config/zero_stage3_config.json --deepspeed_multinode_launcher standard scripts/wan2.2_fun/train_lora.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-04 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--save_state \
--use_deepspeed \
--train_mode="inpaint" \
--low_vram
```
Wan T2V with FSDP:
Wan with FSDP is suitable for 14B Wan at high resolutions. Training shell command is as follows:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP --fsdp_transformer_layer_cls_to_wrap=WanAttentionBlock --fsdp_sharding_strategy "FULL_SHARD" --fsdp_state_dict_type=SHARDED_STATE_DICT --fsdp_backward_prefetch "BACKWARD_PRE" --fsdp_cpu_ram_efficient_loading False scripts/wan2.2_fun/train_lora.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-04 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--boundary_type="low" \
--save_state \
--use_deepspeed \
--train_mode="inpaint" \
--low_vram
```
scripts/wan2.2_fun/train.py
+1916
View File
File diff suppressed because it is too large Load Diff
scripts/wan2.2_fun/train.sh
+43
View File
@@ -0,0 +1,43 @@
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-InP"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" scripts/wan2.2_fun/train.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--low_vram \
--boundary_type="low" \
--train_mode="inpaint" \
--trainable_modules "."
scripts/wan2.2_fun/train_control.py
+2069
View File
File diff suppressed because it is too large Load Diff
scripts/wan2.2_fun/train_control.sh
+45
View File
@@ -0,0 +1,45 @@
export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-Fun-A14B-Control"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" scripts/wan2.2_fun/train_control.py \
--config_path="config/wan2.2/wan_civitai_i2v.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--image_sample_size=1024 \
--video_sample_size=256 \
--token_sample_size=512 \
--video_sample_stride=2 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--video_repeat=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--train_mode="control_ref" \
--control_ref_image="random" \
--add_inpaint_info \
--add_full_ref_image_in_self_attention \
--low_vram \
--trainable_modules "."
scripts/wan2.2_fun/train_lora.sh
+1
View File
@@ -38,4 +38,5 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.2_fun/train_lora.py \
--train_mode="inpaint" \
--boundary_type="low" \
--lora_skip_name="ffn" \
--boundary_type="low" \
--low_vram
videox_fun/pipeline/pipeline_wan2_2_fun_control.py
+1 -1
View File
@@ -158,7 +158,7 @@ class Wan2_2FunControlPipeline(DiffusionPipeline):
"""
_optional_components = ["transformer_2"]
model_cpu_offload_seq = "text_encoder->transformer->transformer_2->vae"
model_cpu_offload_seq = "text_encoder->transformer_2->transformer->vae"
_callback_tensor_inputs = [
"latents",
videox_fun/pipeline/pipeline_wan2_2_ti2v.py
+3 -1
View File
@@ -601,7 +601,7 @@ class Wan2_2TI2VPipeline(DiffusionPipeline):
pbar.update(1)
# Prepare mask latent variables
if init_video is not None:
if init_video is not None and not (mask_video == 255).all():
bs, _, video_length, height, width = video.size()
mask_condition = self.mask_processor.preprocess(rearrange(mask_video, "b c f h w -> (b f) c h w"), height=height, width=width)
mask_condition = mask_condition.to(dtype=torch.float32)
@@ -632,6 +632,8 @@ class Wan2_2TI2VPipeline(DiffusionPipeline):
mask = F.interpolate(mask_condition[:, :1], size=latents.size()[-3:], mode='trilinear', align_corners=True).to(device, weight_dtype)
latents = (1 - mask) * masked_video_latents + mask * latents
else:
init_video = None
if comfyui_progressbar:
pbar.update(1)
videox_fun/ui/controller.py
+2 -1
View File
@@ -256,6 +256,7 @@ class Fun_Controller:
validation_video,
control_video,
):
spatial_compression_ratio = self.vae.config.spatial_compression_ratio if hasattr(self.vae.config, "spatial_compression_ratio") else 8
aspect_ratio_sample_size = {key : [x / 512 * base_resolution for x in ASPECT_RATIO_512[key]] for key in ASPECT_RATIO_512.keys()}
if self.model_type == "Inpaint":
if validation_video is not None:
@@ -265,7 +266,7 @@ class Fun_Controller:
else:
original_width, original_height = Image.fromarray(cv2.VideoCapture(control_video).read()[1]).size
closest_size, closest_ratio = get_closest_ratio(original_height, original_width, ratios=aspect_ratio_sample_size)
height_slider, width_slider = [int(x / 16) * 16 for x in closest_size]
height_slider, width_slider = [int(x / spatial_compression_ratio / 2) * spatial_compression_ratio * 2 for x in closest_size]
return height_slider, width_slider
def save_outputs(self, is_image, length_slider, sample, fps):
videox_fun/ui/wan2_2_fun_ui.py
+803
View File
@@ -0,0 +1,803 @@
"""Modified from https://github.com/guoyww/AnimateDiff/blob/main/app.py
"""
import os
import random
import cv2
import gradio as gr
import numpy as np
import torch
from omegaconf import OmegaConf
from PIL import Image
from safetensors import safe_open
from ..data.bucket_sampler import ASPECT_RATIO_512, get_closest_ratio
from ..dist import set_multi_gpus_devices, shard_model
from ..models import (AutoencoderKLWan, AutoencoderKLWan3_8, AutoTokenizer,
CLIPModel, Wan2_2Transformer3DModel, WanT5EncoderModel)
from ..models.cache_utils import get_teacache_coefficients
from ..pipeline import Wan2_2FunControlPipeline, Wan2_2FunPipeline, Wan2_2FunInpaintPipeline
from ..utils.fp8_optimization import (convert_model_weight_to_float8,
convert_weight_dtype_wrapper,
replace_parameters_by_name)
from ..utils.lora_utils import merge_lora, unmerge_lora
from ..utils.utils import (filter_kwargs, get_image_latent,
get_image_to_video_latent,
get_video_to_video_latent, save_videos_grid, timer)
from .controller import (Fun_Controller, Fun_Controller_Client,
all_cheduler_dict, css, ddpm_scheduler_dict,
flow_scheduler_dict, gradio_version,
gradio_version_is_above_4)
from .ui import (create_cfg_and_seedbox, create_cfg_riflex_k,
create_cfg_skip_params, create_config,
create_fake_finetune_models_checkpoints,
create_fake_height_width, create_fake_model_checkpoints,
create_fake_model_type, create_finetune_models_checkpoints,
create_generation_method,
create_generation_methods_and_video_length,
create_height_width, create_model_checkpoints,
create_model_type, create_prompts, create_samplers,
create_teacache_params, create_ui_outputs)
class Wan2_2_Fun_Controller(Fun_Controller):
def update_diffusion_transformer(self, diffusion_transformer_dropdown):
print(f"Update diffusion transformer: {diffusion_transformer_dropdown}")
self.model_name = diffusion_transformer_dropdown
self.diffusion_transformer_dropdown = diffusion_transformer_dropdown
if diffusion_transformer_dropdown == "none":
return gr.update()
Choosen_AutoencoderKL = {
"AutoencoderKLWan": AutoencoderKLWan,
"AutoencoderKLWan3_8": AutoencoderKLWan3_8
}[self.config['vae_kwargs'].get('vae_type', 'AutoencoderKLWan')]
self.vae = Choosen_AutoencoderKL.from_pretrained(
os.path.join(diffusion_transformer_dropdown, self.config['vae_kwargs'].get('vae_subpath', 'vae')),
additional_kwargs=OmegaConf.to_container(self.config['vae_kwargs']),
).to(self.weight_dtype)
# Get Transformer
self.transformer = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(diffusion_transformer_dropdown, self.config['transformer_additional_kwargs'].get('transformer_low_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(self.config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=self.weight_dtype,
)
if self.config['transformer_additional_kwargs'].get('transformer_combination_type', 'single') == "moe":
self.transformer_2 = Wan2_2Transformer3DModel.from_pretrained(
os.path.join(diffusion_transformer_dropdown, self.config['transformer_additional_kwargs'].get('transformer_high_noise_model_subpath', 'transformer')),
transformer_additional_kwargs=OmegaConf.to_container(self.config['transformer_additional_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=self.weight_dtype,
)
else:
self.transformer_2 = None
# Get Tokenizer
self.tokenizer = AutoTokenizer.from_pretrained(
os.path.join(diffusion_transformer_dropdown, self.config['text_encoder_kwargs'].get('tokenizer_subpath', 'tokenizer')),
)
# Get Text encoder
self.text_encoder = WanT5EncoderModel.from_pretrained(
os.path.join(diffusion_transformer_dropdown, self.config['text_encoder_kwargs'].get('text_encoder_subpath', 'text_encoder')),
additional_kwargs=OmegaConf.to_container(self.config['text_encoder_kwargs']),
low_cpu_mem_usage=True,
torch_dtype=self.weight_dtype,
)
self.text_encoder = self.text_encoder.eval()
Choosen_Scheduler = self.scheduler_dict[list(self.scheduler_dict.keys())[0]]
self.scheduler = Choosen_Scheduler(
**filter_kwargs(Choosen_Scheduler, OmegaConf.to_container(self.config['scheduler_kwargs']))
)
# Get pipeline
if self.model_type == "Inpaint":
if self.transformer.config.in_channels != self.vae.config.latent_channels:
self.pipeline = Wan2_2FunInpaintPipeline(
vae=self.vae,
tokenizer=self.tokenizer,
text_encoder=self.text_encoder,
transformer=self.transformer,
transformer_2=self.transformer_2,
scheduler=self.scheduler,
)
else:
self.pipeline = Wan2_2FunPipeline(
vae=self.vae,
tokenizer=self.tokenizer,
text_encoder=self.text_encoder,
transformer=self.transformer,
transformer_2=self.transformer_2,
scheduler=self.scheduler,
)
else:
self.pipeline = Wan2_2FunControlPipeline(
vae=self.vae,
tokenizer=self.tokenizer,
text_encoder=self.text_encoder,
transformer=self.transformer,
transformer_2=self.transformer_2,
scheduler=self.scheduler,
)
if self.ulysses_degree > 1 or self.ring_degree > 1:
from functools import partial
self.transformer.enable_multi_gpus_inference()
if self.transformer_2 is not None:
self.transformer_2.enable_multi_gpus_inference()
if self.fsdp_dit:
shard_fn = partial(shard_model, device_id=self.device, param_dtype=self.weight_dtype)
self.pipeline.transformer = shard_fn(self.pipeline.transformer)
if self.transformer_2 is not None:
self.pipeline.transformer_2 = shard_fn(self.pipeline.transformer_2)
print("Add FSDP DIT")
if self.fsdp_text_encoder:
shard_fn = partial(shard_model, device_id=self.device, param_dtype=self.weight_dtype)
self.pipeline.text_encoder = shard_fn(self.pipeline.text_encoder)
print("Add FSDP TEXT ENCODER")
if self.compile_dit:
for i in range(len(self.pipeline.transformer.blocks)):
self.pipeline.transformer.blocks[i] = torch.compile(self.pipeline.transformer.blocks[i])
if self.transformer_2 is not None:
for i in range(len(self.pipeline.transformer_2.blocks)):
self.pipeline.transformer_2.blocks[i] = torch.compile(self.pipeline.transformer_2.blocks[i])
print("Add Compile")
if self.GPU_memory_mode == "sequential_cpu_offload":
replace_parameters_by_name(self.transformer, ["modulation",], device=self.device)
self.transformer.freqs = self.transformer.freqs.to(device=self.device)
if self.transformer_2 is not None:
replace_parameters_by_name(self.transformer_2, ["modulation",], device=self.device)
self.transformer_2.freqs = self.transformer_2.freqs.to(device=self.device)
self.pipeline.enable_sequential_cpu_offload(device=self.device)
elif self.GPU_memory_mode == "model_cpu_offload_and_qfloat8":
convert_model_weight_to_float8(self.transformer, exclude_module_name=["modulation",], device=self.device)
convert_weight_dtype_wrapper(self.transformer, self.weight_dtype)
if self.transformer_2 is not None:
convert_model_weight_to_float8(self.transformer_2, exclude_module_name=["modulation",], device=self.device)
convert_weight_dtype_wrapper(self.transformer_2, self.weight_dtype)
self.pipeline.enable_model_cpu_offload(device=self.device)
elif self.GPU_memory_mode == "model_cpu_offload":
self.pipeline.enable_model_cpu_offload(device=self.device)
elif self.GPU_memory_mode == "model_full_load_and_qfloat8":
convert_model_weight_to_float8(self.transformer, exclude_module_name=["modulation",], device=self.device)
convert_weight_dtype_wrapper(self.transformer, self.weight_dtype)
if self.transformer_2 is not None:
convert_model_weight_to_float8(self.transformer_2, exclude_module_name=["modulation",], device=self.device)
convert_weight_dtype_wrapper(self.transformer_2, self.weight_dtype)
self.pipeline.to(self.device)
else:
self.pipeline.to(self.device)
print("Update diffusion transformer done")
return gr.update()
@timer
def generate(
self,
diffusion_transformer_dropdown,
base_model_dropdown,
lora_model_dropdown,
lora_alpha_slider,
prompt_textbox,
negative_prompt_textbox,
sampler_dropdown,
sample_step_slider,
resize_method,
width_slider,
height_slider,
base_resolution,
generation_method,
length_slider,
overlap_video_length,
partial_video_length,
cfg_scale_slider,
start_image,
end_image,
validation_video,
validation_video_mask,
control_video,
denoise_strength,
seed_textbox,
ref_image = None,
enable_teacache = None,
teacache_threshold = None,
num_skip_start_steps = None,
teacache_offload = None,
cfg_skip_ratio = None,
enable_riflex = None,
riflex_k = None,
base_model_2_dropdown=None,
lora_model_2_dropdown=None,
fps = None,
is_api = False,
):
self.clear_cache()
print(f"Input checking.")
_, comment = self.input_check(
resize_method, generation_method, start_image, end_image, validation_video,control_video, is_api
)
print(f"Input checking down")
if comment != "OK":
return "", comment
is_image = True if generation_method == "Image Generation" else False
if self.base_model_path != base_model_dropdown:
self.update_base_model(base_model_dropdown)
if self.base_model_2_path != base_model_2_dropdown:
self.update_lora_model(base_model_2_dropdown, is_checkpoint_2=True)
if self.lora_model_path != lora_model_dropdown:
self.update_lora_model(lora_model_dropdown)
if self.lora_model_2_path != lora_model_2_dropdown:
self.update_lora_model(lora_model_2_dropdown, is_checkpoint_2=True)
print(f"Load scheduler.")
scheduler_config = self.pipeline.scheduler.config
if sampler_dropdown == "Flow_Unipc" or sampler_dropdown == "Flow_DPM++":
scheduler_config['shift'] = 1
self.pipeline.scheduler = self.scheduler_dict[sampler_dropdown].from_config(scheduler_config)
print(f"Load scheduler down.")
if resize_method == "Resize according to Reference":
print(f"Calculate height and width according to Reference.")
height_slider, width_slider = self.get_height_width_from_reference(
base_resolution, start_image, validation_video, control_video,
)
if self.lora_model_path != "none":
print(f"Merge Lora.")
self.pipeline = merge_lora(self.pipeline, self.lora_model_path, multiplier=lora_alpha_slider)
if self.transformer_2 is not None:
self.pipeline = merge_lora(self.pipeline, self.lora_model_2_path, multiplier=lora_alpha_slider, sub_transformer_name="transformer_2")
print(f"Merge Lora done.")
coefficients = get_teacache_coefficients(self.diffusion_transformer_dropdown) if enable_teacache else None
if coefficients is not None:
print(f"Enable TeaCache with threshold {teacache_threshold} and skip the first {num_skip_start_steps} steps.")
self.pipeline.transformer.enable_teacache(
coefficients, sample_step_slider, teacache_threshold, num_skip_start_steps=num_skip_start_steps, offload=teacache_offload
)
if self.transformer_2 is not None:
self.pipeline.transformer_2.share_teacache(self.pipeline.transformer)
else:
print(f"Disable TeaCache.")
self.pipeline.transformer.disable_teacache()
if self.transformer_2 is not None:
self.pipeline.transformer_2.disable_teacache()
if cfg_skip_ratio is not None and cfg_skip_ratio >= 0:
print(f"Enable cfg_skip_ratio {cfg_skip_ratio}.")
self.pipeline.transformer.enable_cfg_skip(cfg_skip_ratio, sample_step_slider)
if self.transformer_2 is not None:
self.pipeline.transformer_2.share_cfg_skip(self.pipeline.transformer)
print(f"Generate seed.")
if int(seed_textbox) != -1 and seed_textbox != "": torch.manual_seed(int(seed_textbox))
else: seed_textbox = np.random.randint(0, 1e10)
generator = torch.Generator(device=self.device).manual_seed(int(seed_textbox))
print(f"Generate seed done.")
if fps is None:
fps = 16
boundary = self.config['transformer_additional_kwargs'].get('boundary', 0.875)
if enable_riflex:
print(f"Enable riflex")
latent_frames = (int(length_slider) - 1) // self.vae.config.temporal_compression_ratio + 1
self.pipeline.transformer.enable_riflex(k = riflex_k, L_test = latent_frames if not is_image else 1)
if self.transformer_2 is not None:
self.pipeline.transformer_2.enable_riflex(k = riflex_k, L_test = latent_frames if not is_image else 1)
try:
print(f"Generation.")
if self.model_type == "Inpaint":
if self.transformer.config.in_channels != self.vae.config.latent_channels:
if validation_video is not None:
input_video, input_video_mask, _, clip_image = get_video_to_video_latent(validation_video, length_slider if not is_image else 1, sample_size=(height_slider, width_slider), validation_video_mask=validation_video_mask, fps=fps)
else:
input_video, input_video_mask, clip_image = get_image_to_video_latent(start_image, end_image, length_slider if not is_image else 1, sample_size=(height_slider, width_slider))
sample = self.pipeline(
prompt_textbox,
negative_prompt = negative_prompt_textbox,
num_inference_steps = sample_step_slider,
guidance_scale = cfg_scale_slider,
width = width_slider,
height = height_slider,
num_frames = length_slider if not is_image else 1,
generator = generator,
video = input_video,
mask_video = input_video_mask,
boundary = boundary
).videos
else:
sample = self.pipeline(
prompt_textbox,
negative_prompt = negative_prompt_textbox,
num_inference_steps = sample_step_slider,
guidance_scale = cfg_scale_slider,
width = width_slider,
height = height_slider,
num_frames = length_slider if not is_image else 1,
generator = generator,
boundary = boundary
).videos
else:
inpaint_video, inpaint_video_mask, clip_image = get_image_to_video_latent(start_image, end_image, video_length=length_slider if not is_image else 1, sample_size=(height_slider, width_slider))
if ref_image is not None:
ref_image = get_image_latent(ref_image, sample_size=(height_slider, width_slider))
input_video, input_video_mask, _, _ = get_video_to_video_latent(control_video, video_length=length_slider if not is_image else 1, sample_size=(height_slider, width_slider), fps=fps, ref_image=None)
sample = self.pipeline(
prompt_textbox,
negative_prompt = negative_prompt_textbox,
num_inference_steps = sample_step_slider,
guidance_scale = cfg_scale_slider,
width = width_slider,
height = height_slider,
num_frames = length_slider if not is_image else 1,
generator = generator,
video = inpaint_video,
mask_video = inpaint_video_mask,
control_video = input_video,
ref_image = ref_image,
boundary = boundary,
).videos
print(f"Generation done.")
except Exception as e:
self.auto_model_clear_cache(self.pipeline.transformer)
self.auto_model_clear_cache(self.pipeline.text_encoder)
self.auto_model_clear_cache(self.pipeline.vae)
self.clear_cache()
print(f"Error. error information is {str(e)}")
if self.lora_model_path != "none":
self.pipeline = unmerge_lora(self.pipeline, self.lora_model_path, multiplier=lora_alpha_slider)
if is_api:
return "", f"Error. error information is {str(e)}"
else:
return gr.update(), gr.update(), f"Error. error information is {str(e)}"
self.clear_cache()
# lora part
if self.lora_model_path != "none":
print(f"Unmerge Lora.")
self.pipeline = unmerge_lora(self.pipeline, self.lora_model_path, multiplier=lora_alpha_slider)
print(f"Unmerge Lora done.")
print(f"Saving outputs.")
save_sample_path = self.save_outputs(
is_image, length_slider, sample, fps=fps
)
print(f"Saving outputs done.")
if is_image or length_slider == 1:
if is_api:
return save_sample_path, "Success"
else:
if gradio_version_is_above_4:
return gr.Image(value=save_sample_path, visible=True), gr.Video(value=None, visible=False), "Success"
else:
return gr.Image.update(value=save_sample_path, visible=True), gr.Video.update(value=None, visible=False), "Success"
else:
if is_api:
return save_sample_path, "Success"
else:
if gradio_version_is_above_4:
return gr.Image(visible=False, value=None), gr.Video(value=save_sample_path, visible=True), "Success"
else:
return gr.Image.update(visible=False, value=None), gr.Video.update(value=save_sample_path, visible=True), "Success"
Wan2_2_Fun_Controller_Host = Wan2_2_Fun_Controller
Wan2_2_Fun_Controller_Client = Fun_Controller_Client
def ui(GPU_memory_mode, scheduler_dict, config_path, compile_dit, weight_dtype, savedir_sample=None):
controller = Wan2_2_Fun_Controller(
GPU_memory_mode, scheduler_dict, model_name=None, model_type="Inpaint",
config_path=config_path, compile_dit=compile_dit,
weight_dtype=weight_dtype, savedir_sample=savedir_sample,
)
with gr.Blocks(css=css) as demo:
gr.Markdown(
"""
# Wan2.2-Fun:
A Wan with more flexible generation conditions, capable of producing videos of different resolutions, around 5 seconds, and fps 16 (frames 1 to 81), as well as image generated videos.
[Github](https://github.com/aigc-apps/VideoX-Fun/)
"""
)
with gr.Column(variant="panel"):
config_dropdown, config_refresh_button = create_config(controller)
model_type = create_model_type(visible=True)
diffusion_transformer_dropdown, diffusion_transformer_refresh_button = \
create_model_checkpoints(controller, visible=True)
base_model_dropdown, lora_model_dropdown, lora_alpha_slider, personalized_refresh_button = \
create_finetune_models_checkpoints(controller, visible=True, add_checkpoint_2=True)
base_model_dropdown, base_model_2_dropdown = base_model_dropdown
lora_model_dropdown, lora_model_2_dropdown = lora_model_dropdown
with gr.Row():
enable_teacache, teacache_threshold, num_skip_start_steps, teacache_offload = \
create_teacache_params(True, 0.10, 1, False)
cfg_skip_ratio = create_cfg_skip_params(0)
enable_riflex, riflex_k = create_cfg_riflex_k(False, 6)
with gr.Column(variant="panel"):
prompt_textbox, negative_prompt_textbox = create_prompts(negative_prompt="色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走")
with gr.Row():
with gr.Column():
sampler_dropdown, sample_step_slider = create_samplers(controller)
resize_method, width_slider, height_slider, base_resolution = create_height_width(
default_height = 480, default_width = 832, maximum_height = 1344,
maximum_width = 1344,
)
generation_method, length_slider, overlap_video_length, partial_video_length = \
create_generation_methods_and_video_length(
["Video Generation", "Image Generation"],
default_video_length=81,
maximum_video_length=161,
)
image_to_video_col, video_to_video_col, control_video_col, source_method, start_image, template_gallery, end_image, validation_video, validation_video_mask, denoise_strength, control_video, ref_image = create_generation_method(
["Text to Video (文本到视频)", "Image to Video (图片到视频)", "Video Control (视频控制)"], prompt_textbox, support_ref_image=True
)
cfg_scale_slider, seed_textbox, seed_button = create_cfg_and_seedbox(gradio_version_is_above_4)
generate_button = gr.Button(value="Generate (生成)", variant='primary')
result_image, result_video, infer_progress = create_ui_outputs()
config_dropdown.change(
fn=controller.update_config,
inputs=[config_dropdown],
outputs=[]
)
model_type.change(
fn=controller.update_model_type,
inputs=[model_type],
outputs=[]
)
def upload_generation_method(generation_method):
if generation_method == "Video Generation":
return [gr.update(visible=True, maximum=161, value=81, interactive=True), gr.update(visible=False), gr.update(visible=False)]
elif generation_method == "Image Generation":
return [gr.update(minimum=1, maximum=1, value=1, interactive=False), gr.update(visible=False), gr.update(visible=False)]
else:
return [gr.update(visible=True, maximum=1344), gr.update(visible=True), gr.update(visible=True)]
generation_method.change(
upload_generation_method, generation_method, [length_slider, overlap_video_length, partial_video_length]
)
def upload_source_method(source_method):
if source_method == "Text to Video (文本到视频)":
return [gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update(value=None)]
elif source_method == "Image to Video (图片到视频)":
return [gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(), gr.update(), gr.update(value=None), gr.update(value=None), gr.update(value=None)]
elif source_method == "Video to Video (视频到视频)":
return [gr.update(visible=False), gr.update(visible=True), gr.update(visible=False), gr.update(value=None), gr.update(value=None), gr.update(), gr.update(), gr.update(value=None)]
else:
return [gr.update(visible=False), gr.update(visible=False), gr.update(visible=True), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update()]
source_method.change(
upload_source_method, source_method, [
image_to_video_col, video_to_video_col, control_video_col, start_image, end_image,
validation_video, validation_video_mask, control_video
]
)
def upload_resize_method(resize_method):
if resize_method == "Generate by":
return [gr.update(visible=True), gr.update(visible=True), gr.update(visible=False)]
else:
return [gr.update(visible=False), gr.update(visible=False), gr.update(visible=True)]
resize_method.change(
upload_resize_method, resize_method, [width_slider, height_slider, base_resolution]
)
generate_button.click(
fn=controller.generate,
inputs=[
diffusion_transformer_dropdown,
base_model_dropdown,
lora_model_dropdown,
lora_alpha_slider,
prompt_textbox,
negative_prompt_textbox,
sampler_dropdown,
sample_step_slider,
resize_method,
width_slider,
height_slider,
base_resolution,
generation_method,
length_slider,
overlap_video_length,
partial_video_length,
cfg_scale_slider,
start_image,
end_image,
validation_video,
validation_video_mask,
control_video,
denoise_strength,
seed_textbox,
ref_image,
enable_teacache,
teacache_threshold,
num_skip_start_steps,
teacache_offload,
cfg_skip_ratio,
enable_riflex,
riflex_k,
base_model_2_dropdown,
lora_model_2_dropdown
],
outputs=[result_image, result_video, infer_progress]
)
return demo, controller
def ui_host(GPU_memory_mode, scheduler_dict, model_name, model_type, config_path, compile_dit, weight_dtype, savedir_sample=None):
controller = Wan2_2_Fun_Controller_Host(
GPU_memory_mode, scheduler_dict, model_name=model_name, model_type=model_type,
config_path=config_path, compile_dit=compile_dit,
weight_dtype=weight_dtype, savedir_sample=savedir_sample,
)
with gr.Blocks(css=css) as demo:
gr.Markdown(
"""
# Wan2.2-Fun:
A Wan with more flexible generation conditions, capable of producing videos of different resolutions, around 5 seconds, and fps 16 (frames 1 to 81), as well as image generated videos.
[Github](https://github.com/aigc-apps/VideoX-Fun/)
"""
)
with gr.Column(variant="panel"):
model_type = create_fake_model_type(visible=False)
diffusion_transformer_dropdown = create_fake_model_checkpoints(model_name, visible=True)
base_model_dropdown, lora_model_dropdown, lora_alpha_slider = \
create_fake_finetune_models_checkpoints(visible=True, add_checkpoint_2=True)
base_model_dropdown, base_model_2_dropdown = base_model_dropdown
lora_model_dropdown, lora_model_2_dropdown = lora_model_dropdown
with gr.Row():
enable_teacache, teacache_threshold, num_skip_start_steps, teacache_offload = \
create_teacache_params(True, 0.10, 1, False)
cfg_skip_ratio = create_cfg_skip_params(0)
enable_riflex, riflex_k = create_cfg_riflex_k(False, 6)
with gr.Column(variant="panel"):
prompt_textbox, negative_prompt_textbox = create_prompts(negative_prompt="色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走")
with gr.Row():
with gr.Column():
sampler_dropdown, sample_step_slider = create_samplers(controller)
resize_method, width_slider, height_slider, base_resolution = create_height_width(
default_height = 480, default_width = 832, maximum_height = 1344,
maximum_width = 1344,
)
generation_method, length_slider, overlap_video_length, partial_video_length = \
create_generation_methods_and_video_length(
["Video Generation", "Image Generation"],
default_video_length=81,
maximum_video_length=161,
)
image_to_video_col, video_to_video_col, control_video_col, source_method, start_image, template_gallery, end_image, validation_video, validation_video_mask, denoise_strength, control_video, ref_image = create_generation_method(
["Text to Video (文本到视频)", "Image to Video (图片到视频)", "Video Control (视频控制)"], prompt_textbox, support_ref_image=True
)
cfg_scale_slider, seed_textbox, seed_button = create_cfg_and_seedbox(gradio_version_is_above_4)
generate_button = gr.Button(value="Generate (生成)", variant='primary')
result_image, result_video, infer_progress = create_ui_outputs()
def upload_generation_method(generation_method):
if generation_method == "Video Generation":
return gr.update(visible=True, minimum=1, maximum=161, value=81, interactive=True)
elif generation_method == "Image Generation":
return gr.update(minimum=1, maximum=1, value=1, interactive=False)
generation_method.change(
upload_generation_method, generation_method, [length_slider]
)
def upload_source_method(source_method):
if source_method == "Text to Video (文本到视频)":
return [gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update(value=None)]
elif source_method == "Image to Video (图片到视频)":
return [gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(), gr.update(), gr.update(value=None), gr.update(value=None), gr.update(value=None)]
elif source_method == "Video to Video (视频到视频)":
return [gr.update(visible=False), gr.update(visible=True), gr.update(visible=False), gr.update(value=None), gr.update(value=None), gr.update(), gr.update(), gr.update(value=None)]
else:
return [gr.update(visible=False), gr.update(visible=False), gr.update(visible=True), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update()]
source_method.change(
upload_source_method, source_method, [
image_to_video_col, video_to_video_col, control_video_col, start_image, end_image,
validation_video, validation_video_mask, control_video
]
)
def upload_resize_method(resize_method):
if resize_method == "Generate by":
return [gr.update(visible=True), gr.update(visible=True), gr.update(visible=False)]
else:
return [gr.update(visible=False), gr.update(visible=False), gr.update(visible=True)]
resize_method.change(
upload_resize_method, resize_method, [width_slider, height_slider, base_resolution]
)
generate_button.click(
fn=controller.generate,
inputs=[
diffusion_transformer_dropdown,
base_model_dropdown,
lora_model_dropdown,
lora_alpha_slider,
prompt_textbox,
negative_prompt_textbox,
sampler_dropdown,
sample_step_slider,
resize_method,
width_slider,
height_slider,
base_resolution,
generation_method,
length_slider,
overlap_video_length,
partial_video_length,
cfg_scale_slider,
start_image,
end_image,
validation_video,
validation_video_mask,
control_video,
denoise_strength,
seed_textbox,
ref_image,
enable_teacache,
teacache_threshold,
num_skip_start_steps,
teacache_offload,
cfg_skip_ratio,
enable_riflex,
riflex_k,
base_model_2_dropdown,
lora_model_2_dropdown
],
outputs=[result_image, result_video, infer_progress]
)
return demo, controller
def ui_client(scheduler_dict, model_name, savedir_sample=None):
controller = Wan2_2_Fun_Controller_Client(scheduler_dict, savedir_sample)
with gr.Blocks(css=css) as demo:
gr.Markdown(
"""
# Wan2.2-Fun:
A Wan with more flexible generation conditions, capable of producing videos of different resolutions, around 5 seconds, and fps 16 (frames 1 to 81), as well as image generated videos.
[Github](https://github.com/aigc-apps/VideoX-Fun/)
"""
)
with gr.Column(variant="panel"):
diffusion_transformer_dropdown = create_fake_model_checkpoints(model_name, visible=True)
base_model_dropdown, lora_model_dropdown, lora_alpha_slider = \
create_fake_finetune_models_checkpoints(visible=True, add_checkpoint_2=True)
base_model_dropdown, base_model_2_dropdown = base_model_dropdown
lora_model_dropdown, lora_model_2_dropdown = lora_model_dropdown
with gr.Row():
enable_teacache, teacache_threshold, num_skip_start_steps, teacache_offload = \
create_teacache_params(True, 0.10, 1, False)
cfg_skip_ratio = create_cfg_skip_params(0)
enable_riflex, riflex_k = create_cfg_riflex_k(False, 6)
with gr.Column(variant="panel"):
prompt_textbox, negative_prompt_textbox = create_prompts(negative_prompt="色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走")
with gr.Row():
with gr.Column():
sampler_dropdown, sample_step_slider = create_samplers(controller, maximum_step=50)
resize_method, width_slider, height_slider, base_resolution = create_fake_height_width(
default_height = 480, default_width = 832, maximum_height = 1344,
maximum_width = 1344,
)
generation_method, length_slider, overlap_video_length, partial_video_length = \
create_generation_methods_and_video_length(
["Video Generation", "Image Generation"],
default_video_length=81,
maximum_video_length=161,
)
image_to_video_col, video_to_video_col, control_video_col, source_method, start_image, template_gallery, end_image, validation_video, validation_video_mask, denoise_strength, control_video, ref_image = create_generation_method(
["Text to Video (文本到视频)", "Image to Video (图片到视频)"], prompt_textbox
)
cfg_scale_slider, seed_textbox, seed_button = create_cfg_and_seedbox(gradio_version_is_above_4)
generate_button = gr.Button(value="Generate (生成)", variant='primary')
result_image, result_video, infer_progress = create_ui_outputs()
def upload_generation_method(generation_method):
if generation_method == "Video Generation":
return gr.update(visible=True, minimum=5, maximum=161, value=49, interactive=True)
elif generation_method == "Image Generation":
return gr.update(minimum=1, maximum=1, value=1, interactive=False)
generation_method.change(
upload_generation_method, generation_method, [length_slider]
)
def upload_source_method(source_method):
if source_method == "Text to Video (文本到视频)":
return [gr.update(visible=False), gr.update(visible=False), gr.update(value=None), gr.update(value=None), gr.update(value=None), gr.update(value=None)]
elif source_method == "Image to Video (图片到视频)":
return [gr.update(visible=True), gr.update(visible=False), gr.update(), gr.update(), gr.update(value=None), gr.update(value=None)]
else:
return [gr.update(visible=False), gr.update(visible=True), gr.update(value=None), gr.update(value=None), gr.update(), gr.update()]
source_method.change(
upload_source_method, source_method, [image_to_video_col, video_to_video_col, start_image, end_image, validation_video, validation_video_mask]
)
def upload_resize_method(resize_method):
if resize_method == "Generate by":
return [gr.update(visible=True), gr.update(visible=True), gr.update(visible=False)]
else:
return [gr.update(visible=False), gr.update(visible=False), gr.update(visible=True)]
resize_method.change(
upload_resize_method, resize_method, [width_slider, height_slider, base_resolution]
)
generate_button.click(
fn=controller.generate,
inputs=[
diffusion_transformer_dropdown,
base_model_dropdown,
lora_model_dropdown,
lora_alpha_slider,
prompt_textbox,
negative_prompt_textbox,
sampler_dropdown,
sample_step_slider,
resize_method,
width_slider,
height_slider,
base_resolution,
generation_method,
length_slider,
cfg_scale_slider,
start_image,
end_image,
validation_video,
validation_video_mask,
denoise_strength,
seed_textbox,
ref_image,
enable_teacache,
teacache_threshold,
num_skip_start_steps,
teacache_offload,
cfg_skip_ratio,
enable_riflex,
riflex_k,
base_model_2_dropdown,
lora_model_2_dropdown
],
outputs=[result_image, result_video, infer_progress]
)
return demo, controller
videox_fun/ui/wan2_2_ui.py
+1 -1
View File
@@ -93,7 +93,7 @@ class Wan2_2_Controller(Fun_Controller):
# Get pipeline
if self.model_type == "Inpaint":
if "ti2v" in self.config_path:
if "wan_civitai_5b" in self.config_path:
self.pipeline = Wan2_2TI2VPipeline(
vae=self.vae,
tokenizer=self.tokenizer,