diff --git a/README.ja.md b/README.ja.md index f04c571..0796c87 100644 --- a/README.ja.md +++ b/README.ja.md @@ -57,9 +57,67 @@ pip install -r requirements.txt [MIT License](LICENSE) +## 更新履歴 + +### v2.0.0 - musubi-tuner完全互換対応 (2025-01-XX) + +複数リファレンス画像使用時の推論結果の一貫性を改善するため、**musubi-tuner仕様との完全互換性**を実現しました。 + +#### 主要な変更点 + +**1. エンベッディング統合方式の改善** +- ❌ 従来：重み付き平均による統合処理（入力画像70%、参照画像30%） +- ✅ **新版：** musubi-tuner互換の処理方式（最初の参照画像エンベッディングを使用） + +**2. Latent結合構造の統一** +- ❌ 従来：入力画像と参照画像を分離管理後に結合 +- ✅ **新版：** musubi-tuner仕様のcontrol_latents直接結合 + ```python + control_latents = [入力画像, 参照画像1, 参照画像2, ..., ゼロlatent] + clean_latents = torch.cat(control_latents, dim=2) + ``` + +**3. マスク適用タイミングの最適化** +- ❌ 従来：latent結合前の個別適用 +- ✅ **新版：** clean_latents生成後にマスク適用（musubi-tuner仕様） + +**4. インデックス設定の動的処理** +- ❌ 従来：固定的なclean_latent_indices設定 +- ✅ **新版：** control_indicesパラメータの動的適用 + ```python + # control_index="0;7;8;9;10" → clean_latent_indices = [0, 7, 8, 9, 10] + while i < len(control_indices_list) and i < clean_latent_indices.shape[1]: + clean_latent_indices[:, i] = control_indices_list[i] + ``` + +**5. latent_indicesの初期化改善** +- ❌ 従来：ComfyUI独自の初期化方式 +- ✅ **新版：** musubi-tuner仕様の初期化 + ```python + latent_indices = torch.zeros((1, 1), dtype=torch.int64) + latent_indices[:, 0] = latent_window_size # デフォルト値 + latent_indices[:, 0] = target_index # パラメータ適用 + ``` + +#### 期待される効果 + +- **推論結果の一貫性向上**: 同じリファレンス画像・同じパラメータでmusubi-tunerと完全に同じ結果を生成 +- **複数リファレンス処理の安定化**: より正確なインデックス管理による安定した品質 +- **パラメータ互換性**: musubi-tunerのcontrol_indexとtarget_indexパラメータが正しく動作 + +#### 技術的詳細 + +この更新により、以下の処理フローがmusubi-tunerと完全に一致します： + +1. **制御画像処理**: `--control_image_path`で指定された複数画像の順次処理 +2. **インデックス管理**: `--one_frame_inference="control_index=0;7;8;9;10,target_index=5"`の動的適用 +3. **エンベッディング処理**: セクション毎の個別処理を模擬した実装 +4. **マスク適用**: clean_latents構築後の統一的なマスク処理 + ## クレジット - [FramePack](https://github.com/lllyasviel/FramePack): @lllyasviel 氏によるオリジナルの実装です。 - [ComfyUI-FramePackWrapper](https://github.com/kijai/ComfyUI-FramePackWrapper): @kijai 氏によるオリジナルの実装です。 - [ComfyUI-FramePackWrapper_Plus](https://github.com/ShmuelRonen/ComfyUI-FramePackWrapper_Plus): @ShmuelRonen 氏によるF1対応のフォークです。 - [ComfyUI-FramePackWrapper_PlusOne](https://github.com/tori29umai0123/ComfyUI-FramePackWrapper_PlusOne): @tori29umai0123 氏による1フレーム推論対応のフォークです。 +- [musubi-tuner](https://github.com/kohya-ss/musubi-tuner): @kohya-ss 氏による高品質なFramePack学習・推論ライブラリ diff --git a/README.md b/README.md index 71b891d..4c60714 100644 --- a/README.md +++ b/README.md @@ -59,9 +59,67 @@ See [example_workflows](./example_workflows). [MIT License](LICENSE) +## Changelog + +### v2.0.0 - Full musubi-tuner Compatibility (2025-01-XX) + +Achieved **complete compatibility with musubi-tuner specifications** to improve inference result consistency when using multiple reference images. + +#### Major Changes + +**1. Improved Embedding Integration Method** +- ❌ Previous: Weighted average integration (70% input image, 30% reference images) +- ✅ **New:** musubi-tuner compatible processing (using first reference image embedding) + +**2. Unified Latent Combination Structure** +- ❌ Previous: Separate management of input and reference images before combination +- ✅ **New:** Direct control_latents combination following musubi-tuner specification + ```python + control_latents = [input_image, reference_image1, reference_image2, ..., zero_latent] + clean_latents = torch.cat(control_latents, dim=2) + ``` + +**3. Optimized Mask Application Timing** +- ❌ Previous: Individual application before latent combination +- ✅ **New:** Mask application after clean_latents generation (musubi-tuner specification) + +**4. Dynamic Index Setting Processing** +- ❌ Previous: Fixed clean_latent_indices configuration +- ✅ **New:** Dynamic application of control_indices parameters + ```python + # control_index="0;7;8;9;10" → clean_latent_indices = [0, 7, 8, 9, 10] + while i < len(control_indices_list) and i < clean_latent_indices.shape[1]: + clean_latent_indices[:, i] = control_indices_list[i] + ``` + +**5. Improved latent_indices Initialization** +- ❌ Previous: ComfyUI-specific initialization method +- ✅ **New:** musubi-tuner specification initialization + ```python + latent_indices = torch.zeros((1, 1), dtype=torch.int64) + latent_indices[:, 0] = latent_window_size # default value + latent_indices[:, 0] = target_index # parameter application + ``` + +#### Expected Benefits + +- **Improved Inference Consistency**: Generate identical results to musubi-tuner with same reference images and parameters +- **Stabilized Multi-Reference Processing**: More stable quality through accurate index management +- **Parameter Compatibility**: Correct functionality of musubi-tuner's control_index and target_index parameters + +#### Technical Details + +This update ensures the following processing flow matches musubi-tuner completely: + +1. **Control Image Processing**: Sequential processing of multiple images specified by `--control_image_path` +2. **Index Management**: Dynamic application of `--one_frame_inference="control_index=0;7;8;9;10,target_index=5"` +3. **Embedding Processing**: Implementation simulating section-wise individual processing +4. **Mask Application**: Unified mask processing after clean_latents construction + ## Credits - [FramePack](https://github.com/lllyasviel/FramePack): @lllyasviel's original implementation. - [ComfyUI-FramePackWrapper](https://github.com/kijai/ComfyUI-FramePackWrapper): @kijai's original implementation. - [ComfyUI-FramePackWrapper_Plus](https://github.com/ShmuelRonen/ComfyUI-FramePackWrapper_Plus): @ShmuelRonen's F1-supported fork. - [ComfyUI-FramePackWrapper_PlusOne](https://github.com/tori29umai0123/ComfyUI-FramePackWrapper_PlusOne): @tori29umai0123's 1-frame inference-supported fork. +- [musubi-tuner](https://github.com/kohya-ss/musubi-tuner): @kohya-ss's high-quality FramePack training and inference library diff --git a/__init__.py b/__init__.py index 51b5973..0832cae 100644 --- a/__init__.py +++ b/__init__.py @@ -1,11 +1,11 @@ -from .nodes import NODE_CLASS_MAPPINGS as NODES_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS as NODES_DISPLAY_MAPPINGS -from .nodes_F1 import NODE_CLASS_MAPPINGS as F1_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS as F1_DISPLAY_MAPPINGS -from .nodes_one import NODE_CLASS_MAPPINGS as ONE_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS as ONE_DISPLAY_MAPPINGS -from .nodes_resize import NODE_CLASS_MAPPINGS as RESIZE_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS as RESIZE_DISPLAY_MAPPINGS - - -# Combine the mappings -NODE_CLASS_MAPPINGS = {**NODES_MAPPINGS, **F1_MAPPINGS, **ONE_MAPPINGS, **RESIZE_MAPPINGS} -NODE_DISPLAY_NAME_MAPPINGS = {**NODES_DISPLAY_MAPPINGS, **F1_DISPLAY_MAPPINGS, **ONE_DISPLAY_MAPPINGS, **RESIZE_DISPLAY_MAPPINGS} - +from .nodes import NODE_CLASS_MAPPINGS as NODES_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS as NODES_DISPLAY_MAPPINGS +from .nodes_F1 import NODE_CLASS_MAPPINGS as F1_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS as F1_DISPLAY_MAPPINGS +from .nodes_one import NODE_CLASS_MAPPINGS as ONE_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS as ONE_DISPLAY_MAPPINGS +from .nodes_resize import NODE_CLASS_MAPPINGS as RESIZE_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS as RESIZE_DISPLAY_MAPPINGS + + +# Combine the mappings +NODE_CLASS_MAPPINGS = {**NODES_MAPPINGS, **F1_MAPPINGS, **ONE_MAPPINGS, **RESIZE_MAPPINGS} +NODE_DISPLAY_NAME_MAPPINGS = {**NODES_DISPLAY_MAPPINGS, **F1_DISPLAY_MAPPINGS, **ONE_DISPLAY_MAPPINGS, **RESIZE_DISPLAY_MAPPINGS} + __all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS'] \ No newline at end of file diff --git a/diffusers_helper/models/hunyuan_video_packed.py b/diffusers_helper/models/hunyuan_video_packed.py index 29ab32f..9c4018e 100644 --- a/diffusers_helper/models/hunyuan_video_packed.py +++ b/diffusers_helper/models/hunyuan_video_packed.py @@ -832,8 +832,33 @@ class HunyuanVideoTransformer3DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, hidden_states = hidden_states.flatten(2).transpose(1, 2) + # Generate rope frequencies for the full sequence rope_freqs = self.rope(frame_indices=latent_indices, height=H, width=W, device=hidden_states.device) rope_freqs = rope_freqs.flatten(2).transpose(1, 2) + + # Fix rope_freqs size to match packed hidden_states + if rope_freqs.shape[1] != hidden_states.shape[1]: + # Calculate the ratio for expansion + hidden_seq_len = hidden_states.shape[1] # 23562 + rope_seq_len = rope_freqs.shape[1] # 11781 + + if hidden_seq_len > rope_seq_len: + # Calculate how many times to repeat + repeat_factor = hidden_seq_len // rope_seq_len + remainder = hidden_seq_len % rope_seq_len + + print(f"Expanding rope_freqs: {rope_seq_len} -> {hidden_seq_len} (repeat_factor={repeat_factor}, remainder={remainder})") + + # Repeat rope_freqs to match hidden_states size + rope_freqs = rope_freqs.repeat(1, repeat_factor, 1) + if remainder > 0: + rope_freqs = torch.cat([rope_freqs, rope_freqs[:, :remainder]], dim=1) + + print(f"Final rope_freqs shape: {rope_freqs.shape}") + elif hidden_seq_len < rope_seq_len: + # Trim rope_freqs to match hidden_states size + rope_freqs = rope_freqs[:, :hidden_seq_len, :] + print(f"Trimmed rope_freqs shape: {rope_freqs.shape}") if clean_latents is not None and clean_latent_indices is not None: clean_latents = clean_latents.to(hidden_states) @@ -842,6 +867,20 @@ class HunyuanVideoTransformer3DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, clean_latent_rope_freqs = self.rope(frame_indices=clean_latent_indices, height=H, width=W, device=clean_latents.device) clean_latent_rope_freqs = clean_latent_rope_freqs.flatten(2).transpose(1, 2) + + # Ensure clean_latent_rope_freqs matches clean_latents size + if clean_latent_rope_freqs.shape[1] != clean_latents.shape[1]: + clean_seq_len = clean_latents.shape[1] + clean_rope_seq_len = clean_latent_rope_freqs.shape[1] + + if clean_seq_len > clean_rope_seq_len: + repeat_factor = clean_seq_len // clean_rope_seq_len + remainder = clean_seq_len % clean_rope_seq_len + clean_latent_rope_freqs = clean_latent_rope_freqs.repeat(1, repeat_factor, 1) + if remainder > 0: + clean_latent_rope_freqs = torch.cat([clean_latent_rope_freqs, clean_latent_rope_freqs[:, :remainder]], dim=1) + elif clean_seq_len < clean_rope_seq_len: + clean_latent_rope_freqs = clean_latent_rope_freqs[:, :clean_seq_len, :] hidden_states = torch.cat([clean_latents, hidden_states], dim=1) rope_freqs = torch.cat([clean_latent_rope_freqs, rope_freqs], dim=1) @@ -856,6 +895,20 @@ class HunyuanVideoTransformer3DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, clean_latent_2x_rope_freqs = pad_for_3d_conv(clean_latent_2x_rope_freqs, (2, 2, 2)) clean_latent_2x_rope_freqs = center_down_sample_3d(clean_latent_2x_rope_freqs, (2, 2, 2)) clean_latent_2x_rope_freqs = clean_latent_2x_rope_freqs.flatten(2).transpose(1, 2) + + # Ensure clean_latent_2x_rope_freqs matches clean_latents_2x size + if clean_latent_2x_rope_freqs.shape[1] != clean_latents_2x.shape[1]: + clean_2x_seq_len = clean_latents_2x.shape[1] + clean_2x_rope_seq_len = clean_latent_2x_rope_freqs.shape[1] + + if clean_2x_seq_len > clean_2x_rope_seq_len: + repeat_factor = clean_2x_seq_len // clean_2x_rope_seq_len + remainder = clean_2x_seq_len % clean_2x_rope_seq_len + clean_latent_2x_rope_freqs = clean_latent_2x_rope_freqs.repeat(1, repeat_factor, 1) + if remainder > 0: + clean_latent_2x_rope_freqs = torch.cat([clean_latent_2x_rope_freqs, clean_latent_2x_rope_freqs[:, :remainder]], dim=1) + elif clean_2x_seq_len < clean_2x_rope_seq_len: + clean_latent_2x_rope_freqs = clean_latent_2x_rope_freqs[:, :clean_2x_seq_len, :] hidden_states = torch.cat([clean_latents_2x, hidden_states], dim=1) rope_freqs = torch.cat([clean_latent_2x_rope_freqs, rope_freqs], dim=1) @@ -870,6 +923,20 @@ class HunyuanVideoTransformer3DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, clean_latent_4x_rope_freqs = pad_for_3d_conv(clean_latent_4x_rope_freqs, (4, 4, 4)) clean_latent_4x_rope_freqs = center_down_sample_3d(clean_latent_4x_rope_freqs, (4, 4, 4)) clean_latent_4x_rope_freqs = clean_latent_4x_rope_freqs.flatten(2).transpose(1, 2) + + # Ensure clean_latent_4x_rope_freqs matches clean_latents_4x size + if clean_latent_4x_rope_freqs.shape[1] != clean_latents_4x.shape[1]: + clean_4x_seq_len = clean_latents_4x.shape[1] + clean_4x_rope_seq_len = clean_latent_4x_rope_freqs.shape[1] + + if clean_4x_seq_len > clean_4x_rope_seq_len: + repeat_factor = clean_4x_seq_len // clean_4x_rope_seq_len + remainder = clean_4x_seq_len % clean_4x_rope_seq_len + clean_latent_4x_rope_freqs = clean_latent_4x_rope_freqs.repeat(1, repeat_factor, 1) + if remainder > 0: + clean_latent_4x_rope_freqs = torch.cat([clean_latent_4x_rope_freqs, clean_latent_4x_rope_freqs[:, :remainder]], dim=1) + elif clean_4x_seq_len < clean_4x_rope_seq_len: + clean_latent_4x_rope_freqs = clean_latent_4x_rope_freqs[:, :clean_4x_seq_len, :] hidden_states = torch.cat([clean_latents_4x, hidden_states], dim=1) rope_freqs = torch.cat([clean_latent_4x_rope_freqs, rope_freqs], dim=1) diff --git a/example_workflows/Oneframe_2Dstyle_V2.json b/example_workflows/Oneframe_2Dstyle_V2.json new file mode 100644 index 0000000..80a32d4 --- /dev/null +++ b/example_workflows/Oneframe_2Dstyle_V2.json @@ -0,0 +1,1362 @@ +{ + "id": "3b7225ab-6ecc-42a6-a98a-f5bfdbde3d6f", + "revision": 0, + "last_node_id": 133, + "last_link_id": 292, + "nodes": [ + { + "id": 12, + "type": "VAELoader", + "pos": [ + 570.5363159179688, + -282.70068359375 + ], + "size": [ + 469.0488586425781, + 58 + ], + "flags": {}, + "order": 0, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "VAE", + "type": "VAE", + "links": [ + 153 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.28", + "Node name for S&R": "VAELoader", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "hunyuan_video_vae_bf16.safetensors" + ], + "color": "#322", + "bgcolor": "#533" + }, + { + "id": 66, + "type": "SetNode", + "pos": [ + 1068.862060546875, + -235.50686645507812 + ], + "size": [ + 210, + 60 + ], + "flags": { + "collapsed": true + }, + "order": 10, + "mode": 0, + "inputs": [ + { + "name": "VAE", + "type": "VAE", + "link": 153 + } + ], + "outputs": [ + { + "name": "*", + "type": "*", + "links": null + } + ], + "title": "Set_VAE", + "properties": { + "previousName": "VAE", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "VAE" + ], + "color": "#322", + "bgcolor": "#533" + }, + { + "id": 63, + "type": "SetNode", + "pos": [ + 1056.7816162109375, + -388.6712341308594 + ], + "size": [ + 210, + 60 + ], + "flags": { + "collapsed": true + }, + "order": 11, + "mode": 0, + "inputs": [ + { + "name": "CLIP_VISION", + "type": "CLIP_VISION", + "link": 148 + } + ], + "outputs": [ + { + "name": "*", + "type": "*", + "links": null + } + ], + "title": "Set_ClipVisionModle", + "properties": { + "previousName": "ClipVisionModle", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "ClipVisionModle" + ], + "color": "#233", + "bgcolor": "#355" + }, + { + "id": 18, + "type": "CLIPVisionLoader", + "pos": [ + 611.4688720703125, + -424.41949462890625 + ], + "size": [ + 388.87139892578125, + 58 + ], + "flags": {}, + "order": 1, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "CLIP_VISION", + "type": "CLIP_VISION", + "links": [ + 148 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.28", + "Node name for S&R": "CLIPVisionLoader", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "sigclip_vision_patch14_384.safetensors" + ], + "color": "#2a363b", + "bgcolor": "#3f5159" + }, + { + "id": 13, + "type": "DualCLIPLoader", + "pos": [ + 793.314208984375, + 165.36949157714844 + ], + "size": [ + 340.2243957519531, + 130 + ], + "flags": {}, + "order": 2, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "CLIP", + "type": "CLIP", + "links": [ + 102 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.28", + "Node name for S&R": "DualCLIPLoader", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "clip_l.safetensors", + "llava_llama3_fp16.safetensors", + "hunyuan_video", + "default" + ], + "color": "#432", + "bgcolor": "#653" + }, + { + "id": 67, + "type": "GetNode", + "pos": [ + 2564.861572265625, + -154.08343505859375 + ], + "size": [ + 210, + 60 + ], + "flags": { + "collapsed": true + }, + "order": 3, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "VAE", + "type": "VAE", + "links": [ + 154 + ] + } + ], + "title": "Get_VAE", + "properties": { + "widget_ue_connectable": {} + }, + "widgets_values": [ + "VAE" + ], + "color": "#322", + "bgcolor": "#533" + }, + { + "id": 15, + "type": "ConditioningZeroOut", + "pos": [ + 1748.7374267578125, + 142.03829956054688 + ], + "size": [ + 317.4000244140625, + 26 + ], + "flags": { + "collapsed": true + }, + "order": 15, + "mode": 0, + "inputs": [ + { + "name": "conditioning", + "type": "CONDITIONING", + "link": 118 + } + ], + "outputs": [ + { + "name": "CONDITIONING", + "type": "CONDITIONING", + "links": [ + 231 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.28", + "Node name for S&R": "ConditioningZeroOut", + "widget_ue_connectable": {} + }, + "widgets_values": [], + "color": "#332922", + "bgcolor": "#593930" + }, + { + "id": 68, + "type": "GetNode", + "pos": [ + 1148.14208984375, + 595.065673828125 + ], + "size": [ + 210, + 58 + ], + "flags": { + "collapsed": false + }, + "order": 4, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "VAE", + "type": "VAE", + "links": [ + 155 + ] + } + ], + "title": "Get_VAE", + "properties": { + "widget_ue_connectable": {} + }, + "widgets_values": [ + "VAE" + ], + "color": "#322", + "bgcolor": "#533" + }, + { + "id": 64, + "type": "GetNode", + "pos": [ + 911.8778686523438, + 407.6289978027344 + ], + "size": [ + 210, + 60 + ], + "flags": { + "collapsed": false + }, + "order": 5, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "CLIP_VISION", + "type": "CLIP_VISION", + "links": [ + 149 + ] + } + ], + "title": "Get_ClipVisionModle", + "properties": { + "widget_ue_connectable": {} + }, + "widgets_values": [ + "ClipVisionModle" + ], + "color": "#233", + "bgcolor": "#355" + }, + { + "id": 20, + "type": "VAEEncode", + "pos": [ + 1533.1563720703125, + 632.9931640625 + ], + "size": [ + 210, + 46 + ], + "flags": { + "collapsed": false + }, + "order": 17, + "mode": 0, + "inputs": [ + { + "name": "pixels", + "type": "IMAGE", + "link": 253 + }, + { + "name": "vae", + "type": "VAE", + "link": 155 + } + ], + "outputs": [ + { + "name": "LATENT", + "type": "LATENT", + "slot_index": 0, + "links": [ + 229, + 242 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.28", + "Node name for S&R": "VAEEncode", + "widget_ue_connectable": {} + }, + "widgets_values": [], + "color": "#322", + "bgcolor": "#533" + }, + { + "id": 17, + "type": "CLIPVisionEncode", + "pos": [ + 1286.9630126953125, + 439.6824951171875 + ], + "size": [ + 380.4000244140625, + 78 + ], + "flags": { + "collapsed": false + }, + "order": 18, + "mode": 0, + "inputs": [ + { + "name": "clip_vision", + "type": "CLIP_VISION", + "link": 149 + }, + { + "name": "image", + "type": "IMAGE", + "link": 254 + } + ], + "outputs": [ + { + "name": "CLIP_VISION_OUTPUT", + "type": "CLIP_VISION_OUTPUT", + "links": [ + 232 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.28", + "Node name for S&R": "CLIPVisionEncode", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "none" + ], + "color": "#233", + "bgcolor": "#355" + }, + { + "id": 27, + "type": "FramePackTorchCompileSettings", + "pos": [ + 623.3660278320312, + -140.94215393066406 + ], + "size": [ + 531.5999755859375, + 202 + ], + "flags": {}, + "order": 6, + "mode": 4, + "inputs": [], + "outputs": [ + { + "name": "torch_compile_args", + "type": "FRAMEPACKCOMPILEARGS", + "slot_index": 0, + "links": [ + 274 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "0e5fe5d7ca13c76fb8e13708f4b92e7c7a34f20c", + "Node name for S&R": "FramePackTorchCompileSettings", + "aux_id": "lllyasviel/FramePack", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "inductor", + false, + "default", + false, + 64, + true, + true + ] + }, + { + "id": 52, + "type": "LoadFramePackModel", + "pos": [ + 1608.3046875, + -312.9743347167969 + ], + "size": [ + 480.7601013183594, + 174 + ], + "flags": {}, + "order": 14, + "mode": 0, + "inputs": [ + { + "name": "compile_args", + "shape": 7, + "type": "FRAMEPACKCOMPILEARGS", + "link": 274 + }, + { + "name": "lora", + "shape": 7, + "type": "FPLORA", + "link": 159 + } + ], + "outputs": [ + { + "name": "model", + "type": "FramePackMODEL", + "slot_index": 0, + "links": [ + 236 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "49fe507eca8246cc9d08a8093892f40c1180e88f", + "Node name for S&R": "LoadFramePackModel", + "aux_id": "kijai/ComfyUI-FramePackWrapper", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "FramePackI2V_HY_fp8_e4m3fn.safetensors", + "bf16", + "fp8_e5m2", + "offload_device", + "sageattn" + ] + }, + { + "id": 47, + "type": "CLIPTextEncode", + "pos": [ + 1288.7686767578125, + -81.72645568847656 + ], + "size": [ + 400, + 200 + ], + "flags": {}, + "order": 12, + "mode": 0, + "inputs": [ + { + "name": "clip", + "type": "CLIP", + "link": 102 + }, + { + "name": "text", + "type": "STRING", + "widget": { + "name": "text" + }, + "link": 292 + } + ], + "outputs": [ + { + "name": "CONDITIONING", + "type": "CONDITIONING", + "slot_index": 0, + "links": [ + 118, + 230 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.28", + "Node name for S&R": "CLIPTextEncode", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "A static photograph morphs into a black-and-white line drawing style, with the camera and composition held perfectly fixed throughout.", + [ + false, + true + ] + ], + "color": "#232", + "bgcolor": "#353" + }, + { + "id": 114, + "type": "ImageResize+", + "pos": [ + 1010.8599853515625, + 755.2196044921875 + ], + "size": [ + 315, + 218 + ], + "flags": {}, + "order": 16, + "mode": 0, + "inputs": [ + { + "name": "image", + "type": "IMAGE", + "link": 275 + }, + { + "name": "width", + "type": "INT", + "widget": { + "name": "width" + }, + "link": 251 + }, + { + "name": "height", + "type": "INT", + "widget": { + "name": "height" + }, + "link": 252 + } + ], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "slot_index": 0, + "links": [ + 253, + 254, + 263 + ] + }, + { + "name": "width", + "type": "INT", + "links": null + }, + { + "name": "height", + "type": "INT", + "links": null + } + ], + "properties": { + "cnr_id": "comfyui_essentials", + "ver": "1.1.0", + "Node name for S&R": "ImageResize+" + }, + "widgets_values": [ + 512, + 512, + "lanczos", + "stretch", + "always", + 0 + ] + }, + { + "id": 33, + "type": "VAEDecodeTiled", + "pos": [ + 2516.953125, + 550.6734619140625 + ], + "size": [ + 315, + 150 + ], + "flags": { + "collapsed": true + }, + "order": 20, + "mode": 0, + "inputs": [ + { + "name": "samples", + "type": "LATENT", + "link": 260 + }, + { + "name": "vae", + "type": "VAE", + "link": 154 + } + ], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "links": [ + 261, + 267 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.28", + "Node name for S&R": "VAEDecodeTiled", + "widget_ue_connectable": {} + }, + "widgets_values": [ + 256, + 64, + 64, + 8 + ], + "color": "#322", + "bgcolor": "#533" + }, + { + "id": 118, + "type": "SaveImage", + "pos": [ + 2792.264892578125, + 1127.350830078125 + ], + "size": [ + 315, + 270 + ], + "flags": {}, + "order": 23, + "mode": 0, + "inputs": [ + { + "name": "images", + "type": "IMAGE", + "link": 280 + } + ], + "outputs": [], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.39", + "Node name for S&R": "SaveImage" + }, + "widgets_values": [ + "ComfyUI" + ] + }, + { + "id": 117, + "type": "ImageConcatMulti", + "pos": [ + 2867.469970703125, + 875.7557373046875 + ], + "size": [ + 315, + 170 + ], + "flags": {}, + "order": 22, + "mode": 0, + "inputs": [ + { + "name": "image_1", + "type": "IMAGE", + "link": 263 + }, + { + "name": "image_2", + "shape": 7, + "type": "IMAGE", + "link": 267 + } + ], + "outputs": [ + { + "name": "images", + "type": "IMAGE", + "links": [ + 280 + ] + } + ], + "properties": { + "cnr_id": "comfyui-kjnodes", + "ver": "9a52b7dfd28b8079e6c63636fd2e2d1dff37d403" + }, + "widgets_values": [ + 2, + "right", + false, + null + ] + }, + { + "id": 87, + "type": "SaveImage", + "pos": [ + 3665.5546875, + 678.6862182617188 + ], + "size": [ + 623.8917236328125, + 711.2797241210938 + ], + "flags": {}, + "order": 21, + "mode": 0, + "inputs": [ + { + "name": "images", + "type": "IMAGE", + "link": 261 + } + ], + "outputs": [], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.29", + "Node name for S&R": "SaveImage", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "ComfyUI" + ] + }, + { + "id": 130, + "type": "Text Multiline", + "pos": [ + 782.9840698242188, + 1090.4462890625 + ], + "size": [ + 400, + 200 + ], + "flags": {}, + "order": 7, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "STRING", + "type": "STRING", + "links": [ + 292 + ] + } + ], + "properties": { + "cnr_id": "was-node-suite-comfyui", + "ver": "1cd8d304eda256c412b8589ce1f00be3c61cf9ec", + "Node name for S&R": "Text Multiline" + }, + "widgets_values": [ + "Converts the input image into a 2D illustration style while preserving the content.", + [ + false, + true + ] + ] + }, + { + "id": 113, + "type": "FramePackFindNearestBucket", + "pos": [ + 546.7589111328125, + 777.2312622070312 + ], + "size": [ + 315, + 78 + ], + "flags": {}, + "order": 13, + "mode": 0, + "inputs": [ + { + "name": "image", + "type": "IMAGE", + "link": 248 + } + ], + "outputs": [ + { + "name": "width", + "type": "INT", + "slot_index": 0, + "links": [ + 251 + ] + }, + { + "name": "height", + "type": "INT", + "slot_index": 1, + "links": [ + 252 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "93e60c83ffc73c249339955f8684d86487484c27", + "Node name for S&R": "FramePackFindNearestBucket", + "aux_id": "ShmuelRonen/ComfyUI-FramePackWrapper_Plus" + }, + "widgets_values": [ + 1072 + ] + }, + { + "id": 77, + "type": "LoadImage", + "pos": [ + -137.56338500976562, + 635.01123046875 + ], + "size": [ + 615.048828125, + 680.713134765625 + ], + "flags": {}, + "order": 8, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "slot_index": 0, + "links": [ + 248, + 275 + ] + }, + { + "name": "MASK", + "type": "MASK", + "links": null + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.29", + "Node name for S&R": "LoadImage", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "025_4.png", + "image" + ] + }, + { + "id": 70, + "type": "FramePackLoraSelect", + "pos": [ + 1275.2337646484375, + -275.54351806640625 + ], + "size": [ + 270, + 106 + ], + "flags": {}, + "order": 9, + "mode": 4, + "inputs": [ + { + "name": "prev_lora", + "shape": 7, + "type": "FPLORA", + "link": null + } + ], + "outputs": [ + { + "name": "lora", + "type": "FPLORA", + "links": [ + 159 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "5300cdb786950ff44b6688481ae4b307b2d8d6f2", + "Node name for S&R": "FramePackLoraSelect", + "aux_id": "kijai/ComfyUI-FramePackWrapper", + "widget_ue_connectable": {} + }, + "widgets_values": [ + "framepack\\2Dstyle_V1_1024_dim4-000016.safetensors", + 1.0000000000000002, + false + ] + }, + { + "id": 107, + "type": "FramePackSingleFrameSampler", + "pos": [ + 2064.802490234375, + 899.3915405273438 + ], + "size": [ + 380.4000244140625, + 590 + ], + "flags": {}, + "order": 19, + "mode": 0, + "inputs": [ + { + "name": "model", + "type": "FramePackMODEL", + "link": 236 + }, + { + "name": "positive", + "type": "CONDITIONING", + "link": 230 + }, + { + "name": "negative", + "type": "CONDITIONING", + "link": 231 + }, + { + "name": "start_latent", + "type": "LATENT", + "link": 229 + }, + { + "name": "image_embeds", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": 232 + }, + { + "name": "initial_samples", + "shape": 7, + "type": "LATENT", + "link": 242 + }, + { + "name": "reference_latents", + "shape": 7, + "type": "REFERENCE_LATENT_LIST", + "link": null + }, + { + "name": "reference_image_embeds", + "shape": 7, + "type": "REFERENCE_EMBEDS_LIST", + "link": null + }, + { + "name": "reference_masks", + "shape": 7, + "type": "REFERENCE_MASK_LIST", + "link": null + }, + { + "name": "input_mask", + "shape": 7, + "type": "MASK", + "link": null + }, + { + "name": "reference_latent", + "shape": 7, + "type": "LATENT", + "link": null + }, + { + "name": "reference_mask", + "shape": 7, + "type": "MASK", + "link": null + } + ], + "outputs": [ + { + "name": "samples", + "type": "LATENT", + "slot_index": 0, + "links": [ + 260 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "93e60c83ffc73c249339955f8684d86487484c27", + "Node name for S&R": "FramePackSingleFrameSampler", + "aux_id": "ShmuelRonen/ComfyUI-FramePackWrapper_Plus" + }, + "widgets_values": [ + 10, + false, + 0.15, + 1, + 10, + 0, + 175660897657082, + "randomize", + 9, + 25, + "unipc_bh1", + 1, + 1, + "0,10" + ] + } + ], + "links": [ + [ + 102, + 13, + 0, + 47, + 0, + "CLIP" + ], + [ + 118, + 47, + 0, + 15, + 0, + "CONDITIONING" + ], + [ + 148, + 18, + 0, + 63, + 0, + "*" + ], + [ + 149, + 64, + 0, + 17, + 0, + "CLIP_VISION" + ], + [ + 153, + 12, + 0, + 66, + 0, + "*" + ], + [ + 154, + 67, + 0, + 33, + 1, + "VAE" + ], + [ + 155, + 68, + 0, + 20, + 1, + "VAE" + ], + [ + 159, + 70, + 0, + 52, + 1, + "FPLORA" + ], + [ + 229, + 20, + 0, + 107, + 3, + "LATENT" + ], + [ + 230, + 47, + 0, + 107, + 1, + "CONDITIONING" + ], + [ + 231, + 15, + 0, + 107, + 2, + "CONDITIONING" + ], + [ + 232, + 17, + 0, + 107, + 4, + "CLIP_VISION_OUTPUT" + ], + [ + 236, + 52, + 0, + 107, + 0, + "FramePackMODEL" + ], + [ + 242, + 20, + 0, + 107, + 5, + "LATENT" + ], + [ + 248, + 77, + 0, + 113, + 0, + "IMAGE" + ], + [ + 251, + 113, + 0, + 114, + 1, + "INT" + ], + [ + 252, + 113, + 1, + 114, + 2, + "INT" + ], + [ + 253, + 114, + 0, + 20, + 0, + "IMAGE" + ], + [ + 254, + 114, + 0, + 17, + 1, + "IMAGE" + ], + [ + 260, + 107, + 0, + 33, + 0, + "LATENT" + ], + [ + 261, + 33, + 0, + 87, + 0, + "IMAGE" + ], + [ + 263, + 114, + 0, + 117, + 0, + "IMAGE" + ], + [ + 267, + 33, + 0, + 117, + 1, + "IMAGE" + ], + [ + 274, + 27, + 0, + 52, + 0, + "FRAMEPACKCOMPILEARGS" + ], + [ + 275, + 77, + 0, + 114, + 0, + "IMAGE" + ], + [ + 280, + 117, + 0, + 118, + 0, + "IMAGE" + ], + [ + 292, + 130, + 0, + 47, + 1, + "STRING" + ] + ], + "groups": [], + "config": {}, + "extra": { + "ds": { + "scale": 0.4305676431342499, + "offset": [ + 312.9786189616261, + 793.6365528872677 + ] + }, + "frontendVersion": "1.20.7", + "VHS_latentpreview": false, + "VHS_latentpreviewrate": 0, + "VHS_MetadataImage": true, + "VHS_KeepIntermediate": true, + "ue_links": [], + "links_added_by_ue": [] + }, + "version": 0.4 +} \ No newline at end of file diff --git a/example_workflows/Oneframe_kisekae_V2.json b/example_workflows/Oneframe_kisekae_V2.json new file mode 100644 index 0000000..6e674cb --- /dev/null +++ b/example_workflows/Oneframe_kisekae_V2.json @@ -0,0 +1,1703 @@ +{ + "id": "204b7339-5d8d-48b8-89c8-1c069671ebc7", + "revision": 0, + "last_node_id": 145, + "last_link_id": 146, + "nodes": [ + { + "id": 12, + "type": "VAELoader", + "pos": [ + 100, + 130 + ], + "size": [ + 315, + 58 + ], + "flags": {}, + "order": 0, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "VAE", + "type": "VAE", + "links": [ + 34, + 36, + 49 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "VAELoader" + }, + "widgets_values": [ + "hunyuan_video_vae_bf16.safetensors" + ] + }, + { + "id": 13, + "type": "DualCLIPLoader", + "pos": [ + 100, + 318 + ], + "size": [ + 315, + 130 + ], + "flags": {}, + "order": 1, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "CLIP", + "type": "CLIP", + "links": [ + 37 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "DualCLIPLoader" + }, + "widgets_values": [ + "clip_l.safetensors", + "llava_llama3_fp16.safetensors", + "hunyuan_video", + "default" + ] + }, + { + "id": 17, + "type": "CLIPVisionEncode", + "pos": [ + 1455.199951171875, + 130 + ], + "size": [ + 342.5999755859375, + 78 + ], + "flags": {}, + "order": 15, + "mode": 0, + "inputs": [ + { + "name": "clip_vision", + "type": "CLIP_VISION", + "link": 31 + }, + { + "name": "image", + "type": "IMAGE", + "link": 32 + } + ], + "outputs": [ + { + "name": "CLIP_VISION_OUTPUT", + "type": "CLIP_VISION_OUTPUT", + "links": [ + 138 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "CLIPVisionEncode" + }, + "widgets_values": [ + "none" + ] + }, + { + "id": 18, + "type": "CLIPVisionLoader", + "pos": [ + 100, + 554 + ], + "size": [ + 315, + 58 + ], + "flags": {}, + "order": 2, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "CLIP_VISION", + "type": "CLIP_VISION", + "links": [ + 31, + 50 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "CLIPVisionLoader" + }, + "widgets_values": [ + "sigclip_vision_patch14_384.safetensors" + ] + }, + { + "id": 20, + "type": "VAEEncode", + "pos": [ + 1455.199951171875, + 338 + ], + "size": [ + 210, + 46 + ], + "flags": {}, + "order": 16, + "mode": 0, + "inputs": [ + { + "name": "pixels", + "type": "IMAGE", + "link": 33 + }, + { + "name": "vae", + "type": "VAE", + "link": 34 + } + ], + "outputs": [ + { + "name": "LATENT", + "type": "LATENT", + "links": [ + 137, + 139 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "VAEEncode" + }, + "widgets_values": [] + }, + { + "id": 114, + "type": "ImageResize+", + "pos": [ + 1040.199951171875, + 286 + ], + "size": [ + 315, + 218 + ], + "flags": {}, + "order": 12, + "mode": 0, + "inputs": [ + { + "name": "image", + "type": "IMAGE", + "link": 55 + }, + { + "name": "width", + "type": "INT", + "widget": { + "name": "width" + }, + "link": 53 + }, + { + "name": "height", + "type": "INT", + "widget": { + "name": "height" + }, + "link": 54 + } + ], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "links": [ + 32, + 33, + 130, + 131 + ] + }, + { + "name": "width", + "type": "INT", + "links": null + }, + { + "name": "height", + "type": "INT", + "links": null + } + ], + "properties": { + "cnr_id": "comfyui_essentials", + "ver": "9d9f4bedfc9f0321c19faf71855e228c93bd0dc9", + "Node name for S&R": "ImageResize+" + }, + "widgets_values": [ + 512, + 512, + "nearest", + "stretch", + "always", + 0 + ] + }, + { + "id": 122, + "type": "FramePackTorchCompileSettings", + "pos": [ + -249.45616149902344, + 232.02886962890625 + ], + "size": [ + 315, + 202 + ], + "flags": {}, + "order": 3, + "mode": 4, + "inputs": [], + "outputs": [ + { + "name": "torch_compile_args", + "type": "FRAMEPACKCOMPILEARGS", + "slot_index": 0, + "links": [ + 75 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "c406391902715cb895ca4db3386f475a5aa1e76b", + "Node name for S&R": "FramePackTorchCompileSettings", + "aux_id": "tori29umai0123/ComfyUI-FramePackWrapper_PlusOne" + }, + "widgets_values": [ + "inductor", + false, + "default", + false, + 64, + true, + true + ] + }, + { + "id": 115, + "type": "ImageResize+", + "pos": [ + 931.3854370117188, + 681.4827880859375 + ], + "size": [ + 315, + 218 + ], + "flags": {}, + "order": 13, + "mode": 0, + "inputs": [ + { + "name": "image", + "type": "IMAGE", + "link": 58 + }, + { + "name": "width", + "type": "INT", + "widget": { + "name": "width" + }, + "link": 56 + }, + { + "name": "height", + "type": "INT", + "widget": { + "name": "height" + }, + "link": 57 + } + ], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "slot_index": 0, + "links": [ + 48, + 51 + ] + }, + { + "name": "width", + "type": "INT", + "links": null + }, + { + "name": "height", + "type": "INT", + "links": null + } + ], + "properties": { + "cnr_id": "comfyui_essentials", + "ver": "9d9f4bedfc9f0321c19faf71855e228c93bd0dc9", + "Node name for S&R": "ImageResize+" + }, + "widgets_values": [ + 512, + 512, + "nearest", + "stretch", + "always", + 0 + ] + }, + { + "id": 124, + "type": "ImageResize+", + "pos": [ + 945.208740234375, + 1073.523193359375 + ], + "size": [ + 315, + 218 + ], + "flags": {}, + "order": 14, + "mode": 0, + "inputs": [ + { + "name": "image", + "type": "IMAGE", + "link": null + }, + { + "name": "width", + "type": "INT", + "widget": { + "name": "width" + }, + "link": 96 + }, + { + "name": "height", + "type": "INT", + "widget": { + "name": "height" + }, + "link": 97 + } + ], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "slot_index": 0, + "links": [] + }, + { + "name": "width", + "type": "INT", + "links": null + }, + { + "name": "height", + "type": "INT", + "links": null + } + ], + "properties": { + "cnr_id": "comfyui_essentials", + "ver": "1.1.0", + "Node name for S&R": "ImageResize+" + }, + "widgets_values": [ + 512, + 512, + "nearest", + "stretch", + "always", + 0 + ] + }, + { + "id": 33, + "type": "VAEDecodeTiled", + "pos": [ + 2378.199951171875, + 130 + ], + "size": [ + 315, + 150 + ], + "flags": {}, + "order": 23, + "mode": 0, + "inputs": [ + { + "name": "samples", + "type": "LATENT", + "link": 141 + }, + { + "name": "vae", + "type": "VAE", + "link": 36 + } + ], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "slot_index": 0, + "links": [ + 121, + 129 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "VAEDecodeTiled" + }, + "widgets_values": [ + 256, + 64, + 64, + 8 + ] + }, + { + "id": 87, + "type": "SaveImage", + "pos": [ + 3690.87890625, + 600.3234252929688 + ], + "size": [ + 315, + 270 + ], + "flags": {}, + "order": 26, + "mode": 0, + "inputs": [ + { + "name": "images", + "type": "IMAGE", + "link": 123 + } + ], + "outputs": [], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "SaveImage" + }, + "widgets_values": [ + "ComfyUI" + ] + }, + { + "id": 126, + "type": "PreviewImage", + "pos": [ + 2608.559814453125, + 858.3909301757812 + ], + "size": [ + 210, + 246 + ], + "flags": {}, + "order": 17, + "mode": 0, + "inputs": [ + { + "name": "images", + "type": "IMAGE", + "link": 130 + } + ], + "outputs": [], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.39", + "Node name for S&R": "PreviewImage" + }, + "widgets_values": [] + }, + { + "id": 140, + "type": "SaveImage", + "pos": [ + 3033.750732421875, + 384.37677001953125 + ], + "size": [ + 315, + 270 + ], + "flags": {}, + "order": 25, + "mode": 0, + "inputs": [ + { + "name": "images", + "type": "IMAGE", + "link": 129 + } + ], + "outputs": [], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.39", + "Node name for S&R": "SaveImage" + }, + "widgets_values": [ + "ComfyUI" + ] + }, + { + "id": 15, + "type": "ConditioningZeroOut", + "pos": [ + 1041.587158203125, + 148.02928161621094 + ], + "size": [ + 210, + 26 + ], + "flags": {}, + "order": 11, + "mode": 0, + "inputs": [ + { + "name": "conditioning", + "type": "CONDITIONING", + "link": 30 + } + ], + "outputs": [ + { + "name": "CONDITIONING", + "type": "CONDITIONING", + "links": [ + 136 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "ConditioningZeroOut" + }, + "widgets_values": [] + }, + { + "id": 113, + "type": "FramePackFindNearestBucket", + "pos": [ + 506.02679443359375, + 737.021240234375 + ], + "size": [ + 315, + 78 + ], + "flags": {}, + "order": 10, + "mode": 0, + "inputs": [ + { + "name": "image", + "type": "IMAGE", + "link": 52 + } + ], + "outputs": [ + { + "name": "width", + "type": "INT", + "slot_index": 0, + "links": [ + 53, + 56, + 96 + ] + }, + { + "name": "height", + "type": "INT", + "slot_index": 1, + "links": [ + 54, + 57, + 97 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "dd9dfd08a4c1b6a8439278d2d46a86c40a8bf81c", + "Node name for S&R": "FramePackFindNearestBucket", + "aux_id": "tori29umai0123/ComfyUI-FramePackWrapper_PlusOne" + }, + "widgets_values": [ + 1024 + ] + }, + { + "id": 121, + "type": "FramePackLoraSelect", + "pos": [ + -426.4157409667969, + 473.0777893066406 + ], + "size": [ + 315, + 106 + ], + "flags": {}, + "order": 4, + "mode": 0, + "inputs": [ + { + "name": "prev_lora", + "shape": 7, + "type": "FPLORA", + "link": null + } + ], + "outputs": [ + { + "name": "lora", + "type": "FPLORA", + "slot_index": 0, + "links": [ + 74 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "c406391902715cb895ca4db3386f475a5aa1e76b", + "Node name for S&R": "FramePackLoraSelect", + "aux_id": "tori29umai0123/ComfyUI-FramePackWrapper_PlusOne" + }, + "widgets_values": [ + "framepack\\fp-1f-kisekae-1024-v4-2-PfPHEMA.safetensors", + 1, + false + ] + }, + { + "id": 137, + "type": "ImageConcatMulti", + "pos": [ + 3096.015625, + 1131.695068359375 + ], + "size": [ + 315, + 150 + ], + "flags": {}, + "order": 24, + "mode": 0, + "inputs": [ + { + "name": "image_1", + "type": "IMAGE", + "link": 131 + }, + { + "name": "image_2", + "shape": 7, + "type": "IMAGE", + "link": 121 + } + ], + "outputs": [ + { + "name": "images", + "type": "IMAGE", + "slot_index": 0, + "links": [ + 123 + ] + } + ], + "properties": { + "cnr_id": "comfyui-kjnodes", + "ver": "9a52b7dfd28b8079e6c63636fd2e2d1dff37d403" + }, + "widgets_values": [ + 2, + "right", + true, + null + ] + }, + { + "id": 47, + "type": "CLIPTextEncode", + "pos": [ + 540.2000122070312, + 130 + ], + "size": [ + 400, + 200 + ], + "flags": {}, + "order": 8, + "mode": 0, + "inputs": [ + { + "name": "clip", + "type": "CLIP", + "link": 37 + } + ], + "outputs": [ + { + "name": "CONDITIONING", + "type": "CONDITIONING", + "links": [ + 30, + 135 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "CLIPTextEncode" + }, + "widgets_values": [ + "The charactor stays in the same pose, but her outfit changes, then she changes into another charactor wearing the same outfit.\n", + [ + false, + true + ] + ] + }, + { + "id": 142, + "type": "ReferenceMaskList", + "pos": [ + 1648.2318115234375, + 861.83740234375 + ], + "size": [ + 179.056640625, + 166 + ], + "flags": {}, + "order": 5, + "mode": 0, + "inputs": [ + { + "name": "mask_1", + "shape": 7, + "type": "MASK", + "link": null + }, + { + "name": "mask_2", + "shape": 7, + "type": "MASK", + "link": null + }, + { + "name": "mask_3", + "shape": 7, + "type": "MASK", + "link": null + }, + { + "name": "mask_4", + "shape": 7, + "type": "MASK", + "link": null + }, + { + "name": "mask_5", + "shape": 7, + "type": "MASK", + "link": null + }, + { + "name": "mask_6", + "shape": 7, + "type": "MASK", + "link": null + }, + { + "name": "mask_7", + "shape": 7, + "type": "MASK", + "link": null + }, + { + "name": "mask_8", + "shape": 7, + "type": "MASK", + "link": null + } + ], + "outputs": [ + { + "name": "reference_masks", + "type": "REFERENCE_MASK_LIST", + "links": null + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "1081647f2a6c98b0709bf5e3c5341e0ad6af22b6", + "Node name for S&R": "ReferenceMaskList" + }, + "widgets_values": [] + }, + { + "id": 109, + "type": "VAEEncode", + "pos": [ + 1341.4912109375, + 521.106689453125 + ], + "size": [ + 210, + 46 + ], + "flags": {}, + "order": 18, + "mode": 0, + "inputs": [ + { + "name": "pixels", + "type": "IMAGE", + "link": 48 + }, + { + "name": "vae", + "type": "VAE", + "link": 49 + } + ], + "outputs": [ + { + "name": "LATENT", + "type": "LATENT", + "links": [ + 142 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "VAEEncode" + }, + "widgets_values": [] + }, + { + "id": 143, + "type": "ReferenceLatentList", + "pos": [ + 1923.5877685546875, + 789.0232543945312 + ], + "size": [ + 201.767578125, + 166 + ], + "flags": {}, + "order": 20, + "mode": 0, + "inputs": [ + { + "name": "reference_1", + "shape": 7, + "type": "LATENT", + "link": 142 + }, + { + "name": "reference_2", + "shape": 7, + "type": "LATENT", + "link": null + }, + { + "name": "reference_3", + "shape": 7, + "type": "LATENT", + "link": null + }, + { + "name": "reference_4", + "shape": 7, + "type": "LATENT", + "link": null + }, + { + "name": "reference_5", + "shape": 7, + "type": "LATENT", + "link": null + }, + { + "name": "reference_6", + "shape": 7, + "type": "LATENT", + "link": null + }, + { + "name": "reference_7", + "shape": 7, + "type": "LATENT", + "link": null + }, + { + "name": "reference_8", + "shape": 7, + "type": "LATENT", + "link": null + } + ], + "outputs": [ + { + "name": "reference_latents", + "type": "REFERENCE_LATENT_LIST", + "links": [ + 144 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "1081647f2a6c98b0709bf5e3c5341e0ad6af22b6", + "Node name for S&R": "ReferenceLatentList" + }, + "widgets_values": [] + }, + { + "id": 112, + "type": "CLIPVisionEncode", + "pos": [ + 1337.9378662109375, + 682.1826171875 + ], + "size": [ + 342.5999755859375, + 78 + ], + "flags": {}, + "order": 19, + "mode": 0, + "inputs": [ + { + "name": "clip_vision", + "type": "CLIP_VISION", + "link": 50 + }, + { + "name": "image", + "type": "IMAGE", + "link": 51 + } + ], + "outputs": [ + { + "name": "CLIP_VISION_OUTPUT", + "type": "CLIP_VISION_OUTPUT", + "links": [ + 146 + ] + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "CLIPVisionEncode" + }, + "widgets_values": [ + "none" + ] + }, + { + "id": 145, + "type": "ReferenceEmbedsList", + "pos": [ + 1357.8133544921875, + 850.8614501953125 + ], + "size": [ + 201.099609375, + 166 + ], + "flags": {}, + "order": 21, + "mode": 0, + "inputs": [ + { + "name": "embeds_1", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": 146 + }, + { + "name": "embeds_2", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": null + }, + { + "name": "embeds_3", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": null + }, + { + "name": "embeds_4", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": null + }, + { + "name": "embeds_5", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": null + }, + { + "name": "embeds_6", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": null + }, + { + "name": "embeds_7", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": null + }, + { + "name": "embeds_8", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": null + } + ], + "outputs": [ + { + "name": "reference_embeds", + "type": "REFERENCE_EMBEDS_LIST", + "links": [ + 145 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "1081647f2a6c98b0709bf5e3c5341e0ad6af22b6", + "Node name for S&R": "ReferenceEmbedsList" + }, + "widgets_values": [] + }, + { + "id": 77, + "type": "LoadImage", + "pos": [ + -68.3819351196289, + 1356.135009765625 + ], + "size": [ + 315, + 314 + ], + "flags": {}, + "order": 6, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "links": [ + 52, + 55 + ] + }, + { + "name": "MASK", + "type": "MASK", + "links": null + } + ], + "title": "CharacterImage", + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "LoadImage" + }, + "widgets_values": [ + "clipspace/clipspace-mask-10783786.2.png [input]", + "image" + ] + }, + { + "id": 76, + "type": "LoadImage", + "pos": [ + 298.3189697265625, + 1379.0186767578125 + ], + "size": [ + 315, + 314 + ], + "flags": {}, + "order": 7, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "links": [ + 58 + ] + }, + { + "name": "MASK", + "type": "MASK", + "links": null + } + ], + "title": "ClothImage", + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.10", + "Node name for S&R": "LoadImage" + }, + "widgets_values": [ + "clipspace/clipspace-mask-11094294.2.png [input]", + "image" + ] + }, + { + "id": 52, + "type": "LoadFramePackModel", + "pos": [ + 540.2000122070312, + 460 + ], + "size": [ + 315, + 174 + ], + "flags": {}, + "order": 9, + "mode": 0, + "inputs": [ + { + "name": "compile_args", + "shape": 7, + "type": "FRAMEPACKCOMPILEARGS", + "link": 75 + }, + { + "name": "lora", + "shape": 7, + "type": "FPLORA", + "link": 74 + } + ], + "outputs": [ + { + "name": "model", + "type": "FramePackMODEL", + "links": [ + 134 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "dd9dfd08a4c1b6a8439278d2d46a86c40a8bf81c", + "Node name for S&R": "LoadFramePackModel", + "aux_id": "tori29umai0123/ComfyUI-FramePackWrapper_PlusOne" + }, + "widgets_values": [ + "FramePackI2V_HY_fp8_e4m3fn.safetensors", + "bf16", + "disabled", + "offload_device", + "sageattn" + ] + }, + { + "id": 141, + "type": "FramePackSingleFrameSampler", + "pos": [ + 1897.800048828125, + 130 + ], + "size": [ + 314.6372985839844, + 550 + ], + "flags": {}, + "order": 22, + "mode": 0, + "inputs": [ + { + "name": "model", + "type": "FramePackMODEL", + "link": 134 + }, + { + "name": "positive", + "type": "CONDITIONING", + "link": 135 + }, + { + "name": "negative", + "type": "CONDITIONING", + "link": 136 + }, + { + "name": "start_latent", + "type": "LATENT", + "link": 137 + }, + { + "name": "image_embeds", + "shape": 7, + "type": "CLIP_VISION_OUTPUT", + "link": 138 + }, + { + "name": "initial_samples", + "shape": 7, + "type": "LATENT", + "link": 139 + }, + { + "name": "reference_latents", + "shape": 7, + "type": "REFERENCE_LATENT_LIST", + "link": 144 + }, + { + "name": "reference_image_embeds", + "shape": 7, + "type": "REFERENCE_EMBEDS_LIST", + "link": 145 + }, + { + "name": "reference_masks", + "shape": 7, + "type": "REFERENCE_MASK_LIST", + "link": null + }, + { + "name": "input_mask", + "shape": 7, + "type": "MASK", + "link": null + } + ], + "outputs": [ + { + "name": "samples", + "type": "LATENT", + "links": [ + 141 + ] + } + ], + "properties": { + "cnr_id": "ComfyUI-FramePackWrapper_PlusOne", + "ver": "1081647f2a6c98b0709bf5e3c5341e0ad6af22b6", + "Node name for S&R": "FramePackSingleFrameSampler" + }, + "widgets_values": [ + 20, + false, + 0.15, + 1, + 10, + 0, + 292613918460138, + "randomize", + 9, + 25, + "unipc_bh1", + 1, + 5, + "0;10" + ] + } + ], + "links": [ + [ + 30, + 47, + 0, + 15, + 0, + "CONDITIONING" + ], + [ + 31, + 18, + 0, + 17, + 0, + "CLIP_VISION" + ], + [ + 32, + 114, + 0, + 17, + 1, + "IMAGE" + ], + [ + 33, + 114, + 0, + 20, + 0, + "IMAGE" + ], + [ + 34, + 12, + 0, + 20, + 1, + "VAE" + ], + [ + 36, + 12, + 0, + 33, + 1, + "VAE" + ], + [ + 37, + 13, + 0, + 47, + 0, + "CLIP" + ], + [ + 48, + 115, + 0, + 109, + 0, + "IMAGE" + ], + [ + 49, + 12, + 0, + 109, + 1, + "VAE" + ], + [ + 50, + 18, + 0, + 112, + 0, + "CLIP_VISION" + ], + [ + 51, + 115, + 0, + 112, + 1, + "IMAGE" + ], + [ + 52, + 77, + 0, + 113, + 0, + "IMAGE" + ], + [ + 53, + 113, + 0, + 114, + 1, + "INT" + ], + [ + 54, + 113, + 1, + 114, + 2, + "INT" + ], + [ + 55, + 77, + 0, + 114, + 0, + "IMAGE" + ], + [ + 56, + 113, + 0, + 115, + 1, + "INT" + ], + [ + 57, + 113, + 1, + 115, + 2, + "INT" + ], + [ + 58, + 76, + 0, + 115, + 0, + "IMAGE" + ], + [ + 74, + 121, + 0, + 52, + 1, + "FPLORA" + ], + [ + 75, + 122, + 0, + 52, + 0, + "FRAMEPACKCOMPILEARGS" + ], + [ + 96, + 113, + 0, + 124, + 1, + "INT" + ], + [ + 97, + 113, + 1, + 124, + 2, + "INT" + ], + [ + 121, + 33, + 0, + 137, + 1, + "IMAGE" + ], + [ + 123, + 137, + 0, + 87, + 0, + "IMAGE" + ], + [ + 129, + 33, + 0, + 140, + 0, + "IMAGE" + ], + [ + 130, + 114, + 0, + 126, + 0, + "IMAGE" + ], + [ + 131, + 114, + 0, + 137, + 0, + "IMAGE" + ], + [ + 134, + 52, + 0, + 141, + 0, + "FramePackMODEL" + ], + [ + 135, + 47, + 0, + 141, + 1, + "CONDITIONING" + ], + [ + 136, + 15, + 0, + 141, + 2, + "CONDITIONING" + ], + [ + 137, + 20, + 0, + 141, + 3, + "LATENT" + ], + [ + 138, + 17, + 0, + 141, + 4, + "CLIP_VISION_OUTPUT" + ], + [ + 139, + 20, + 0, + 141, + 5, + "LATENT" + ], + [ + 141, + 141, + 0, + 33, + 0, + "LATENT" + ], + [ + 142, + 109, + 0, + 143, + 0, + "LATENT" + ], + [ + 144, + 143, + 0, + 141, + 6, + "REFERENCE_LATENT_LIST" + ], + [ + 145, + 145, + 0, + 141, + 7, + "REFERENCE_EMBEDS_LIST" + ], + [ + 146, + 112, + 0, + 145, + 0, + "CLIP_VISION_OUTPUT" + ] + ], + "groups": [], + "config": {}, + "extra": { + "ds": { + "scale": 0.9849732675807764, + "offset": [ + -817.394152987616, + -234.96062523744845 + ] + }, + "VHS_latentpreview": false, + "VHS_latentpreviewrate": 0, + "VHS_MetadataImage": true, + "VHS_KeepIntermediate": true, + "frontendVersion": "1.20.7" + }, + "version": 0.4 +} \ No newline at end of file diff --git a/nodes.py b/nodes.py index 5ab3962..57717d9 100644 --- a/nodes.py +++ b/nodes.py @@ -8,8 +8,8 @@ from pathlib import Path logger = logging.getLogger(__name__) -from accelerate import init_empty_weights -from accelerate.utils import set_module_tensor_to_device +# from accelerate import init_empty_weights +# from accelerate.utils import set_module_tensor_to_device import folder_paths import comfy.model_management as mm @@ -24,7 +24,7 @@ script_directory = os.path.dirname(os.path.abspath(__file__)) vae_scaling_factor = 0.476986 from .diffusers_helper.models.hunyuan_video_packed import HunyuanVideoTransformer3DModel -from .diffusers_helper.memory import DynamicSwapInstaller, move_model_to_device_with_memory_preservation +from .diffusers_helper.memory import move_model_to_device_with_memory_preservation from .diffusers_helper.pipelines.k_diffusion_hunyuan import sample_hunyuan from .diffusers_helper.utils import crop_or_pad_yield_mask from .diffusers_helper.bucket_tools import find_nearest_bucket @@ -439,7 +439,7 @@ class DownloadAndLoadFramePackModel: local_dir_use_symlinks=False, ) - transformer = HunyuanVideoTransformer3DModel.from_pretrained(model_path, torch_dtype=base_dtype, attention_mode=attention_mode).cpu() + transformer = HunyuanVideoTransformer3DModel.from_pretrained(model_path, torch_dtype=base_dtype, attention_mode=attention_mode) params_to_keep = {"norm", "bias", "time_in", "vector_in", "guidance_in", "txt_in", "img_in"} if quantization == 'fp8_e4m3fn' or quantization == 'fp8_e4m3fn_fast': transformer = transformer.to(torch.float8_e4m3fn) @@ -451,7 +451,8 @@ class DownloadAndLoadFramePackModel: else: transformer = transformer.to(base_dtype) - DynamicSwapInstaller.install_model(transformer, device=device) + # Move model directly to device + transformer = transformer.to(device) if compile_args is not None: if compile_args["compile_single_blocks"]: @@ -546,10 +547,14 @@ class LoadFramePackModel: import json with open(model_config_path, "r") as f: config = json.load(f) - sd = load_torch_file(model_path, device=offload_device, safe_load=True) + + # Load state dict directly to the target device + sd = load_torch_file(model_path, device=transformer_load_device, safe_load=True) model_weight_dtype = sd['single_transformer_blocks.0.attn.to_k.weight'].dtype - with init_empty_weights(): - transformer = HunyuanVideoTransformer3DModel(**config, attention_mode=attention_mode) + + # Create model on the target device directly with proper dtype + transformer = HunyuanVideoTransformer3DModel(**config, attention_mode=attention_mode) + params_to_keep = {"norm", "bias", "time_in", "vector_in", "guidance_in", "txt_in", "img_in"} if quantization == "fp8_e4m3fn" or quantization == "fp8_e4m3fn_fast" or quantization == "fp8_scaled": dtype = torch.float8_e4m3fn @@ -557,17 +562,45 @@ class LoadFramePackModel: dtype = torch.float8_e5m2 else: dtype = base_dtype + + # Store the original dtype for later use with LoRA + after_lora_dtype = dtype if lora is not None: - after_lora_dtype = dtype + # When LoRA is present, we need to use base_dtype initially dtype = base_dtype - print("Using accelerate to load and assign model weights to device...") - param_count = sum(1 for _ in transformer.named_parameters()) - for name, param in tqdm(transformer.named_parameters(), - desc=f"Loading transformer parameters to {transformer_load_device}", - total=param_count, - leave=True): - dtype_to_use = base_dtype if any(keyword in name for keyword in params_to_keep) else dtype - set_module_tensor_to_device(transformer, name, device=transformer_load_device, dtype=dtype_to_use, value=sd[name]) + + # Convert dtype for state dict if needed when LoRA is present + # Only convert if dtypes actually differ to avoid unnecessary memory operations + if dtype != model_weight_dtype: + print(f"Converting model weights from {model_weight_dtype} to {dtype} for LoRA compatibility") + for key in sd.keys(): + if any(keyword in key for keyword in params_to_keep): + sd[key] = sd[key].to(base_dtype) + else: + sd[key] = sd[key].to(dtype) + else: + # When no LoRA, only convert if model dtype differs from target dtype + # AND if quantization is disabled (otherwise we handle it later) + if quantization == "disabled" and dtype != model_weight_dtype: + print(f"Converting model weights from {model_weight_dtype} to {dtype}") + for key in sd.keys(): + if any(keyword in key for keyword in params_to_keep): + sd[key] = sd[key].to(base_dtype) + else: + sd[key] = sd[key].to(dtype) + + # Load state dict all at once + print(f"Loading transformer state dict to {transformer_load_device}") + info = transformer.load_state_dict(sd, strict=True) + print(f"Loaded state dict: {info}") + + # Move model to device with proper dtype + if quantization != "disabled" and lora is None: + # For quantized models without LoRA, convert to target dtype directly + transformer = transformer.to(transformer_load_device).to(after_lora_dtype) + else: + # For non-quantized or LoRA models, just move to device + transformer = transformer.to(transformer_load_device) if lora is not None: adapter_list = [] @@ -634,18 +667,31 @@ class LoadFramePackModel: transformer.fuse_lora(lora_scale=lora_scale) transformer.delete_adapters(adapter_list) + # if quantization == "fp8_e4m3fn" or quantization == "fp8_e4m3fn_fast" or quantization == "fp8_e5m2": + # params_to_keep = {"norm", "bias", "time_in", "vector_in", "guidance_in", "txt_in", "img_in"} + # for name, param in transformer.named_parameters(): + # # Make sure to not cast the LoRA weights to fp8. + # if not any(keyword in name for keyword in params_to_keep) and not 'lora' in name: + # param.data = param.data.to(after_lora_dtype) + + if quantization == "fp8_e4m3fn" or quantization == "fp8_e4m3fn_fast" or quantization == "fp8_e5m2": params_to_keep = {"norm", "bias", "time_in", "vector_in", "guidance_in", "txt_in", "img_in"} for name, param in transformer.named_parameters(): - # Make sure to not cast the LoRA weights to fp8. + # Make sure to not cast the LORA weights to fp8. if not any(keyword in name for keyword in params_to_keep) and not 'lora' in name: param.data = param.data.to(after_lora_dtype) + + + if quantization == "fp8_e4m3fn_fast": from .fp8_optimization import convert_fp8_linear convert_fp8_linear(transformer, base_dtype, params_to_keep=params_to_keep) - DynamicSwapInstaller.install_model(transformer, device=device) + # Move model to device if it's not already there + if transformer_load_device != device: + transformer = transformer.to(device) if compile_args is not None: if compile_args["compile_single_blocks"]: diff --git a/nodes_one.py b/nodes_one.py index 34905a1..4aca01e 100644 --- a/nodes_one.py +++ b/nodes_one.py @@ -30,10 +30,10 @@ class FramePackSingleFrameSampler: "LATENT", {"tooltip": "init Latents to use for image2image"}, ), - "steps": ("INT", {"default": 30, "min": 1}), + "steps": ("INT", {"default": 20, "min": 1, "tooltip": "Sampling steps (10=fast preview, 20=balanced, 30=high quality)"}), "use_teacache": ( "BOOLEAN", - {"default": True, "tooltip": "Use teacache for faster sampling."}, + {"default": False, "tooltip": "Use TeaCache for faster sampling (recommended for 15+ steps)."}, ), "teacache_rel_l1_thresh": ( "FLOAT", @@ -79,13 +79,6 @@ class FramePackSingleFrameSampler: }, ), "sampler": (["unipc_bh1", "unipc_bh2"], {"default": "unipc_bh1"}), - "use_kisekaeichi": ( - "BOOLEAN", - { - "default": False, - "tooltip": "Enable Kisekaeichi mode for style transfer", - }, - ), }, "optional": { "image_embeds": ("CLIP_VISION_OUTPUT",), @@ -97,42 +90,39 @@ class FramePackSingleFrameSampler: "FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.01}, ), - "reference_latent": ( - "LATENT", - {"tooltip": "Reference image latent for kisekaeichi mode"}, + "reference_latents": ( + "REFERENCE_LATENT_LIST", + {"tooltip": "List of reference image latents for kisekaeichi mode (up to 8)"}, ), "reference_image_embeds": ( - "CLIP_VISION_OUTPUT", - {"tooltip": "Reference image CLIP embeds for kisekaeichi mode"}, + "REFERENCE_EMBEDS_LIST", + {"tooltip": "List of reference image CLIP embeds for kisekaeichi mode (up to 8)"}, + ), + "reference_masks": ( + "REFERENCE_MASK_LIST", + {"tooltip": "List of reference masks for selective features (up to 8, optional)"}, ), "target_index": ( "INT", { - "default": 1, + "default": 5, "min": 0, - "max": 8, + "max": 16, "step": 1, "tooltip": "Target index for kisekaeichi mode (recommended: 1)", }, ), - "history_index": ( - "INT", + "control_index": ( + "STRING", { - "default": 13, - "min": 0, - "max": 16, - "step": 1, - "tooltip": "History index for kisekaeichi mode (recommended: 13)", + "default": "0;10", + "tooltip": "Control indices separated by semicolon (preferred) or comma: 0;7;8;9;10 or 0,7,8,9,10 (default: 0;10)", }, ), "input_mask": ( "MASK", {"tooltip": "Input mask for selective application"}, ), - "reference_mask": ( - "MASK", - {"tooltip": "Reference mask for selective features"}, - ), }, } @@ -140,7 +130,7 @@ class FramePackSingleFrameSampler: RETURN_NAMES = ("samples",) FUNCTION = "process" CATEGORY = "FramePackWrapper" - DESCRIPTION = "Single frame sampler with Kisekaeichi (style transfer) support" + DESCRIPTION = "Single frame sampler with auto-enabled Kisekaeichi (style transfer) support for multiple references" def process( self, @@ -161,18 +151,36 @@ class FramePackSingleFrameSampler: image_embeds=None, initial_samples=None, denoise_strength=1.0, - use_kisekaeichi=False, - reference_latent=None, + reference_latents=None, reference_image_embeds=None, + reference_masks=None, target_index=1, - history_index=13, + control_index="0;10", input_mask=None, - reference_mask=None, ): - print("=== 1フレーム推論モード ===") + print("=== 1フレーム推論モード（musubi-tuner完全互換） ===") + + # Use control_index parameter directly (no more one_frame_inference) + + # Auto-enable kisekaeichi when reference inputs are provided + use_kisekaeichi = reference_latents is not None and len(reference_latents) > 0 if use_kisekaeichi: - print("Kisekaeichi（着せ替え）モード有効") - print(f"target_index: {target_index}, history_index: {history_index}") + print("Kisekaeichi（着せ替え）モード有効 (自動検出)") + print(f"target_index: {target_index}") + print(f"control_index raw value: '{control_index}'") + print(f"control_index type: {type(control_index)}") + + # Parse control indices from parameter (support both semicolon and comma for backward compatibility) + if ';' in control_index: + # New format (musubi-tuner compatible): semicolon-separated + control_indices_list = [int(x.strip()) for x in control_index.split(';') if x.strip()] + else: + # Old format: comma-separated for backward compatibility + control_indices_list = [int(x.strip()) for x in control_index.split(',') if x.strip()] + + if len(control_indices_list) == 0: + control_indices_list = [0, 10] # default + print(f"parsed control_index: {control_indices_list}") transformer = model["transformer"] base_dtype = model["dtype"] @@ -188,9 +196,13 @@ class FramePackSingleFrameSampler: start_latent = start_latent["samples"] * vae_scaling_factor if initial_samples is not None: initial_samples = initial_samples["samples"] * vae_scaling_factor - if use_kisekaeichi and reference_latent is not None: - reference_latent = reference_latent["samples"] * vae_scaling_factor - print(f"参照画像latent: {reference_latent.shape}") + # Process reference latents if provided + processed_reference_latents = [] + if use_kisekaeichi and reference_latents is not None: + for i, ref_latent in enumerate(reference_latents): + processed_latent = ref_latent["samples"] * vae_scaling_factor + processed_reference_latents.append(processed_latent) + print(f"参照画像latent {i+1}: {processed_latent.shape}") print("start_latent", start_latent.shape) B, C, T, H, W = start_latent.shape @@ -203,13 +215,20 @@ class FramePackSingleFrameSampler: else: start_image_encoder_last_hidden_state = None + # Process reference image embeds if provided + processed_reference_embeds = [] if use_kisekaeichi and reference_image_embeds is not None: - reference_image_encoder_last_hidden_state = reference_image_embeds[ - "last_hidden_state" - ].to(device, base_dtype) - print("参照画像のCLIP embeddingを設定しました") - else: - reference_image_encoder_last_hidden_state = None + for i, ref_embeds in enumerate(reference_image_embeds): + embed_state = ref_embeds["last_hidden_state"].to(device, base_dtype) + processed_reference_embeds.append(embed_state) + print(f"参照画像 {i+1} のCLIP embeddingを設定しました") + + # Validate that latents and embeds have same count + if use_kisekaeichi: + if len(processed_reference_latents) != len(processed_reference_embeds): + raise ValueError(f"Reference latents count ({len(processed_reference_latents)}) must match embeds count ({len(processed_reference_embeds)})") + if len(processed_reference_latents) == 0: + raise ValueError("No reference latents provided for kisekaeichi mode") # テキストエンベッディング処理 llama_vec = positive[0][0].to(device, base_dtype) @@ -270,164 +289,166 @@ class FramePackSingleFrameSampler: # 一つ目のコードの重要な処理：1フレームモード時のone_frame_inference処理 if sample_num_frames == 1: - if use_kisekaeichi and reference_latent is not None: + if use_kisekaeichi and len(processed_reference_latents) > 0: print("=== Kisekaeichi モード設定（完全版） ===") # 一つ目のコードのone_frame_inference処理を完全再現 one_frame_inference = set() one_frame_inference.add(f"target_index={target_index}") - one_frame_inference.add(f"history_index={history_index}") + # control_indexは後でcontrol_index_listから設定されるため、ここでは追加しない - # 公式実装に従った処理 - latent_indices = indices[:, -1:] # デフォルトは最後のフレーム + # musubi-tuner仕様：latent_indicesを初期化 + latent_indices = torch.zeros((1, 1), dtype=torch.int64) # 1x1 latent index for target image + latent_indices[:, 0] = latent_window_size # musubi-tunerのデフォルト値 + + # Apply target_index parameter (musubi-tuner仕様) + latent_indices[:, 0] = target_index + print(f"target_index設定 (musubi-tuner仕様): {target_index}") + + # この部分は削除（後でまとめて処理するため） - # パラメータ解析と処理（一つ目のコードと同じ） - for one_frame_param in one_frame_inference: - if one_frame_param.startswith("target_index="): - target_idx = int(one_frame_param.split("=")[1]) - latent_indices[:, 0] = target_idx - print(f"latent_indices設定: target_index={target_idx}") - - elif one_frame_param.startswith("history_index="): - history_idx = int(one_frame_param.split("=")[1]) - clean_latent_indices_post[:, 0] = history_idx - print( - f"clean_latent_indices_post設定: history_index={history_idx}" - ) - - # history_latentsのダミーを作成（一つ目のコードと同じ構造） - history_latents = torch.zeros( + # control_latentsのダミーを作成（一つ目のコードと同じ構造） + control_latents = torch.zeros( size=(1, 16, 1 + 2 + 16, H, W), dtype=torch.float32, device="cpu" ) - # clean_latents_preの設定（入力画像） - clean_latents_pre = start_latent.to(history_latents.dtype).to( - history_latents.device - ) + # musubi-tuner仕様：control_latentsを直接結合（マスク適用は後で実行） + control_latents = [] + + # 入力画像をcontrol_latentsに追加 + clean_latents_pre = start_latent.to(torch.float32).cpu() if len(clean_latents_pre.shape) < 5: clean_latents_pre = clean_latents_pre.unsqueeze(2) + control_latents.append(clean_latents_pre) + + # 参照画像をcontrol_latentsに追加 + for i, ref_latent in enumerate(processed_reference_latents): + clean_latent_post = ref_latent[:, :, 0:1, :, :].to(torch.float32).cpu() + control_latents.append(clean_latent_post) + + # musubi-tuner仕様：ゼロlatentをclean_latents_postとして追加 + control_latents.append(torch.zeros((1, 16, 1, H, W), dtype=torch.float32)) + print(f"Add zero latents as clean latents post for one frame inference.") + + # musubi-tuner仕様：control_latentsを直接結合 + clean_latents = torch.cat(control_latents, dim=2) # (1, 16, num_control_images, H//8, W//8) - # マスクの適用（入力画像） - if input_mask is not None: - print("入力画像マスクを適用中...") - try: - height_latent, width_latent = clean_latents_pre.shape[-2:] - - if isinstance(input_mask, torch.Tensor): - input_mask_tensor = input_mask - else: - input_mask_tensor = torch.from_numpy(input_mask) - - input_mask_resized = ( - common_upscale( - input_mask_tensor.unsqueeze(0).unsqueeze(0), - width_latent, - height_latent, - "bilinear", - "center", + # musubi-tuner仕様：clean_latent_indicesを設定 + clean_latent_indices = torch.zeros((1, len(control_latents)), dtype=torch.int64) + if "no_post" not in []: # one_frame_inferenceは空なので常にtrue + clean_latent_indices[:, -1] = 1 + latent_window_size # default index for clean latents post + + # musubi-tuner仕様：control_indices_listをclean_latent_indicesに動的適用 + if control_indices_list: + print(f"control_indices動的適用開始: {control_indices_list}") + i = 0 + while i < len(control_indices_list) and i < clean_latent_indices.shape[1]: + control_index = int(control_indices_list[i]) + clean_latent_indices[:, i] = control_index + print(f"clean_latent_indices[:, {i}] = {control_index}") + i += 1 + print(f"最終的なclean_latent_indices: {clean_latent_indices}") + else: + print("control_indices_listが空のため、デフォルト設定を使用") + + # musubi-tuner仕様：マスク適用（clean_latents生成後） + for i in range(len(control_latents)): + mask_image = None + + # 入力画像のマスク適用 + if i == 0 and input_mask is not None: + print("入力画像マスクを適用中（musubi-tuner仕様）...") + try: + height_latent, width_latent = clean_latents.shape[-2:] + if isinstance(input_mask, torch.Tensor): + input_mask_tensor = input_mask + else: + input_mask_tensor = torch.from_numpy(input_mask) + + input_mask_resized = ( + common_upscale( + input_mask_tensor.unsqueeze(0).unsqueeze(0), + width_latent, + height_latent, + "bilinear", + "center", + ) + .squeeze(0) + .squeeze(0) ) - .squeeze(0) - .squeeze(0) - ) - input_mask_resized = input_mask_resized.to( - clean_latents_pre.device - )[None, None, :, :] - clean_latents_pre = clean_latents_pre * input_mask_resized - print("入力画像マスクを適用しました") - except Exception as e: - print(f"入力マスク適用エラー: {e}") - - # clean_latents_postの設定（参照画像） - clean_latents_post = ( - reference_latent[:, :, 0:1, :, :] - .to(history_latents.dtype) - .to(history_latents.device) - ) - - # マスクの適用（参照画像） - if reference_mask is not None: - print("参照画像マスクを適用中...") - try: - height_latent, width_latent = clean_latents_post.shape[-2:] - - if isinstance(reference_mask, torch.Tensor): - reference_mask_tensor = reference_mask - else: - reference_mask_tensor = torch.from_numpy(reference_mask) - - reference_mask_resized = ( - common_upscale( - reference_mask_tensor.unsqueeze(0).unsqueeze(0), - width_latent, - height_latent, - "bilinear", - "center", - ) - .squeeze(0) - .squeeze(0) - ) - reference_mask_resized = reference_mask_resized.to( - clean_latents_post.device - )[None, None, :, :] - clean_latents_post = clean_latents_post * reference_mask_resized - print("参照画像マスクを適用しました") - except Exception as e: - print(f"参照マスク適用エラー: {e}") - - # clean_latentsを結合 - clean_latents = torch.cat( - [clean_latents_pre, clean_latents_post], dim=2 - ) - - # clean_latent_indicesを結合 - clean_latent_indices = torch.cat( - [clean_latent_indices_pre, clean_latent_indices_post], dim=1 - ) + mask_image = input_mask_resized.to(clean_latents.device)[None, None, :, :] + print("入力画像マスクを適用しました") + except Exception as e: + print(f"入力マスク適用エラー: {e}") + + # 参照画像のマスク適用 + elif i > 0 and i < len(control_latents) - 1: # -1はゼロlatentなので除外 + ref_index = i - 1 # 最初が入力画像なので-1 + if reference_masks is not None and ref_index < len(reference_masks): + print(f"参照画像 {ref_index+1} マスクを適用中（musubi-tuner仕様）...") + try: + height_latent, width_latent = clean_latents.shape[-2:] + reference_mask = reference_masks[ref_index] + + if isinstance(reference_mask, torch.Tensor): + reference_mask_tensor = reference_mask + else: + reference_mask_tensor = torch.from_numpy(reference_mask) + + reference_mask_resized = ( + common_upscale( + reference_mask_tensor.unsqueeze(0).unsqueeze(0), + width_latent, + height_latent, + "bilinear", + "center", + ) + .squeeze(0) + .squeeze(0) + ) + mask_image = reference_mask_resized.to(clean_latents.device)[None, None, :, :] + print(f"参照画像 {ref_index+1} マスクを適用しました") + except Exception as e: + print(f"参照マスク {ref_index+1} 適用エラー: {e}") + + # マスクを適用 + if mask_image is not None: + clean_latents[:, :, i : i + 1, :, :] = clean_latents[:, :, i : i + 1, :, :] * mask_image + # Always disable 2x/4x processing for kisekaeichi mode clean_latents_2x_param = None - clean_latents_4x_param = None clean_latent_2x_indices = None - clean_latent_4x_indices = None - - # 2x, 4x latentsの設定も無効化 clean_latents_2x = None + + clean_latents_4x_param = None + clean_latent_4x_indices = None clean_latents_4x = None print("Kisekaeichi: 2x/4xインデックスを無効化しました") - # 画像エンベッディングの処理（両方を活用） - if ( - reference_image_encoder_last_hidden_state is not None - and start_image_encoder_last_hidden_state is not None - ): - # 重み付き平均で統合 - ref_weight = 0.3 # 参照画像の重み - input_weight = 1.0 - ref_weight - image_encoder_last_hidden_state = ( - start_image_encoder_last_hidden_state * input_weight - + reference_image_encoder_last_hidden_state * ref_weight - ) - print( - f"画像エンベッディングを統合 (入力:{input_weight:.2f}, 参照:{ref_weight:.2f})" - ) - elif reference_image_encoder_last_hidden_state is not None: - image_encoder_last_hidden_state = ( - reference_image_encoder_last_hidden_state - ) + # 画像エンベッディングの処理（musubi-tuner仕様：セクション毎の個別処理） + # musubi-tunerではセクション毎に個別のimage_encoder_last_hidden_stateを使用 + # 複数リファレンス時は最初の参照画像のエンベッディングを使用（musubi-tuner互換） + if len(processed_reference_embeds) > 0: + # musubi-tuner互換：最初の参照画像エンベッディングを使用 + image_encoder_last_hidden_state = processed_reference_embeds[0] + print(f"musubi-tuner互換：最初の参照画像エンベッディングを使用 (参照画像数: {len(processed_reference_embeds)})") else: - image_encoder_last_hidden_state = ( - start_image_encoder_last_hidden_state - ) + image_encoder_last_hidden_state = start_image_encoder_last_hidden_state - print(f"Kisekaeichi設定完了:") - print(f" - clean_latents.shape: {clean_latents.shape} (入力+参照)") - print(f" - latent_indices: {latent_indices}") - print(f" - clean_latent_indices: {clean_latent_indices}") + print(f"Kisekaeichi設定完了（musubi-tuner完全互換）:") + print(f" - clean_latents.shape: {clean_latents.shape} (入力+参照{len(processed_reference_latents)}個+ゼロlatent)") + print(f" - latent_indices: {latent_indices} (初期値: {latent_window_size} -> target: {target_index})") + print(f" - clean_latent_indices: {clean_latent_indices} (control_indices適用済み)") print(f" - sample_num_frames: {sample_num_frames}") + print(f" - control_indices適用: {control_indices_list}") + print(f" - control_latents数: {len(control_latents)}") + print(f" - マスク適用: clean_latents生成後") print(f" - 2x/4x無効化: True") else: # 通常モード（参照画像なし） + # control_indicesは通常モードでは使用しない all_indices = torch.arange(0, latent_window_size).unsqueeze(0) latent_indices = all_indices[:, -1:] @@ -552,9 +573,10 @@ class FramePackSingleFrameSampler: mm.soft_empty_cache() # 出力処理 - mode_info = ( - "Kisekaeichi（latent+エンベッディング）" if use_kisekaeichi else "通常" - ) + if use_kisekaeichi: + mode_info = f"Kisekaeichi（musubi-tuner完全互換、参照{len(processed_reference_latents)}個）" + else: + mode_info = "通常" print( f"=== 1フレーム生成完了 ({mode_info}モード): {generated_latents.shape} ===" ) @@ -562,10 +584,127 @@ class FramePackSingleFrameSampler: return ({"samples": generated_latents / vae_scaling_factor},) +class ReferenceLatentList: + @classmethod + def INPUT_TYPES(s): + return { + "required": {}, + "optional": { + "reference_1": ("LATENT", {"tooltip": "Reference latent 1"}), + "reference_2": ("LATENT", {"tooltip": "Reference latent 2"}), + "reference_3": ("LATENT", {"tooltip": "Reference latent 3"}), + "reference_4": ("LATENT", {"tooltip": "Reference latent 4"}), + "reference_5": ("LATENT", {"tooltip": "Reference latent 5"}), + "reference_6": ("LATENT", {"tooltip": "Reference latent 6"}), + "reference_7": ("LATENT", {"tooltip": "Reference latent 7"}), + "reference_8": ("LATENT", {"tooltip": "Reference latent 8"}), + }, + } + + RETURN_TYPES = ("REFERENCE_LATENT_LIST",) + RETURN_NAMES = ("reference_latents",) + FUNCTION = "create_list" + CATEGORY = "FramePackWrapper/References" + DESCRIPTION = "Combine up to 8 reference latents into a list for multi-reference Kisekaeichi mode" + + def create_list(self, **kwargs): + reference_list = [] + for i in range(1, 9): + ref_key = f"reference_{i}" + if ref_key in kwargs and kwargs[ref_key] is not None: + reference_list.append(kwargs[ref_key]) + + if len(reference_list) == 0: + return (None,) + + print(f"Created reference latent list with {len(reference_list)} items") + return (reference_list,) + + +class ReferenceEmbedsList: + @classmethod + def INPUT_TYPES(s): + return { + "required": {}, + "optional": { + "embeds_1": ("CLIP_VISION_OUTPUT", {"tooltip": "Reference embeds 1"}), + "embeds_2": ("CLIP_VISION_OUTPUT", {"tooltip": "Reference embeds 2"}), + "embeds_3": ("CLIP_VISION_OUTPUT", {"tooltip": "Reference embeds 3"}), + "embeds_4": ("CLIP_VISION_OUTPUT", {"tooltip": "Reference embeds 4"}), + "embeds_5": ("CLIP_VISION_OUTPUT", {"tooltip": "Reference embeds 5"}), + "embeds_6": ("CLIP_VISION_OUTPUT", {"tooltip": "Reference embeds 6"}), + "embeds_7": ("CLIP_VISION_OUTPUT", {"tooltip": "Reference embeds 7"}), + "embeds_8": ("CLIP_VISION_OUTPUT", {"tooltip": "Reference embeds 8"}), + }, + } + + RETURN_TYPES = ("REFERENCE_EMBEDS_LIST",) + RETURN_NAMES = ("reference_embeds",) + FUNCTION = "create_list" + CATEGORY = "FramePackWrapper/References" + DESCRIPTION = "Combine up to 8 reference CLIP vision embeddings into a list for multi-reference Kisekaeichi mode" + + def create_list(self, **kwargs): + embeds_list = [] + for i in range(1, 9): + embeds_key = f"embeds_{i}" + if embeds_key in kwargs and kwargs[embeds_key] is not None: + embeds_list.append(kwargs[embeds_key]) + + if len(embeds_list) == 0: + return (None,) + + print(f"Created reference embeds list with {len(embeds_list)} items") + return (embeds_list,) + + +class ReferenceMaskList: + @classmethod + def INPUT_TYPES(s): + return { + "required": {}, + "optional": { + "mask_1": ("MASK", {"tooltip": "Reference mask 1"}), + "mask_2": ("MASK", {"tooltip": "Reference mask 2"}), + "mask_3": ("MASK", {"tooltip": "Reference mask 3"}), + "mask_4": ("MASK", {"tooltip": "Reference mask 4"}), + "mask_5": ("MASK", {"tooltip": "Reference mask 5"}), + "mask_6": ("MASK", {"tooltip": "Reference mask 6"}), + "mask_7": ("MASK", {"tooltip": "Reference mask 7"}), + "mask_8": ("MASK", {"tooltip": "Reference mask 8"}), + }, + } + + RETURN_TYPES = ("REFERENCE_MASK_LIST",) + RETURN_NAMES = ("reference_masks",) + FUNCTION = "create_list" + CATEGORY = "FramePackWrapper/References" + DESCRIPTION = "Combine up to 8 reference masks into a list for multi-reference Kisekaeichi mode (optional)" + + def create_list(self, **kwargs): + mask_list = [] + for i in range(1, 9): + mask_key = f"mask_{i}" + if mask_key in kwargs and kwargs[mask_key] is not None: + mask_list.append(kwargs[mask_key]) + + if len(mask_list) == 0: + return (None,) + + print(f"Created reference mask list with {len(mask_list)} items") + return (mask_list,) + + NODE_CLASS_MAPPINGS = { "FramePackSingleFrameSampler": FramePackSingleFrameSampler, + "ReferenceLatentList": ReferenceLatentList, + "ReferenceEmbedsList": ReferenceEmbedsList, + "ReferenceMaskList": ReferenceMaskList, } NODE_DISPLAY_NAME_MAPPINGS = { "FramePackSingleFrameSampler": "FramePack Single Frame Sampler", + "ReferenceLatentList": "Reference Latent List", + "ReferenceEmbedsList": "Reference Embeds List", + "ReferenceMaskList": "Reference Mask List", } diff --git a/nodes_resize.py b/nodes_resize.py index ac6a1b8..76e266b 100644 --- a/nodes_resize.py +++ b/nodes_resize.py @@ -1,861 +1,861 @@ -import numpy as np -from PIL import Image -import logging -import math -from tqdm import tqdm -from latent_preview import prepare_callback - -import torch - -logger = logging.getLogger(__name__) - -import comfy.model_base -import comfy.model_management as mm - -from .diffusers_helper.memory import move_model_to_device_with_memory_preservation -from .diffusers_helper.pipelines.k_diffusion_hunyuan import sample_hunyuan -from .diffusers_helper.utils import crop_or_pad_yield_mask -from .nodes import HyVideoModel, HyVideoModelConfig - -# Constants -VAE_SCALING_FACTOR = 0.476986 -DEFAULT_BLEND_ALPHA = 0.5 -OVERLAP_RATIO = 0.5 # 50% オーバーラップ - - -class TileProcessor: - """50%オーバーラップを持つ2次元画像タイル処理を担当するクラス""" - - def __init__(self, tile_size: int, tile_resized: int): - self.tile_size = tile_size - self.tile_resized = tile_resized - self.upscale_factor = float(tile_resized) / float(tile_size) - self.step_size = int(tile_size * (1 - OVERLAP_RATIO)) # 50%オーバーラップのステップサイズ - - def split_image_to_tiles(self, image): - """画像を50%オーバーラップでタイルに分割し、白パディングを追加して指定サイズにリサイズ""" - pil_image = self._convert_to_pil(image) - original_width, original_height = pil_image.size - - # オーバーラップを考慮したタイル数を計算 - tiles_x = self._calculate_tile_count(original_width) - tiles_y = self._calculate_tile_count(original_height) - - self._log_split_info(original_width, original_height, tiles_x, tiles_y) - - metadata = self._create_metadata(original_width, original_height, tiles_x, tiles_y) - tiles = self._extract_overlapping_tiles(pil_image, tiles_x, tiles_y, metadata) - - return tiles, metadata - - def _calculate_tile_count(self, dimension: int) -> int: - """オーバーラップを考慮したタイル数を計算""" - if dimension <= self.tile_size: - return 1 - # 最後のタイルが完全に収まるように計算 - return math.ceil((dimension - self.tile_size) / self.step_size) + 1 - - def _convert_to_pil(self, image): - """ComfyUIのIMAGE形式からPIL形式に変換""" - if isinstance(image, torch.Tensor): - image_np = (image[0].cpu().numpy() * 255).astype(np.uint8) - return Image.fromarray(image_np) - return image - - def _log_split_info(self, width: int, height: int, tiles_x: int, tiles_y: int): - """分割情報をログ出力""" - print(f"=== 50%オーバーラップ画像分割情報 ===") - print(f"元画像サイズ: {width} × {height}") - print(f"分割タイルサイズ: {self.tile_size} × {self.tile_size}") - print(f"リサイズ後タイルサイズ: {self.tile_resized} × {self.tile_resized}") - print(f"アップスケール倍率: {self.upscale_factor:.2f}x") - print(f"ステップサイズ: {self.step_size} (50%オーバーラップ)") - print(f"タイル数: {tiles_x} × {tiles_y} = {tiles_x * tiles_y}個") - - def _create_metadata(self, width: int, height: int, tiles_x: int, tiles_y: int) -> dict: - """メタデータを作成""" - return { - "original_size": [width, height], - "tile_size": self.tile_size, - "tile_resized": self.tile_resized, - "upscale_factor": self.upscale_factor, - "step_size": self.step_size, - "overlap_ratio": OVERLAP_RATIO, - "tiles_x": tiles_x, - "tiles_y": tiles_y, - "tiles": [] - } - - def _extract_overlapping_tiles(self, pil_image: Image.Image, tiles_x: int, tiles_y: int, metadata: dict) -> list: - """50%オーバーラップでタイルを切り出す""" - tiles = [] - tile_index = 0 - - for y in range(tiles_y): - for x in range(tiles_x): - tile_tensor, tile_info = self._process_overlapping_tile( - pil_image, x, y, tile_index - ) - tiles.append(tile_tensor) - metadata["tiles"].append(tile_info) - tile_index += 1 - - return tiles - - def _process_overlapping_tile(self, pil_image: Image.Image, x: int, y: int, tile_index: int): - """50%オーバーラップを考慮した単一タイルを処理""" - # オーバーラップを考慮した切り出し位置を計算 - left = x * self.step_size - top = y * self.step_size - right = min(left + self.tile_size, pil_image.width) - bottom = min(top + self.tile_size, pil_image.height) - - # 画像端で調整（右端・下端のタイルが小さくなる場合） - if right - left < self.tile_size: - left = max(0, right - self.tile_size) - if bottom - top < self.tile_size: - top = max(0, bottom - self.tile_size) - - # タイルを切り出し - tile = pil_image.crop((left, top, right, bottom)) - tile_width, tile_height = tile.size - - # パディング処理 - padding_needed = (tile_width < self.tile_size or tile_height < self.tile_size) - if padding_needed: - tile = self._add_white_padding(tile) - - # アップスケール - upscaled_tile = tile.resize((self.tile_resized, self.tile_resized), Image.LANCZOS) - - # テンソル変換 - tile_tensor = torch.from_numpy(np.array(upscaled_tile)).float() / 255.0 - tile_tensor = tile_tensor.unsqueeze(0) # バッチ次元を追加 - - # オーバーラップ領域の情報を計算 - overlap_info = self._calculate_overlap_info(x, y, left, top, right, bottom, pil_image.size) - - # メタデータ作成 - tile_info = { - "index": tile_index, - "grid_position": [x, y], - "crop_box": [left, top, right, bottom], - "original_tile_size": [tile_width, tile_height], - "padding_needed": padding_needed, - "padding_right": self.tile_size - tile_width if padding_needed else 0, - "padding_bottom": self.tile_size - tile_height if padding_needed else 0, - "overlap_info": overlap_info - } - - print(f"タイル {tile_index}: グリッド({x},{y}) 切り出し({left},{top},{right},{bottom}) " - f"オーバーラップ:{overlap_info}") - - return tile_tensor, tile_info - - def _calculate_overlap_info(self, grid_x: int, grid_y: int, left: int, top: int, - right: int, bottom: int, image_size: tuple) -> dict: - """オーバーラップ領域の情報を計算""" - image_width, image_height = image_size - overlap_info = { - "has_left_overlap": grid_x > 0, - "has_right_overlap": right < image_width and grid_x < self._calculate_tile_count(image_width) - 1, - "has_top_overlap": grid_y > 0, - "has_bottom_overlap": bottom < image_height and grid_y < self._calculate_tile_count(image_height) - 1, - } - - # オーバーラップピクセル数を計算 - overlap_pixels = int(self.tile_size * OVERLAP_RATIO) - overlap_info.update({ - "left_overlap_pixels": overlap_pixels if overlap_info["has_left_overlap"] else 0, - "right_overlap_pixels": overlap_pixels if overlap_info["has_right_overlap"] else 0, - "top_overlap_pixels": overlap_pixels if overlap_info["has_top_overlap"] else 0, - "bottom_overlap_pixels": overlap_pixels if overlap_info["has_bottom_overlap"] else 0, - }) - - return overlap_info - - def _add_white_padding(self, tile: Image.Image) -> Image.Image: - """白パディングを追加""" - padded_tile = Image.new('RGB', (self.tile_size, self.tile_size), color='white') - padded_tile.paste(tile, (0, 0)) - return padded_tile - - -class ImageReconstructor: - """50%オーバーラップしたタイルから画像を再構成するクラス""" - - def __init__(self, metadata: dict): - self.metadata = metadata - self.upscale_factor = metadata["upscale_factor"] - self.tile_size = metadata["tile_size"] - self.step_size = metadata["step_size"] - self.overlap_ratio = metadata["overlap_ratio"] - - def reconstruct_image(self, processed_tiles: list): - """50%オーバーラップしたタイルから画像を再構成""" - final_width, final_height = self._calculate_final_size() - self._log_reconstruction_info(final_width, final_height, len(processed_tiles)) - - # 最終画像とブレンドウェイトマップを初期化 - final_image = np.zeros((final_height, final_width, 3), dtype=np.float32) - weight_map = np.zeros((final_height, final_width), dtype=np.float32) - - # 処理されたタイル画像を準備 - processed_tile_images = self._preprocess_tiles(processed_tiles) - - # オーバーラップブレンドでタイルを配置 - self._place_tiles_with_overlap_blend(final_image, weight_map, processed_tile_images) - - # 正規化して最終画像を作成 - final_image = self._normalize_blended_image(final_image, weight_map) - - return self._convert_to_tensor(Image.fromarray(final_image.astype(np.uint8))) - - def _calculate_final_size(self) -> tuple: - """最終画像サイズを計算""" - original_width, original_height = self.metadata["original_size"] - final_width = int(original_width * self.upscale_factor) - final_height = int(original_height * self.upscale_factor) - return final_width, final_height - - def _log_reconstruction_info(self, width: int, height: int, tile_count: int): - """再構成情報をログ出力""" - print(f"=== 50%オーバーラップ画像再構成情報 ===") - print(f"最終画像サイズ: {width} × {height} ({self.upscale_factor:.2f}x)") - print(f"処理タイル数: {tile_count}") - print(f"オーバーラップ比率: {self.overlap_ratio*100:.0f}%") - - def _preprocess_tiles(self, processed_tiles: list) -> list: - """タイル画像を前処理""" - processed_tile_images = [] - for i, (tile_tensor, tile_info) in enumerate(zip(processed_tiles, self.metadata["tiles"])): - tile_image = self._tensor_to_pil(tile_tensor) - - # パディング部分を除去 - if tile_info["padding_needed"]: - tile_image = self._remove_padding(tile_image, tile_info) - - processed_tile_images.append((tile_image, tile_info)) - - return processed_tile_images - - def _tensor_to_pil(self, tile_tensor) -> Image.Image: - """テンソルをPIL画像に変換""" - if isinstance(tile_tensor, torch.Tensor): - tile_np = (tile_tensor[0].cpu().numpy() * 255).astype(np.uint8) - return Image.fromarray(tile_np) - return tile_tensor - - def _remove_padding(self, tile_image: Image.Image, tile_info: dict) -> Image.Image: - """パディング部分を除去""" - original_tile_width, original_tile_height = tile_info["original_tile_size"] - content_width = int(original_tile_width * self.upscale_factor) - content_height = int(original_tile_height * self.upscale_factor) - return tile_image.crop((0, 0, content_width, content_height)) - - def _place_tiles_with_overlap_blend(self, final_image: np.ndarray, weight_map: np.ndarray, - processed_tile_images: list): - """オーバーラップ領域でブレンドしながらタイルを配置""" - for i, (tile_image, tile_info) in enumerate(processed_tile_images): - grid_x, grid_y = tile_info["grid_position"] - crop_box = tile_info["crop_box"] - overlap_info = tile_info["overlap_info"] - - # 元の切り出し位置をアップスケール倍率で拡大 - left, top, right, bottom = crop_box - paste_left = int(left * self.upscale_factor) - paste_top = int(top * self.upscale_factor) - paste_right = int(right * self.upscale_factor) - paste_bottom = int(bottom * self.upscale_factor) - - # タイル画像を numpy 配列に変換 - tile_array = np.array(tile_image, dtype=np.float32) - tile_height, tile_width = tile_array.shape[:2] - - # 最終画像の境界内に収まるように調整 - paste_right = min(paste_right, final_image.shape[1]) - paste_bottom = min(paste_bottom, final_image.shape[0]) - actual_width = paste_right - paste_left - actual_height = paste_bottom - paste_top - - # タイルサイズが配置領域と異なる場合は調整 - if actual_width != tile_width or actual_height != tile_height: - if actual_width > 0 and actual_height > 0: - tile_array = tile_array[:actual_height, :actual_width] - else: - print(f"警告: タイル {i} の配置領域が無効です") - continue - - # ブレンド用のウェイトマスクを作成 - tile_weight = self._create_blend_weight_mask( - tile_array.shape[:2], overlap_info, crop_box, self.upscale_factor - ) - - # 加算ブレンド方式で配置 - final_image[paste_top:paste_bottom, paste_left:paste_right] += \ - tile_array * tile_weight[..., np.newaxis] - weight_map[paste_top:paste_bottom, paste_left:paste_right] += tile_weight - - print(f"タイル {i} 配置: 元座標({left},{top},{right},{bottom}) -> " - f"配置座標({paste_left},{paste_top},{paste_right},{paste_bottom}) " - f"サイズ({actual_width}×{actual_height})") - - def _create_blend_weight_mask(self, tile_shape: tuple, overlap_info: dict, - crop_box: list, upscale_factor: float) -> np.ndarray: - """オーバーラップ領域用のウェイトマスクを作成""" - height, width = tile_shape - weight_mask = np.ones((height, width), dtype=np.float32) - - # オーバーラップピクセル数を計算（アップスケール後） - overlap_pixels = int(self.tile_size * self.overlap_ratio * upscale_factor) - - # 実際の切り出しボックスを使ってオーバーラップ領域を正確に計算 - left, top, right, bottom = crop_box - - # 左端のフェード（左にオーバーラップがある場合） - if overlap_info["has_left_overlap"]: - # 実際のオーバーラップ開始位置を計算 - overlap_start_x = left % self.step_size - if overlap_start_x > 0: - fade_width = min(int(overlap_start_x * upscale_factor), width) - if fade_width > 0: - fade_weights = np.linspace(0, 1, fade_width) - weight_mask[:, :fade_width] *= fade_weights[np.newaxis, :] - - # 右端のフェード（右にオーバーラップがある場合） - if overlap_info["has_right_overlap"]: - # 右端のオーバーラップ幅を計算 - next_tile_start = (left // self.step_size + 1) * self.step_size - if right > next_tile_start: - overlap_width = right - next_tile_start - fade_width = min(int(overlap_width * upscale_factor), width) - if fade_width > 0: - fade_weights = np.linspace(1, 0, fade_width) - weight_mask[:, -fade_width:] *= fade_weights[np.newaxis, :] - - # 上端のフェード（上にオーバーラップがある場合） - if overlap_info["has_top_overlap"]: - # 実際のオーバーラップ開始位置を計算 - overlap_start_y = top % self.step_size - if overlap_start_y > 0: - fade_height = min(int(overlap_start_y * upscale_factor), height) - if fade_height > 0: - fade_weights = np.linspace(0, 1, fade_height) - weight_mask[:fade_height, :] *= fade_weights[:, np.newaxis] - - # 下端のフェード（下にオーバーラップがある場合） - if overlap_info["has_bottom_overlap"]: - # 下端のオーバーラップ高さを計算 - next_tile_start = (top // self.step_size + 1) * self.step_size - if bottom > next_tile_start: - overlap_height = bottom - next_tile_start - fade_height = min(int(overlap_height * upscale_factor), height) - if fade_height > 0: - fade_weights = np.linspace(1, 0, fade_height) - weight_mask[-fade_height:, :] *= fade_weights[:, np.newaxis] - - return weight_mask - - def _normalize_blended_image(self, final_image: np.ndarray, weight_map: np.ndarray) -> np.ndarray: - """ブレンドされた画像を正規化""" - # ゼロ除算を避けるため、重みが0の部分には小さな値を設定 - weight_map = np.maximum(weight_map, 1e-8) - - # 正規化 - final_image = final_image / weight_map[..., np.newaxis] - - # 値の範囲を0-255にクランプ - final_image = np.clip(final_image, 0, 255) - - return final_image - - def _convert_to_tensor(self, final_image: Image.Image) -> torch.Tensor: - """PIL画像をComfyUI形式のテンソルに変換""" - final_tensor = torch.from_numpy(np.array(final_image)).float() / 255.0 - return final_tensor.unsqueeze(0) # バッチ次元を追加 - - -class VAEProcessor: - """VAE処理を担当するクラス""" - - @staticmethod - def tiled_decode(vae, latents: torch.Tensor, tile_size: int = 256, - overlap: int = 64, temporal_size: int = 64, - temporal_overlap: int = 8) -> torch.Tensor: - """タイル化VAEデコード""" - B, C, T, H, W = latents.shape - device = latents.device - dtype = latents.dtype - - # 出力テンソルを初期化 (VAEは8倍アップスケール) - output_h, output_w = H * 8, W * 8 - output = torch.zeros(B, T, output_h, output_w, 3, device=device, dtype=dtype) - - # タイル分割数を計算 - h_tiles, w_tiles, t_tiles = VAEProcessor._calculate_tile_counts( - H, W, T, tile_size, overlap, temporal_size, temporal_overlap - ) - - print(f"タイル化デコード: H={h_tiles}, W={w_tiles}, T={t_tiles}") - - # タイル処理実行 - VAEProcessor._process_tiles( - vae, latents, output, h_tiles, w_tiles, t_tiles, - tile_size, overlap, temporal_size, temporal_overlap, device, dtype - ) - - return output - - @staticmethod - def _calculate_tile_counts(H: int, W: int, T: int, tile_size: int, - overlap: int, temporal_size: int, temporal_overlap: int) -> tuple: - """タイル分割数を計算""" - h_tiles = max(1, (H - overlap) // (tile_size - overlap) + - (1 if (H - overlap) % (tile_size - overlap) > 0 else 0)) - w_tiles = max(1, (W - overlap) // (tile_size - overlap) + - (1 if (W - overlap) % (tile_size - overlap) > 0 else 0)) - t_tiles = max(1, (T - temporal_overlap) // (temporal_size - temporal_overlap) + - (1 if (T - temporal_overlap) % (temporal_size - temporal_overlap) > 0 else 0)) - return h_tiles, w_tiles, t_tiles - - @staticmethod - def _process_tiles(vae, latents: torch.Tensor, output: torch.Tensor, - h_tiles: int, w_tiles: int, t_tiles: int, - tile_size: int, overlap: int, temporal_size: int, - temporal_overlap: int, device, dtype): - """実際のタイル処理を実行""" - B, C, T, H, W = latents.shape - - for t_idx in range(t_tiles): - t_start, t_end = VAEProcessor._calculate_range( - t_idx, temporal_size, temporal_overlap, T - ) - - for h_idx in range(h_tiles): - h_start, h_end = VAEProcessor._calculate_range( - h_idx, tile_size, overlap, H - ) - - for w_idx in range(w_tiles): - w_start, w_end = VAEProcessor._calculate_range( - w_idx, tile_size, overlap, W - ) - - # タイルを切り出してデコード - tile_latent = latents[:, :, t_start:t_end, h_start:h_end, w_start:w_end] - - try: - with torch.autocast(device_type=mm.get_autocast_device(device), - dtype=dtype, enabled=True): - decoded_tile = vae.decode(tile_latent) - - # 出力に配置（オーバーラップ処理含む） - VAEProcessor._place_decoded_tile( - output, decoded_tile, t_start, t_end, - h_start, h_end, w_start, w_end, h_idx, w_idx - ) - - except Exception as e: - print(f"タイル({t_idx},{h_idx},{w_idx})のデコードに失敗: {e}") - continue - - @staticmethod - def _calculate_range(idx: int, size: int, overlap: int, max_size: int) -> tuple: - """範囲を計算""" - start = idx * (size - overlap) - end = min(start + size, max_size) - start = max(0, end - size) # 境界調整 - return start, end - - @staticmethod - def _place_decoded_tile(output: torch.Tensor, decoded_tile: torch.Tensor, - t_start: int, t_end: int, h_start: int, h_end: int, - w_start: int, w_end: int, h_idx: int, w_idx: int): - """デコードされたタイルを出力に配置""" - out_t_start, out_t_end = t_start, t_end - out_h_start, out_h_end = h_start * 8, h_end * 8 - out_w_start, out_w_end = w_start * 8, w_end * 8 - - # オーバーラップ処理 - if h_idx > 0 or w_idx > 0: - existing = output[:, out_t_start:out_t_end, out_h_start:out_h_end, out_w_start:out_w_end] - if existing.sum() > 0: # 既存データがある場合のみブレンド - output[:, out_t_start:out_t_end, out_h_start:out_h_end, out_w_start:out_w_end] = \ - DEFAULT_BLEND_ALPHA * existing + (1 - DEFAULT_BLEND_ALPHA) * decoded_tile - else: - output[:, out_t_start:out_t_end, out_h_start:out_h_end, out_w_start:out_w_end] = decoded_tile - else: - # 最初のタイルはそのまま配置 - output[:, out_t_start:out_t_end, out_h_start:out_h_end, out_w_start:out_w_end] = decoded_tile - - -class FramePackSingleFrameResizeSampler: - """メインのサンプラークラス""" - - @classmethod - def INPUT_TYPES(s): - return { - "required": { - "model": ("FramePackMODEL",), - "vae": ("VAE",), - "positive": ("CONDITIONING",), - "negative": ("CONDITIONING",), - "input_image": ("IMAGE", {"tooltip": "Input image to be upscaled"}), - "steps": ("INT", {"default": 30, "min": 1}), - "use_teacache": ("BOOLEAN", {"default": True, "tooltip": "Use teacache for faster sampling."}), - "teacache_rel_l1_thresh": ("FLOAT", {"default": 0.15, "min": 0.0, "max": 1.0, "step": 0.01, "tooltip": "The threshold for the relative L1 loss."}), - "cfg": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 30.0, "step": 0.01}), - "guidance_scale": ("FLOAT", {"default": 10.0, "min": 0.0, "max": 32.0, "step": 0.01}), - "shift": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1000.0, "step": 0.01}), - "seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}), - "latent_window_size": ("INT", {"default": 9, "min": 1, "max": 33, "step": 1, "tooltip": "The size of the latent window to use for sampling."}), - "gpu_memory_preservation": ("FLOAT", {"default": 6.0, "min": 0.0, "max": 128.0, "step": 0.1, "tooltip": "The amount of GPU memory to preserve."}), - "sampler": (["unipc_bh1", "unipc_bh2"], {"default": 'unipc_bh1'}), - "tile_size": ("INT", {"default": 256, "min": 256, "max": 1024, "step": 64, "tooltip": "Size of each tile for initial split"}), - "tile_resized": ("INT", {"default": 1024, "min": 512, "max": 2048, "step": 64, "tooltip": "Size to resize each tile to before processing"}), - "vae_tile_size": ("INT", {"default": 256, "min": 64, "max": 512, "step": 64, "tooltip": "Size of each VAE tile for decode"}), - "vae_overlap": ("INT", {"default": 64, "min": 0, "max": 256, "step": 8, "tooltip": "Overlap between VAE tiles"}), - "vae_temporal_size": ("INT", {"default": 64, "min": 1, "max": 128, "step": 1, "tooltip": "Temporal size for VAE tiled decode"}), - "vae_temporal_overlap": ("INT", {"default": 8, "min": 0, "max": 32, "step": 1, "tooltip": "Temporal overlap for VAE tiled decode"}), - }, - "optional": { - "image_embeds": ("CLIP_VISION_OUTPUT", ), - "denoise_strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.01}), - } - } - - RETURN_TYPES = ("IMAGE", ) - RETURN_NAMES = ("upscaled_image",) - FUNCTION = "process" - CATEGORY = "FramePackWrapper" - DESCRIPTION = "Variable ratio image upscaling using 50% overlapping tile-based FramePack single frame sampling" - - def process(self, model, vae, shift, positive, negative, input_image, latent_window_size, - use_teacache, teacache_rel_l1_thresh, steps, cfg, guidance_scale, seed, - sampler, gpu_memory_preservation, tile_size=512, tile_resized=1024, - vae_tile_size=256, vae_overlap=64, vae_temporal_size=64, - vae_temporal_overlap=8, image_embeds=None, denoise_strength=1.0): - - upscale_factor = float(tile_resized) / float(tile_size) - self._log_process_start(upscale_factor, vae_tile_size, vae_overlap, - vae_temporal_size, vae_temporal_overlap, seed) - - # デバイス設定とモデル準備 - transformer, base_dtype, device, offload_device = self._setup_model(model) - - # 1. 画像を50%オーバーラップでタイルに分割 - tile_processor = TileProcessor(tile_size, tile_resized) - tiles, metadata = tile_processor.split_image_to_tiles(input_image) - - # 2. テキストエンベッディング準備 - text_embeddings = self._prepare_text_embeddings( - positive, negative, cfg, device, base_dtype - ) - - # 3. 画像エンベッディング準備 - image_encoder_embeddings = self._prepare_image_embeddings( - image_embeds, device, base_dtype - ) - - # 4. ComfyUIセットアップ - callback = self._setup_comfy_ui(transformer, device, steps, base_dtype) - - # 5. モデル準備 - self._prepare_model(transformer, device, gpu_memory_preservation, - use_teacache, steps, teacache_rel_l1_thresh) - - # 6. 各タイルを処理 - processed_tiles = self._process_all_tiles( - tiles, vae, transformer, sampler, denoise_strength, latent_window_size, - cfg, guidance_scale, shift, steps, seed, device, base_dtype, - text_embeddings, image_encoder_embeddings, callback, - vae_tile_size, vae_overlap, vae_temporal_size, vae_temporal_overlap - ) - - # 7. 50%オーバーラップブレンドで最終画像を再構成 - reconstructor = ImageReconstructor(metadata) - final_image = reconstructor.reconstruct_image(processed_tiles) - - # クリーンアップ - self._cleanup(transformer, offload_device) - - print(f"=== 50%オーバーラップ {upscale_factor:.2f}xアップスケール完了: {final_image.shape} ===") - return (final_image,) - - def _log_process_start(self, upscale_factor: float, vae_tile_size: int, - vae_overlap: int, vae_temporal_size: int, - vae_temporal_overlap: int, seed: int): - """処理開始のログ出力""" - print(f"=== FramePack 50%オーバーラップ Resize Sampler 開始 ({upscale_factor:.2f}x) ===") - print(f"VAE Tile設定: size={vae_tile_size}, overlap={vae_overlap}, " - f"temporal_size={vae_temporal_size}, temporal_overlap={vae_temporal_overlap}") - print(f"使用シード: {seed} (全タイル共通)") - print(f"オーバーラップ比率: {OVERLAP_RATIO*100:.0f}%") - - def _setup_model(self, model: dict) -> tuple: - """モデルとデバイスのセットアップ""" - transformer = model["transformer"] - base_dtype = model["dtype"] - device = mm.get_torch_device() - offload_device = mm.unet_offload_device() - - mm.unload_all_models() - mm.cleanup_models() - mm.soft_empty_cache() - - return transformer, base_dtype, device, offload_device - - def _prepare_text_embeddings(self, positive, negative, cfg: float, - device, base_dtype) -> dict: - """テキストエンベッディングを準備""" - llama_vec = positive[0][0].to(device, base_dtype) - clip_l_pooler = positive[0][1]["pooled_output"].to(device, base_dtype) - - if not math.isclose(cfg, 1.0): - llama_vec_n = negative[0][0].to(device, base_dtype) - clip_l_pooler_n = negative[0][1]["pooled_output"].to(device, base_dtype) - else: - llama_vec_n = torch.zeros_like(llama_vec, device=device) - clip_l_pooler_n = torch.zeros_like(clip_l_pooler, device=device) - - llama_vec, llama_attention_mask = crop_or_pad_yield_mask(llama_vec, length=512) - llama_vec_n, llama_attention_mask_n = crop_or_pad_yield_mask(llama_vec_n, length=512) - - return { - "llama_vec": llama_vec, - "llama_attention_mask": llama_attention_mask, - "clip_l_pooler": clip_l_pooler, - "llama_vec_n": llama_vec_n, - "llama_attention_mask_n": llama_attention_mask_n, - "clip_l_pooler_n": clip_l_pooler_n - } - - def _prepare_image_embeddings(self, image_embeds, device, base_dtype): - """画像エンベッディングを準備""" - if image_embeds is not None: - return image_embeds["last_hidden_state"].to(device, base_dtype) - return None - - def _setup_comfy_ui(self, transformer, device, steps, base_dtype): - """ComfyUIセットアップ""" - comfy_model = HyVideoModel( - HyVideoModelConfig(base_dtype), - model_type=comfy.model_base.ModelType.FLOW, - device=device, - ) - patcher = comfy.model_patcher.ModelPatcher(comfy_model, device, torch.device("cpu")) - return prepare_callback(patcher, steps) - - def _prepare_model(self, transformer, device, gpu_memory_preservation: float, - use_teacache: bool, steps: int, teacache_rel_l1_thresh: float): - """モデルの準備（GPU移動とTeaCache設定）""" - # モデルをGPUに移動 - move_model_to_device_with_memory_preservation( - transformer, target_device=device, - preserved_memory_gb=gpu_memory_preservation - ) - - # TeaCacheの設定 - if use_teacache: - transformer.initialize_teacache( - enable_teacache=True, num_steps=steps, - rel_l1_thresh=teacache_rel_l1_thresh - ) - else: - transformer.initialize_teacache(enable_teacache=False) - - def _process_all_tiles(self, tiles: list, vae, transformer, sampler: str, - denoise_strength: float, latent_window_size: int, - cfg: float, guidance_scale: float, shift: float, - steps: int, seed: int, device, base_dtype, - text_embeddings: dict, image_encoder_embeddings, - callback, vae_tile_size: int, vae_overlap: int, - vae_temporal_size: int, vae_temporal_overlap: int) -> list: - """全てのタイルを処理""" - processed_tiles = [] - - print(f"=== {len(tiles)}個のタイルを順次処理（50%オーバーラップ、各タイルで同一シード使用） ===") - - for i, tile_tensor in enumerate(tqdm(tiles, desc="オーバーラップタイル処理")): - print(f"\n--- タイル {i+1}/{len(tiles)} 処理中（シード: {seed}） ---") - - # 各タイルで同じシードを使用するためにジェネレータをリセット - rnd = torch.Generator("cpu").manual_seed(seed) - - processed_tile = self._process_single_tile( - tile_tensor, vae, transformer, sampler, denoise_strength, - latent_window_size, cfg, guidance_scale, shift, steps, rnd, - device, base_dtype, text_embeddings, image_encoder_embeddings, - callback, vae_tile_size, vae_overlap, vae_temporal_size, vae_temporal_overlap - ) - - processed_tiles.append(processed_tile) - print(f"オーバーラップタイル {i+1} 完了: {processed_tile.shape}") - - return processed_tiles - - def _process_single_tile(self, tile_tensor: torch.Tensor, vae, transformer, - sampler: str, denoise_strength: float, latent_window_size: int, - cfg: float, guidance_scale: float, shift: float, - steps: int, rnd: torch.Generator, device, base_dtype, - text_embeddings: dict, image_encoder_embeddings, - callback, vae_tile_size: int, vae_overlap: int, - vae_temporal_size: int, vae_temporal_overlap: int) -> torch.Tensor: - """単一タイルを処理""" - tile_tensor = tile_tensor.to(device) - - with torch.autocast(device_type=mm.get_autocast_device(device), - dtype=base_dtype, enabled=True): - # VAEエンコード - start_latent = self._encode_tile(vae, tile_tensor) - - # FramePack推論のためのインデックス設定 - latent_indices, clean_latents, clean_latent_indices = self._setup_latent_indices( - start_latent, latent_window_size - ) - - # FramePack推論実行 - generated_latents = self._run_framepack_inference( - transformer, sampler, denoise_strength, start_latent.shape, - cfg, guidance_scale, shift, steps, rnd, device, base_dtype, - text_embeddings, image_encoder_embeddings, latent_indices, - clean_latents, clean_latent_indices, callback - ) - - # VAEタイル化デコード - decoded_tile = self._decode_tile( - vae, generated_latents, vae_tile_size, vae_overlap, - vae_temporal_size, vae_temporal_overlap - ) - - # 時間次元を除去して画像形式に変換 [B, T, H, W, C] -> [B, H, W, C] - if len(decoded_tile.shape) == 5: - decoded_tile = decoded_tile.squeeze(1) - - return decoded_tile - - def _encode_tile(self, vae, tile_tensor: torch.Tensor) -> torch.Tensor: - """タイルをVAEエンコード""" - start_latent = vae.encode(tile_tensor) - start_latent = start_latent * VAE_SCALING_FACTOR - - print(f"エンコード後latent: {start_latent.shape}") - - # latentを5次元に変換 [B, C, T, H, W] - if len(start_latent.shape) == 4: - start_latent = start_latent.unsqueeze(2) # T次元を追加 - - return start_latent - - def _setup_latent_indices(self, start_latent: torch.Tensor, - latent_window_size: int) -> tuple: - """FramePack推論のためのインデックス設定""" - B, C, T, H, W = start_latent.shape - - # 1フレーム推論のためのインデックス設定 - sample_num_frames = 1 - latent_padding = 0 - latent_padding_size = latent_padding * latent_window_size - - indices = torch.arange(0, sum([1, latent_padding_size, latent_window_size, 1, 2, 16])).unsqueeze(0) - - if latent_padding_size == 0: - clean_latent_indices_pre = indices[:, 0:1] - latent_indices = indices[:, 1:1+latent_window_size] - clean_latent_indices_post = indices[:, 1+latent_window_size:2+latent_window_size] - clean_latent_2x_indices = indices[:, 2+latent_window_size:4+latent_window_size] - clean_latent_4x_indices = indices[:, 4+latent_window_size:20+latent_window_size] - blank_indices = torch.empty((1, 0), dtype=torch.long) - else: - split_sizes = [1, latent_padding_size, latent_window_size, 1, 2, 16] - clean_latent_indices_pre, blank_indices, latent_indices, clean_latent_indices_post, clean_latent_2x_indices, clean_latent_4x_indices = indices.split(split_sizes, dim=1) - - # 1フレーム推論設定 - all_indices = torch.arange(0, latent_window_size).unsqueeze(0) - latent_indices = all_indices[:, -1:] - - clean_latents_pre = start_latent.to(torch.float32).cpu() - clean_latents_post = torch.zeros_like(clean_latents_pre) - clean_latents = torch.cat([clean_latents_pre, clean_latents_post], dim=2) - - clean_latent_indices = torch.tensor([[0]], dtype=clean_latent_indices_pre.dtype, device=clean_latent_indices_pre.device) - clean_latents = clean_latents[:, :, :1, :, :] - - return latent_indices, clean_latents, clean_latent_indices - - def _run_framepack_inference(self, transformer, sampler: str, denoise_strength: float, - latent_shape: tuple, cfg: float, guidance_scale: float, - shift: float, steps: int, rnd: torch.Generator, - device, base_dtype, text_embeddings: dict, - image_encoder_embeddings, latent_indices: torch.Tensor, - clean_latents: torch.Tensor, clean_latent_indices: torch.Tensor, - callback) -> torch.Tensor: - """FramePack推論を実行""" - B, C, T, H, W = latent_shape - sample_num_frames = 1 - - generated_latents = sample_hunyuan( - transformer=transformer, - sampler=sampler, - initial_latent=None, - strength=denoise_strength, - width=W * 8, - height=H * 8, - frames=sample_num_frames, - real_guidance_scale=cfg, - distilled_guidance_scale=guidance_scale, - guidance_rescale=0, - shift=shift if shift != 0 else None, - num_inference_steps=steps, - generator=rnd, - prompt_embeds=text_embeddings["llama_vec"], - prompt_embeds_mask=text_embeddings["llama_attention_mask"], - prompt_poolers=text_embeddings["clip_l_pooler"], - negative_prompt_embeds=text_embeddings["llama_vec_n"], - negative_prompt_embeds_mask=text_embeddings["llama_attention_mask_n"], - negative_prompt_poolers=text_embeddings["clip_l_pooler_n"], - device=device, - dtype=base_dtype, - image_embeddings=image_encoder_embeddings, - latent_indices=latent_indices, - clean_latents=clean_latents, - clean_latent_indices=clean_latent_indices, - clean_latents_2x=None, # 2x, 4x latentsを無効化 - clean_latent_2x_indices=None, - clean_latents_4x=None, - clean_latent_4x_indices=None, - callback=callback, - ) - - return generated_latents - - def _decode_tile(self, vae, generated_latents: torch.Tensor, vae_tile_size: int, - vae_overlap: int, vae_temporal_size: int, vae_temporal_overlap: int) -> torch.Tensor: - """タイルをVAEデコード""" - generated_latents = generated_latents / VAE_SCALING_FACTOR - print(f"デコード前latent: {generated_latents.shape}") - - decoded_tile = VAEProcessor.tiled_decode( - vae, generated_latents, - tile_size=vae_tile_size, - overlap=vae_overlap, - temporal_size=vae_temporal_size, - temporal_overlap=vae_temporal_overlap - ) - - return decoded_tile - - def _cleanup(self, transformer, offload_device): - """リソースのクリーンアップ""" - transformer.to(offload_device) - mm.soft_empty_cache() - - -# ノードクラスマッピング -NODE_CLASS_MAPPINGS = { - "FramePackSingleFrameResizeSampler": FramePackSingleFrameResizeSampler, -} - -NODE_DISPLAY_NAME_MAPPINGS = { - "FramePackSingleFrameResizeSampler": "FramePack Single Frame Resize Sampler (50% Overlap)", +import numpy as np +from PIL import Image +import logging +import math +from tqdm import tqdm +from latent_preview import prepare_callback + +import torch + +logger = logging.getLogger(__name__) + +import comfy.model_base +import comfy.model_management as mm + +from .diffusers_helper.memory import move_model_to_device_with_memory_preservation +from .diffusers_helper.pipelines.k_diffusion_hunyuan import sample_hunyuan +from .diffusers_helper.utils import crop_or_pad_yield_mask +from .nodes import HyVideoModel, HyVideoModelConfig + +# Constants +VAE_SCALING_FACTOR = 0.476986 +DEFAULT_BLEND_ALPHA = 0.5 +OVERLAP_RATIO = 0.5 # 50% オーバーラップ + + +class TileProcessor: + """50%オーバーラップを持つ2次元画像タイル処理を担当するクラス""" + + def __init__(self, tile_size: int, tile_resized: int): + self.tile_size = tile_size + self.tile_resized = tile_resized + self.upscale_factor = float(tile_resized) / float(tile_size) + self.step_size = int(tile_size * (1 - OVERLAP_RATIO)) # 50%オーバーラップのステップサイズ + + def split_image_to_tiles(self, image): + """画像を50%オーバーラップでタイルに分割し、白パディングを追加して指定サイズにリサイズ""" + pil_image = self._convert_to_pil(image) + original_width, original_height = pil_image.size + + # オーバーラップを考慮したタイル数を計算 + tiles_x = self._calculate_tile_count(original_width) + tiles_y = self._calculate_tile_count(original_height) + + self._log_split_info(original_width, original_height, tiles_x, tiles_y) + + metadata = self._create_metadata(original_width, original_height, tiles_x, tiles_y) + tiles = self._extract_overlapping_tiles(pil_image, tiles_x, tiles_y, metadata) + + return tiles, metadata + + def _calculate_tile_count(self, dimension: int) -> int: + """オーバーラップを考慮したタイル数を計算""" + if dimension <= self.tile_size: + return 1 + # 最後のタイルが完全に収まるように計算 + return math.ceil((dimension - self.tile_size) / self.step_size) + 1 + + def _convert_to_pil(self, image): + """ComfyUIのIMAGE形式からPIL形式に変換""" + if isinstance(image, torch.Tensor): + image_np = (image[0].cpu().numpy() * 255).astype(np.uint8) + return Image.fromarray(image_np) + return image + + def _log_split_info(self, width: int, height: int, tiles_x: int, tiles_y: int): + """分割情報をログ出力""" + print(f"=== 50%オーバーラップ画像分割情報 ===") + print(f"元画像サイズ: {width} × {height}") + print(f"分割タイルサイズ: {self.tile_size} × {self.tile_size}") + print(f"リサイズ後タイルサイズ: {self.tile_resized} × {self.tile_resized}") + print(f"アップスケール倍率: {self.upscale_factor:.2f}x") + print(f"ステップサイズ: {self.step_size} (50%オーバーラップ)") + print(f"タイル数: {tiles_x} × {tiles_y} = {tiles_x * tiles_y}個") + + def _create_metadata(self, width: int, height: int, tiles_x: int, tiles_y: int) -> dict: + """メタデータを作成""" + return { + "original_size": [width, height], + "tile_size": self.tile_size, + "tile_resized": self.tile_resized, + "upscale_factor": self.upscale_factor, + "step_size": self.step_size, + "overlap_ratio": OVERLAP_RATIO, + "tiles_x": tiles_x, + "tiles_y": tiles_y, + "tiles": [] + } + + def _extract_overlapping_tiles(self, pil_image: Image.Image, tiles_x: int, tiles_y: int, metadata: dict) -> list: + """50%オーバーラップでタイルを切り出す""" + tiles = [] + tile_index = 0 + + for y in range(tiles_y): + for x in range(tiles_x): + tile_tensor, tile_info = self._process_overlapping_tile( + pil_image, x, y, tile_index + ) + tiles.append(tile_tensor) + metadata["tiles"].append(tile_info) + tile_index += 1 + + return tiles + + def _process_overlapping_tile(self, pil_image: Image.Image, x: int, y: int, tile_index: int): + """50%オーバーラップを考慮した単一タイルを処理""" + # オーバーラップを考慮した切り出し位置を計算 + left = x * self.step_size + top = y * self.step_size + right = min(left + self.tile_size, pil_image.width) + bottom = min(top + self.tile_size, pil_image.height) + + # 画像端で調整（右端・下端のタイルが小さくなる場合） + if right - left < self.tile_size: + left = max(0, right - self.tile_size) + if bottom - top < self.tile_size: + top = max(0, bottom - self.tile_size) + + # タイルを切り出し + tile = pil_image.crop((left, top, right, bottom)) + tile_width, tile_height = tile.size + + # パディング処理 + padding_needed = (tile_width < self.tile_size or tile_height < self.tile_size) + if padding_needed: + tile = self._add_white_padding(tile) + + # アップスケール + upscaled_tile = tile.resize((self.tile_resized, self.tile_resized), Image.LANCZOS) + + # テンソル変換 + tile_tensor = torch.from_numpy(np.array(upscaled_tile)).float() / 255.0 + tile_tensor = tile_tensor.unsqueeze(0) # バッチ次元を追加 + + # オーバーラップ領域の情報を計算 + overlap_info = self._calculate_overlap_info(x, y, left, top, right, bottom, pil_image.size) + + # メタデータ作成 + tile_info = { + "index": tile_index, + "grid_position": [x, y], + "crop_box": [left, top, right, bottom], + "original_tile_size": [tile_width, tile_height], + "padding_needed": padding_needed, + "padding_right": self.tile_size - tile_width if padding_needed else 0, + "padding_bottom": self.tile_size - tile_height if padding_needed else 0, + "overlap_info": overlap_info + } + + print(f"タイル {tile_index}: グリッド({x},{y}) 切り出し({left},{top},{right},{bottom}) " + f"オーバーラップ:{overlap_info}") + + return tile_tensor, tile_info + + def _calculate_overlap_info(self, grid_x: int, grid_y: int, left: int, top: int, + right: int, bottom: int, image_size: tuple) -> dict: + """オーバーラップ領域の情報を計算""" + image_width, image_height = image_size + overlap_info = { + "has_left_overlap": grid_x > 0, + "has_right_overlap": right < image_width and grid_x < self._calculate_tile_count(image_width) - 1, + "has_top_overlap": grid_y > 0, + "has_bottom_overlap": bottom < image_height and grid_y < self._calculate_tile_count(image_height) - 1, + } + + # オーバーラップピクセル数を計算 + overlap_pixels = int(self.tile_size * OVERLAP_RATIO) + overlap_info.update({ + "left_overlap_pixels": overlap_pixels if overlap_info["has_left_overlap"] else 0, + "right_overlap_pixels": overlap_pixels if overlap_info["has_right_overlap"] else 0, + "top_overlap_pixels": overlap_pixels if overlap_info["has_top_overlap"] else 0, + "bottom_overlap_pixels": overlap_pixels if overlap_info["has_bottom_overlap"] else 0, + }) + + return overlap_info + + def _add_white_padding(self, tile: Image.Image) -> Image.Image: + """白パディングを追加""" + padded_tile = Image.new('RGB', (self.tile_size, self.tile_size), color='white') + padded_tile.paste(tile, (0, 0)) + return padded_tile + + +class ImageReconstructor: + """50%オーバーラップしたタイルから画像を再構成するクラス""" + + def __init__(self, metadata: dict): + self.metadata = metadata + self.upscale_factor = metadata["upscale_factor"] + self.tile_size = metadata["tile_size"] + self.step_size = metadata["step_size"] + self.overlap_ratio = metadata["overlap_ratio"] + + def reconstruct_image(self, processed_tiles: list): + """50%オーバーラップしたタイルから画像を再構成""" + final_width, final_height = self._calculate_final_size() + self._log_reconstruction_info(final_width, final_height, len(processed_tiles)) + + # 最終画像とブレンドウェイトマップを初期化 + final_image = np.zeros((final_height, final_width, 3), dtype=np.float32) + weight_map = np.zeros((final_height, final_width), dtype=np.float32) + + # 処理されたタイル画像を準備 + processed_tile_images = self._preprocess_tiles(processed_tiles) + + # オーバーラップブレンドでタイルを配置 + self._place_tiles_with_overlap_blend(final_image, weight_map, processed_tile_images) + + # 正規化して最終画像を作成 + final_image = self._normalize_blended_image(final_image, weight_map) + + return self._convert_to_tensor(Image.fromarray(final_image.astype(np.uint8))) + + def _calculate_final_size(self) -> tuple: + """最終画像サイズを計算""" + original_width, original_height = self.metadata["original_size"] + final_width = int(original_width * self.upscale_factor) + final_height = int(original_height * self.upscale_factor) + return final_width, final_height + + def _log_reconstruction_info(self, width: int, height: int, tile_count: int): + """再構成情報をログ出力""" + print(f"=== 50%オーバーラップ画像再構成情報 ===") + print(f"最終画像サイズ: {width} × {height} ({self.upscale_factor:.2f}x)") + print(f"処理タイル数: {tile_count}") + print(f"オーバーラップ比率: {self.overlap_ratio*100:.0f}%") + + def _preprocess_tiles(self, processed_tiles: list) -> list: + """タイル画像を前処理""" + processed_tile_images = [] + for i, (tile_tensor, tile_info) in enumerate(zip(processed_tiles, self.metadata["tiles"])): + tile_image = self._tensor_to_pil(tile_tensor) + + # パディング部分を除去 + if tile_info["padding_needed"]: + tile_image = self._remove_padding(tile_image, tile_info) + + processed_tile_images.append((tile_image, tile_info)) + + return processed_tile_images + + def _tensor_to_pil(self, tile_tensor) -> Image.Image: + """テンソルをPIL画像に変換""" + if isinstance(tile_tensor, torch.Tensor): + tile_np = (tile_tensor[0].cpu().numpy() * 255).astype(np.uint8) + return Image.fromarray(tile_np) + return tile_tensor + + def _remove_padding(self, tile_image: Image.Image, tile_info: dict) -> Image.Image: + """パディング部分を除去""" + original_tile_width, original_tile_height = tile_info["original_tile_size"] + content_width = int(original_tile_width * self.upscale_factor) + content_height = int(original_tile_height * self.upscale_factor) + return tile_image.crop((0, 0, content_width, content_height)) + + def _place_tiles_with_overlap_blend(self, final_image: np.ndarray, weight_map: np.ndarray, + processed_tile_images: list): + """オーバーラップ領域でブレンドしながらタイルを配置""" + for i, (tile_image, tile_info) in enumerate(processed_tile_images): + grid_x, grid_y = tile_info["grid_position"] + crop_box = tile_info["crop_box"] + overlap_info = tile_info["overlap_info"] + + # 元の切り出し位置をアップスケール倍率で拡大 + left, top, right, bottom = crop_box + paste_left = int(left * self.upscale_factor) + paste_top = int(top * self.upscale_factor) + paste_right = int(right * self.upscale_factor) + paste_bottom = int(bottom * self.upscale_factor) + + # タイル画像を numpy 配列に変換 + tile_array = np.array(tile_image, dtype=np.float32) + tile_height, tile_width = tile_array.shape[:2] + + # 最終画像の境界内に収まるように調整 + paste_right = min(paste_right, final_image.shape[1]) + paste_bottom = min(paste_bottom, final_image.shape[0]) + actual_width = paste_right - paste_left + actual_height = paste_bottom - paste_top + + # タイルサイズが配置領域と異なる場合は調整 + if actual_width != tile_width or actual_height != tile_height: + if actual_width > 0 and actual_height > 0: + tile_array = tile_array[:actual_height, :actual_width] + else: + print(f"警告: タイル {i} の配置領域が無効です") + continue + + # ブレンド用のウェイトマスクを作成 + tile_weight = self._create_blend_weight_mask( + tile_array.shape[:2], overlap_info, crop_box, self.upscale_factor + ) + + # 加算ブレンド方式で配置 + final_image[paste_top:paste_bottom, paste_left:paste_right] += \ + tile_array * tile_weight[..., np.newaxis] + weight_map[paste_top:paste_bottom, paste_left:paste_right] += tile_weight + + print(f"タイル {i} 配置: 元座標({left},{top},{right},{bottom}) -> " + f"配置座標({paste_left},{paste_top},{paste_right},{paste_bottom}) " + f"サイズ({actual_width}×{actual_height})") + + def _create_blend_weight_mask(self, tile_shape: tuple, overlap_info: dict, + crop_box: list, upscale_factor: float) -> np.ndarray: + """オーバーラップ領域用のウェイトマスクを作成""" + height, width = tile_shape + weight_mask = np.ones((height, width), dtype=np.float32) + + # オーバーラップピクセル数を計算（アップスケール後） + overlap_pixels = int(self.tile_size * self.overlap_ratio * upscale_factor) + + # 実際の切り出しボックスを使ってオーバーラップ領域を正確に計算 + left, top, right, bottom = crop_box + + # 左端のフェード（左にオーバーラップがある場合） + if overlap_info["has_left_overlap"]: + # 実際のオーバーラップ開始位置を計算 + overlap_start_x = left % self.step_size + if overlap_start_x > 0: + fade_width = min(int(overlap_start_x * upscale_factor), width) + if fade_width > 0: + fade_weights = np.linspace(0, 1, fade_width) + weight_mask[:, :fade_width] *= fade_weights[np.newaxis, :] + + # 右端のフェード（右にオーバーラップがある場合） + if overlap_info["has_right_overlap"]: + # 右端のオーバーラップ幅を計算 + next_tile_start = (left // self.step_size + 1) * self.step_size + if right > next_tile_start: + overlap_width = right - next_tile_start + fade_width = min(int(overlap_width * upscale_factor), width) + if fade_width > 0: + fade_weights = np.linspace(1, 0, fade_width) + weight_mask[:, -fade_width:] *= fade_weights[np.newaxis, :] + + # 上端のフェード（上にオーバーラップがある場合） + if overlap_info["has_top_overlap"]: + # 実際のオーバーラップ開始位置を計算 + overlap_start_y = top % self.step_size + if overlap_start_y > 0: + fade_height = min(int(overlap_start_y * upscale_factor), height) + if fade_height > 0: + fade_weights = np.linspace(0, 1, fade_height) + weight_mask[:fade_height, :] *= fade_weights[:, np.newaxis] + + # 下端のフェード（下にオーバーラップがある場合） + if overlap_info["has_bottom_overlap"]: + # 下端のオーバーラップ高さを計算 + next_tile_start = (top // self.step_size + 1) * self.step_size + if bottom > next_tile_start: + overlap_height = bottom - next_tile_start + fade_height = min(int(overlap_height * upscale_factor), height) + if fade_height > 0: + fade_weights = np.linspace(1, 0, fade_height) + weight_mask[-fade_height:, :] *= fade_weights[:, np.newaxis] + + return weight_mask + + def _normalize_blended_image(self, final_image: np.ndarray, weight_map: np.ndarray) -> np.ndarray: + """ブレンドされた画像を正規化""" + # ゼロ除算を避けるため、重みが0の部分には小さな値を設定 + weight_map = np.maximum(weight_map, 1e-8) + + # 正規化 + final_image = final_image / weight_map[..., np.newaxis] + + # 値の範囲を0-255にクランプ + final_image = np.clip(final_image, 0, 255) + + return final_image + + def _convert_to_tensor(self, final_image: Image.Image) -> torch.Tensor: + """PIL画像をComfyUI形式のテンソルに変換""" + final_tensor = torch.from_numpy(np.array(final_image)).float() / 255.0 + return final_tensor.unsqueeze(0) # バッチ次元を追加 + + +class VAEProcessor: + """VAE処理を担当するクラス""" + + @staticmethod + def tiled_decode(vae, latents: torch.Tensor, tile_size: int = 256, + overlap: int = 64, temporal_size: int = 64, + temporal_overlap: int = 8) -> torch.Tensor: + """タイル化VAEデコード""" + B, C, T, H, W = latents.shape + device = latents.device + dtype = latents.dtype + + # 出力テンソルを初期化 (VAEは8倍アップスケール) + output_h, output_w = H * 8, W * 8 + output = torch.zeros(B, T, output_h, output_w, 3, device=device, dtype=dtype) + + # タイル分割数を計算 + h_tiles, w_tiles, t_tiles = VAEProcessor._calculate_tile_counts( + H, W, T, tile_size, overlap, temporal_size, temporal_overlap + ) + + print(f"タイル化デコード: H={h_tiles}, W={w_tiles}, T={t_tiles}") + + # タイル処理実行 + VAEProcessor._process_tiles( + vae, latents, output, h_tiles, w_tiles, t_tiles, + tile_size, overlap, temporal_size, temporal_overlap, device, dtype + ) + + return output + + @staticmethod + def _calculate_tile_counts(H: int, W: int, T: int, tile_size: int, + overlap: int, temporal_size: int, temporal_overlap: int) -> tuple: + """タイル分割数を計算""" + h_tiles = max(1, (H - overlap) // (tile_size - overlap) + + (1 if (H - overlap) % (tile_size - overlap) > 0 else 0)) + w_tiles = max(1, (W - overlap) // (tile_size - overlap) + + (1 if (W - overlap) % (tile_size - overlap) > 0 else 0)) + t_tiles = max(1, (T - temporal_overlap) // (temporal_size - temporal_overlap) + + (1 if (T - temporal_overlap) % (temporal_size - temporal_overlap) > 0 else 0)) + return h_tiles, w_tiles, t_tiles + + @staticmethod + def _process_tiles(vae, latents: torch.Tensor, output: torch.Tensor, + h_tiles: int, w_tiles: int, t_tiles: int, + tile_size: int, overlap: int, temporal_size: int, + temporal_overlap: int, device, dtype): + """実際のタイル処理を実行""" + B, C, T, H, W = latents.shape + + for t_idx in range(t_tiles): + t_start, t_end = VAEProcessor._calculate_range( + t_idx, temporal_size, temporal_overlap, T + ) + + for h_idx in range(h_tiles): + h_start, h_end = VAEProcessor._calculate_range( + h_idx, tile_size, overlap, H + ) + + for w_idx in range(w_tiles): + w_start, w_end = VAEProcessor._calculate_range( + w_idx, tile_size, overlap, W + ) + + # タイルを切り出してデコード + tile_latent = latents[:, :, t_start:t_end, h_start:h_end, w_start:w_end] + + try: + with torch.autocast(device_type=mm.get_autocast_device(device), + dtype=dtype, enabled=True): + decoded_tile = vae.decode(tile_latent) + + # 出力に配置（オーバーラップ処理含む） + VAEProcessor._place_decoded_tile( + output, decoded_tile, t_start, t_end, + h_start, h_end, w_start, w_end, h_idx, w_idx + ) + + except Exception as e: + print(f"タイル({t_idx},{h_idx},{w_idx})のデコードに失敗: {e}") + continue + + @staticmethod + def _calculate_range(idx: int, size: int, overlap: int, max_size: int) -> tuple: + """範囲を計算""" + start = idx * (size - overlap) + end = min(start + size, max_size) + start = max(0, end - size) # 境界調整 + return start, end + + @staticmethod + def _place_decoded_tile(output: torch.Tensor, decoded_tile: torch.Tensor, + t_start: int, t_end: int, h_start: int, h_end: int, + w_start: int, w_end: int, h_idx: int, w_idx: int): + """デコードされたタイルを出力に配置""" + out_t_start, out_t_end = t_start, t_end + out_h_start, out_h_end = h_start * 8, h_end * 8 + out_w_start, out_w_end = w_start * 8, w_end * 8 + + # オーバーラップ処理 + if h_idx > 0 or w_idx > 0: + existing = output[:, out_t_start:out_t_end, out_h_start:out_h_end, out_w_start:out_w_end] + if existing.sum() > 0: # 既存データがある場合のみブレンド + output[:, out_t_start:out_t_end, out_h_start:out_h_end, out_w_start:out_w_end] = \ + DEFAULT_BLEND_ALPHA * existing + (1 - DEFAULT_BLEND_ALPHA) * decoded_tile + else: + output[:, out_t_start:out_t_end, out_h_start:out_h_end, out_w_start:out_w_end] = decoded_tile + else: + # 最初のタイルはそのまま配置 + output[:, out_t_start:out_t_end, out_h_start:out_h_end, out_w_start:out_w_end] = decoded_tile + + +class FramePackSingleFrameResizeSampler: + """メインのサンプラークラス""" + + @classmethod + def INPUT_TYPES(s): + return { + "required": { + "model": ("FramePackMODEL",), + "vae": ("VAE",), + "positive": ("CONDITIONING",), + "negative": ("CONDITIONING",), + "input_image": ("IMAGE", {"tooltip": "Input image to be upscaled"}), + "steps": ("INT", {"default": 30, "min": 1}), + "use_teacache": ("BOOLEAN", {"default": True, "tooltip": "Use teacache for faster sampling."}), + "teacache_rel_l1_thresh": ("FLOAT", {"default": 0.15, "min": 0.0, "max": 1.0, "step": 0.01, "tooltip": "The threshold for the relative L1 loss."}), + "cfg": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 30.0, "step": 0.01}), + "guidance_scale": ("FLOAT", {"default": 10.0, "min": 0.0, "max": 32.0, "step": 0.01}), + "shift": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1000.0, "step": 0.01}), + "seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}), + "latent_window_size": ("INT", {"default": 9, "min": 1, "max": 33, "step": 1, "tooltip": "The size of the latent window to use for sampling."}), + "gpu_memory_preservation": ("FLOAT", {"default": 6.0, "min": 0.0, "max": 128.0, "step": 0.1, "tooltip": "The amount of GPU memory to preserve."}), + "sampler": (["unipc_bh1", "unipc_bh2"], {"default": 'unipc_bh1'}), + "tile_size": ("INT", {"default": 256, "min": 256, "max": 1024, "step": 64, "tooltip": "Size of each tile for initial split"}), + "tile_resized": ("INT", {"default": 1024, "min": 512, "max": 2048, "step": 64, "tooltip": "Size to resize each tile to before processing"}), + "vae_tile_size": ("INT", {"default": 256, "min": 64, "max": 512, "step": 64, "tooltip": "Size of each VAE tile for decode"}), + "vae_overlap": ("INT", {"default": 64, "min": 0, "max": 256, "step": 8, "tooltip": "Overlap between VAE tiles"}), + "vae_temporal_size": ("INT", {"default": 64, "min": 1, "max": 128, "step": 1, "tooltip": "Temporal size for VAE tiled decode"}), + "vae_temporal_overlap": ("INT", {"default": 8, "min": 0, "max": 32, "step": 1, "tooltip": "Temporal overlap for VAE tiled decode"}), + }, + "optional": { + "image_embeds": ("CLIP_VISION_OUTPUT", ), + "denoise_strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.01}), + } + } + + RETURN_TYPES = ("IMAGE", ) + RETURN_NAMES = ("upscaled_image",) + FUNCTION = "process" + CATEGORY = "FramePackWrapper" + DESCRIPTION = "Variable ratio image upscaling using 50% overlapping tile-based FramePack single frame sampling" + + def process(self, model, vae, shift, positive, negative, input_image, latent_window_size, + use_teacache, teacache_rel_l1_thresh, steps, cfg, guidance_scale, seed, + sampler, gpu_memory_preservation, tile_size=512, tile_resized=1024, + vae_tile_size=256, vae_overlap=64, vae_temporal_size=64, + vae_temporal_overlap=8, image_embeds=None, denoise_strength=1.0): + + upscale_factor = float(tile_resized) / float(tile_size) + self._log_process_start(upscale_factor, vae_tile_size, vae_overlap, + vae_temporal_size, vae_temporal_overlap, seed) + + # デバイス設定とモデル準備 + transformer, base_dtype, device, offload_device = self._setup_model(model) + + # 1. 画像を50%オーバーラップでタイルに分割 + tile_processor = TileProcessor(tile_size, tile_resized) + tiles, metadata = tile_processor.split_image_to_tiles(input_image) + + # 2. テキストエンベッディング準備 + text_embeddings = self._prepare_text_embeddings( + positive, negative, cfg, device, base_dtype + ) + + # 3. 画像エンベッディング準備 + image_encoder_embeddings = self._prepare_image_embeddings( + image_embeds, device, base_dtype + ) + + # 4. ComfyUIセットアップ + callback = self._setup_comfy_ui(transformer, device, steps, base_dtype) + + # 5. モデル準備 + self._prepare_model(transformer, device, gpu_memory_preservation, + use_teacache, steps, teacache_rel_l1_thresh) + + # 6. 各タイルを処理 + processed_tiles = self._process_all_tiles( + tiles, vae, transformer, sampler, denoise_strength, latent_window_size, + cfg, guidance_scale, shift, steps, seed, device, base_dtype, + text_embeddings, image_encoder_embeddings, callback, + vae_tile_size, vae_overlap, vae_temporal_size, vae_temporal_overlap + ) + + # 7. 50%オーバーラップブレンドで最終画像を再構成 + reconstructor = ImageReconstructor(metadata) + final_image = reconstructor.reconstruct_image(processed_tiles) + + # クリーンアップ + self._cleanup(transformer, offload_device) + + print(f"=== 50%オーバーラップ {upscale_factor:.2f}xアップスケール完了: {final_image.shape} ===") + return (final_image,) + + def _log_process_start(self, upscale_factor: float, vae_tile_size: int, + vae_overlap: int, vae_temporal_size: int, + vae_temporal_overlap: int, seed: int): + """処理開始のログ出力""" + print(f"=== FramePack 50%オーバーラップ Resize Sampler 開始 ({upscale_factor:.2f}x) ===") + print(f"VAE Tile設定: size={vae_tile_size}, overlap={vae_overlap}, " + f"temporal_size={vae_temporal_size}, temporal_overlap={vae_temporal_overlap}") + print(f"使用シード: {seed} (全タイル共通)") + print(f"オーバーラップ比率: {OVERLAP_RATIO*100:.0f}%") + + def _setup_model(self, model: dict) -> tuple: + """モデルとデバイスのセットアップ""" + transformer = model["transformer"] + base_dtype = model["dtype"] + device = mm.get_torch_device() + offload_device = mm.unet_offload_device() + + mm.unload_all_models() + mm.cleanup_models() + mm.soft_empty_cache() + + return transformer, base_dtype, device, offload_device + + def _prepare_text_embeddings(self, positive, negative, cfg: float, + device, base_dtype) -> dict: + """テキストエンベッディングを準備""" + llama_vec = positive[0][0].to(device, base_dtype) + clip_l_pooler = positive[0][1]["pooled_output"].to(device, base_dtype) + + if not math.isclose(cfg, 1.0): + llama_vec_n = negative[0][0].to(device, base_dtype) + clip_l_pooler_n = negative[0][1]["pooled_output"].to(device, base_dtype) + else: + llama_vec_n = torch.zeros_like(llama_vec, device=device) + clip_l_pooler_n = torch.zeros_like(clip_l_pooler, device=device) + + llama_vec, llama_attention_mask = crop_or_pad_yield_mask(llama_vec, length=512) + llama_vec_n, llama_attention_mask_n = crop_or_pad_yield_mask(llama_vec_n, length=512) + + return { + "llama_vec": llama_vec, + "llama_attention_mask": llama_attention_mask, + "clip_l_pooler": clip_l_pooler, + "llama_vec_n": llama_vec_n, + "llama_attention_mask_n": llama_attention_mask_n, + "clip_l_pooler_n": clip_l_pooler_n + } + + def _prepare_image_embeddings(self, image_embeds, device, base_dtype): + """画像エンベッディングを準備""" + if image_embeds is not None: + return image_embeds["last_hidden_state"].to(device, base_dtype) + return None + + def _setup_comfy_ui(self, transformer, device, steps, base_dtype): + """ComfyUIセットアップ""" + comfy_model = HyVideoModel( + HyVideoModelConfig(base_dtype), + model_type=comfy.model_base.ModelType.FLOW, + device=device, + ) + patcher = comfy.model_patcher.ModelPatcher(comfy_model, device, torch.device("cpu")) + return prepare_callback(patcher, steps) + + def _prepare_model(self, transformer, device, gpu_memory_preservation: float, + use_teacache: bool, steps: int, teacache_rel_l1_thresh: float): + """モデルの準備（GPU移動とTeaCache設定）""" + # モデルをGPUに移動 + move_model_to_device_with_memory_preservation( + transformer, target_device=device, + preserved_memory_gb=gpu_memory_preservation + ) + + # TeaCacheの設定 + if use_teacache: + transformer.initialize_teacache( + enable_teacache=True, num_steps=steps, + rel_l1_thresh=teacache_rel_l1_thresh + ) + else: + transformer.initialize_teacache(enable_teacache=False) + + def _process_all_tiles(self, tiles: list, vae, transformer, sampler: str, + denoise_strength: float, latent_window_size: int, + cfg: float, guidance_scale: float, shift: float, + steps: int, seed: int, device, base_dtype, + text_embeddings: dict, image_encoder_embeddings, + callback, vae_tile_size: int, vae_overlap: int, + vae_temporal_size: int, vae_temporal_overlap: int) -> list: + """全てのタイルを処理""" + processed_tiles = [] + + print(f"=== {len(tiles)}個のタイルを順次処理（50%オーバーラップ、各タイルで同一シード使用） ===") + + for i, tile_tensor in enumerate(tqdm(tiles, desc="オーバーラップタイル処理")): + print(f"\n--- タイル {i+1}/{len(tiles)} 処理中（シード: {seed}） ---") + + # 各タイルで同じシードを使用するためにジェネレータをリセット + rnd = torch.Generator("cpu").manual_seed(seed) + + processed_tile = self._process_single_tile( + tile_tensor, vae, transformer, sampler, denoise_strength, + latent_window_size, cfg, guidance_scale, shift, steps, rnd, + device, base_dtype, text_embeddings, image_encoder_embeddings, + callback, vae_tile_size, vae_overlap, vae_temporal_size, vae_temporal_overlap + ) + + processed_tiles.append(processed_tile) + print(f"オーバーラップタイル {i+1} 完了: {processed_tile.shape}") + + return processed_tiles + + def _process_single_tile(self, tile_tensor: torch.Tensor, vae, transformer, + sampler: str, denoise_strength: float, latent_window_size: int, + cfg: float, guidance_scale: float, shift: float, + steps: int, rnd: torch.Generator, device, base_dtype, + text_embeddings: dict, image_encoder_embeddings, + callback, vae_tile_size: int, vae_overlap: int, + vae_temporal_size: int, vae_temporal_overlap: int) -> torch.Tensor: + """単一タイルを処理""" + tile_tensor = tile_tensor.to(device) + + with torch.autocast(device_type=mm.get_autocast_device(device), + dtype=base_dtype, enabled=True): + # VAEエンコード + start_latent = self._encode_tile(vae, tile_tensor) + + # FramePack推論のためのインデックス設定 + latent_indices, clean_latents, clean_latent_indices = self._setup_latent_indices( + start_latent, latent_window_size + ) + + # FramePack推論実行 + generated_latents = self._run_framepack_inference( + transformer, sampler, denoise_strength, start_latent.shape, + cfg, guidance_scale, shift, steps, rnd, device, base_dtype, + text_embeddings, image_encoder_embeddings, latent_indices, + clean_latents, clean_latent_indices, callback + ) + + # VAEタイル化デコード + decoded_tile = self._decode_tile( + vae, generated_latents, vae_tile_size, vae_overlap, + vae_temporal_size, vae_temporal_overlap + ) + + # 時間次元を除去して画像形式に変換 [B, T, H, W, C] -> [B, H, W, C] + if len(decoded_tile.shape) == 5: + decoded_tile = decoded_tile.squeeze(1) + + return decoded_tile + + def _encode_tile(self, vae, tile_tensor: torch.Tensor) -> torch.Tensor: + """タイルをVAEエンコード""" + start_latent = vae.encode(tile_tensor) + start_latent = start_latent * VAE_SCALING_FACTOR + + print(f"エンコード後latent: {start_latent.shape}") + + # latentを5次元に変換 [B, C, T, H, W] + if len(start_latent.shape) == 4: + start_latent = start_latent.unsqueeze(2) # T次元を追加 + + return start_latent + + def _setup_latent_indices(self, start_latent: torch.Tensor, + latent_window_size: int) -> tuple: + """FramePack推論のためのインデックス設定""" + B, C, T, H, W = start_latent.shape + + # 1フレーム推論のためのインデックス設定 + sample_num_frames = 1 + latent_padding = 0 + latent_padding_size = latent_padding * latent_window_size + + indices = torch.arange(0, sum([1, latent_padding_size, latent_window_size, 1, 2, 16])).unsqueeze(0) + + if latent_padding_size == 0: + clean_latent_indices_pre = indices[:, 0:1] + latent_indices = indices[:, 1:1+latent_window_size] + clean_latent_indices_post = indices[:, 1+latent_window_size:2+latent_window_size] + clean_latent_2x_indices = indices[:, 2+latent_window_size:4+latent_window_size] + clean_latent_4x_indices = indices[:, 4+latent_window_size:20+latent_window_size] + blank_indices = torch.empty((1, 0), dtype=torch.long) + else: + split_sizes = [1, latent_padding_size, latent_window_size, 1, 2, 16] + clean_latent_indices_pre, blank_indices, latent_indices, clean_latent_indices_post, clean_latent_2x_indices, clean_latent_4x_indices = indices.split(split_sizes, dim=1) + + # 1フレーム推論設定 + all_indices = torch.arange(0, latent_window_size).unsqueeze(0) + latent_indices = all_indices[:, -1:] + + clean_latents_pre = start_latent.to(torch.float32).cpu() + clean_latents_post = torch.zeros_like(clean_latents_pre) + clean_latents = torch.cat([clean_latents_pre, clean_latents_post], dim=2) + + clean_latent_indices = torch.tensor([[0]], dtype=clean_latent_indices_pre.dtype, device=clean_latent_indices_pre.device) + clean_latents = clean_latents[:, :, :1, :, :] + + return latent_indices, clean_latents, clean_latent_indices + + def _run_framepack_inference(self, transformer, sampler: str, denoise_strength: float, + latent_shape: tuple, cfg: float, guidance_scale: float, + shift: float, steps: int, rnd: torch.Generator, + device, base_dtype, text_embeddings: dict, + image_encoder_embeddings, latent_indices: torch.Tensor, + clean_latents: torch.Tensor, clean_latent_indices: torch.Tensor, + callback) -> torch.Tensor: + """FramePack推論を実行""" + B, C, T, H, W = latent_shape + sample_num_frames = 1 + + generated_latents = sample_hunyuan( + transformer=transformer, + sampler=sampler, + initial_latent=None, + strength=denoise_strength, + width=W * 8, + height=H * 8, + frames=sample_num_frames, + real_guidance_scale=cfg, + distilled_guidance_scale=guidance_scale, + guidance_rescale=0, + shift=shift if shift != 0 else None, + num_inference_steps=steps, + generator=rnd, + prompt_embeds=text_embeddings["llama_vec"], + prompt_embeds_mask=text_embeddings["llama_attention_mask"], + prompt_poolers=text_embeddings["clip_l_pooler"], + negative_prompt_embeds=text_embeddings["llama_vec_n"], + negative_prompt_embeds_mask=text_embeddings["llama_attention_mask_n"], + negative_prompt_poolers=text_embeddings["clip_l_pooler_n"], + device=device, + dtype=base_dtype, + image_embeddings=image_encoder_embeddings, + latent_indices=latent_indices, + clean_latents=clean_latents, + clean_latent_indices=clean_latent_indices, + clean_latents_2x=None, # 2x, 4x latentsを無効化 + clean_latent_2x_indices=None, + clean_latents_4x=None, + clean_latent_4x_indices=None, + callback=callback, + ) + + return generated_latents + + def _decode_tile(self, vae, generated_latents: torch.Tensor, vae_tile_size: int, + vae_overlap: int, vae_temporal_size: int, vae_temporal_overlap: int) -> torch.Tensor: + """タイルをVAEデコード""" + generated_latents = generated_latents / VAE_SCALING_FACTOR + print(f"デコード前latent: {generated_latents.shape}") + + decoded_tile = VAEProcessor.tiled_decode( + vae, generated_latents, + tile_size=vae_tile_size, + overlap=vae_overlap, + temporal_size=vae_temporal_size, + temporal_overlap=vae_temporal_overlap + ) + + return decoded_tile + + def _cleanup(self, transformer, offload_device): + """リソースのクリーンアップ""" + transformer.to(offload_device) + mm.soft_empty_cache() + + +# ノードクラスマッピング +NODE_CLASS_MAPPINGS = { + "FramePackSingleFrameResizeSampler": FramePackSingleFrameResizeSampler, +} + +NODE_DISPLAY_NAME_MAPPINGS = { + "FramePackSingleFrameResizeSampler": "FramePack Single Frame Resize Sampler (50% Overlap)", } \ No newline at end of file