Update pyproject.toml

- Changed FlipStreamChatJson: `temperature` and `top_p` are now inputs.
- Add FlipStreamViewerVideo: streams frames and optional audio to the viewer.
- Add FlipStreamFileSelect_DiffusionModels.
- Add FlipStreamLogger.
- Some fixes.

.github/workflows/publish.yml

@@ -4,21 +4,22 @@ on:
   push:
     branches:
       - main
-      - master
     paths:
       - "pyproject.toml"
 
+permissions:
+  issues: write
+
 jobs:
   publish-node:
     name: Publish Custom Node to registry
     runs-on: ubuntu-latest
-    # if this is a forked repository. Skipping the workflow.
-    if: github.event.repository.fork == false 
+    if: ${{ github.repository_owner == 'sakura1bgx' }}
     steps:
       - name: Check out code
         uses: actions/checkout@v4
       - name: Publish Custom Node
-        uses: Comfy-Org/publish-node-action@main
+        uses: Comfy-Org/publish-node-action@v1
         with:
           ## Add your own personal access token to your Github Repository secrets and reference it here.
           personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }}

README.md

@@ -1,128 +1,158 @@
 # ComfyUI_FlipStreamViewer
 
-**ComfyUI_FlipStreamViewer** is a tool that provides a viewer interface for flipping images with frame interpolation, allowing you to watch high-fidelity pseudo-videos without needing AnimateDiff.
+**ComfyUI_FlipStreamViewer** is a tool that provides a customizable viewer interface for flipping images with frame interpolation.
 
-![sample](https://github.com/user-attachments/assets/4ceecd68-bd35-4eb5-95bf-4ce822ff0591)
-
-## Required Custom Nodes
-
-Please install the following nodes via ComfyUI-Manager:
-
-- ComfyUI Impact Pack
-- ComfyUI Frame Interpolation
-- ComfyUI WD 1.4 Tagger
-- ComfyUI-Advanced-ControlNet
-- ComfyUI-AutomaticCFG
-- LoRA Tag Loader for ComfyUI
+<div><video controls src="https://github.com/user-attachments/assets/2a40c7c6-045e-4d86-a0fd-51a24f5472b4"></video></div>
 
 ## Getting Started
 1. **Run ComfyUI** and from ComfyUI-Manager, select Manager -> Custom Node Manager and install ComfyUI_FlipStreamViewer.
-2. **Restart ComfyUI** and load the `workflow.json` file located in the `workflows` folder.
-3. Access the viewer at the following URL if ComfyUI is running at `http://localhost:8188`:
+2. **Restart ComfyUI** and load the `simple.json` file located in the `workflows` folder.
+3. Access the viewer at the following URL if ComfyUI is running at localhost port 8188:
 
-`http://localhost:8188/viewer`
+`http://localhost:8188/flipstreamviewer`
 
-4. To respond to viewer input:
-- Check 'Extra options', 'Auto Queue', and 'instant' on the ComfyUI control panel.
-- Then click the 'Queue Prompt' button.
+4. Run a workflow on the ComfyUI
 
-5. The 'preset' folder may be created in the current directory for saving and loading preset files. Sample presets can be found in the `presetsamples` folder.
+5. The UI nodes will refresh after the workflow is completed once, so you need to click 'Update' after that.
 
-## Prompt Format
+6. The 'preset' folder may be created in the current directory for saving and loading preset files.
 
-Use the following format for input prompts. The 'pre text' and 'append text' sections apply to all frames. The separator `----` should consist of four hyphens, and each line of 'frameN text' applies to individual frames. The '{lora}' means it will be replaced by the LoRA input contents at the right panel.
-
-```
-pre text,
-{lora}
-----
-- frame1 text
-- frame2 text
-----
-append text
-```
-
-### Limitations
+## Limitations
 
 - Please note that this tool is not designed with security measures against third-party attacks. Therefore, it should only be used for personal purposes within a secure network.
 - There is no save video function available; please customize your workflow as needed.
-- Excessively long text may cause exceptions.
-- The keywords `BREAK`, `AND` cannot be used in the prompt.
 
-## Parameters
+## Optional Custom Nodes dependencies
 
-You can adjust the following parameters on the viewer:
+The following custom nodes can be used as needed:
+- ComfyUI-Inspyrenet-Rembg: For FlipStreamRembg
+- ComfyUI-Frame-Interpolation: For FlipStreamFilmVfi
 
-- **Seed**: Random seed for KSampler.
-- **Keep**: Keeped seed.
-- **Iv**: Interval in seconds for auto-changing the random seed.
-- **Nstp**: Steps for KSampler.
-- **Ncfg**: CFG for KSampler.
-- **Nsta**: Start step for the second KSampler; increasing this value will stabilize the output between frames.
-- **Nfrm**: Number of frames; each frame undergoes x8 interpolation in the sample workflow.
-- **Nspf**: Seconds per frame displayed in the viewer stream.
-- **Rate**: LoRA late for toggle and add tags rank.
-- **Nfst**: Seek first frame for video source
-- **Nskp**: Skip frames for video source
-- **Nstr**: Strength for video source controlnet
-- **Nwth**: Threshould for WD14Tagger.
-- **Ncth**: Character threshould for WD14Tagger.
+The following custom nodes may also be used within workflows:
+- ComfyUI Impact Pack
+- LoRA Tag Loader for ComfyUI
+- ComfyUI-WD14-Tagger
+- ComfyUI-AnimateDiff-Evolved
+- ComfyUI-VideoHelperSuite
+- ComfyUI-DepthAnythingV2
+- ComfyUI-Advanced-ControlNet
 
-## Inputs
+## Optional Python Packages Dependencies
 
-Provide the following text inputs in the viewer:
+llama-cpp-python for FlipStreamChat:
+```
+REM Before that, may be need to install Visual Studio 2022, VC++ development options and cmake
+set PATH=D:\ComfyUI_windows_portable\python_embeded;D:\ComfyUI_windows_portable\python_embeded\Scripts;%PATH%
+cd python_embeded
+python -m pip install scikit-build-core
+python -m pip install cmake
+set CMAKE_ARGS="-DGGML_CUDA=on"
+python -m pip install llama-cpp-python
+```
 
-- **Prompt Input**: Located at the top of the left panel (refer to the Prompt Format section above).
-- **Negative Prompt Input**: Located below the Prompt Input; this applies to all frames.
-- **LoRA Input**: Located in the middle of the right panel; input LoRA such as `<lora:startsfilename:1>`.
+## UI Features
 
-## Controls
+- **Customizable Left Panel**: The left panel UI is customizable with UI Nodes.
+- **UI Node Ordering**: These UI nodes are ordered by their titles, and you can change the node titles to something like '10.steps' or '11.cfg'. You can also click on the title to compact the node on the workflow.
+- **Label Input**: Each UI node has a 'label' input, which should be a unique identifier used as the parameter name. 'lora' is internally used, so they cannot be used for labels.
+- **Enable Output**: Each UI node also has an 'enable' output to use with 'Control Bridge' for switching to bypass some nodes or FlipStreamSwitch* for switching input.
+- **Right Panel**: You can operate several prepared functions such as Status, Darker, Preset, and Lora.
+- **Status**: The right panel shows ComfyUI status and error information.
+- **Darker**: You can adjust the image brightness on the viewer using the darker parameter. It can also be set via query:
+`http://localhost:8188/flipstreamviewer?darker=0.33`
+- **Preset**: You can save and load parameters that are set on viewer controls. There are joking level XOR crypt mode for preset files, it can be use with &px query. The 'M' button means Move to another folder. It can also be set via query:
+`http://localhost:8188/flipstreamviewer?showPresetDialog=1&presetFolder=folder_name&presetFile=file_name.json`
+- **Lora**: You can select LoRA and choose tags. It can also choose random tags. The LoRA preview box can be clicked to jump to the Civitai LoRA page if found. The 'M' button means Move to another folder. The 'T' button means Toggle. The 'R' button means Random choose.
+- **Toggle View**: You can click the center panel to hide the left/right panels and the message box, a subsequent click will show them. It can also be set via query: 
+`http://localhost:8188/flipstreamviewer?toggleView=1`
+- **Run on Load**: The workflow can be run when the page is loaded via query:
+`http://localhost:8188/flipstreamviewer?run=1`
+- **Message Box**: If a message is set by FlipStreamSetMessage, you can view it at the top left of the center panel.
+- **Auto Hide**: Stream and preview in the viewer will automatically hide after 5 minutes if the page is not reloaded.
+- **Reset**: clears the queue, interrupts the current run and resets all viewer parameters, state and stream. It can be called via query:
+`http://localhost:8188/flipstreamviewer?reset=1`
 
-**Left Panel Buttons:**
+## UI Nodes
 
-- **U**: Apply input data to the workflow.
-- **R**: Change the random seed and update.
-- **K**: Keep the seed to search for another good seed.
-- **B**: Go back to the previous seed.
-- **M**: Move the checkpoint file.
+- **FlipStreamSection**: A section label for UI Nodes. 'closed' makes the section collapsed initially.
+- **FlipStreamButton**: Run and Capture buttons. The Capture button can be use with FlipStreamGetFrame to get captured image.
+- **FlipStreamSlider**: A slider for adjusting values.
+- **FlipStreamTextBox**: A text box for inputting multiline text.
+- **FlipStreamInputBox**: An input box for various boxtype inputs. The 'U' button means update. You can choose special boxtype 'seed' or 'r4d' with the 'R' button, which means randomize. The boxtype 'r4d' can be used to generate a 4-digit part of a prompt like 'MOV_{num}'. In this case, FlipStreamTextReplace can help find '{num}' and replace with the output of the input box.
+- **FlipStreamSelectBox_Samplers**: A select box for choosing samplers.
+- **FlipStreamSelectBox_Scheduler**: A select box for choosing schedulers.
+- **FlipStreamSizeSelect**: A select box for choosing size by aspect ratio. The size can also get using FlipStreamGetSize.
+- **FlipStreamFileSelect_Checkpoints**: A file selector for checkpoints. 'mode' is used to select a subfolder such as 'sd15', 'sdxl', 'pony', or 'flux' in 'checkpoints'. 'use_sub' means using subfolders in the 'mode' folder. 'use_move' means using the move file selector.
+- **FlipStreamFileSelect_DiffusionModels**: A file selector for diffusion models.
+- **FlipStreamFileSelect_VAE**: A file selector for VAE models.
+- **FlipStreamFileSelect_LLM**: A file selector for LLM models.
+- **FlipStreamFileSelect_ControlNetModel**: A file selector for ControlNet models.
+- **FlipStreamFileSelect_TensorRT**: A file selector for TensorRT models. It may be used with ComfyUI_TensorRT.
+- **FlipStreamFileSelect_AnimateDiffModel**: A file selector for AnimateDiff models. It may be used with ComfyUI-AnimateDiff-Evolved.
+- **FlipStreamFileSelect_Input**: A file selector for ComfyUI input folder.
+- **FlipStreamFileSelect_Output**: A file selector for ComfyUI output folder.
+- **FlipStreamPreviewBox**: A box for previewing the input image. You can select the ROI (Region of Interest) in the preview by dragging the mouse.
+- **FlipStreamPasteBox**: A box for paste an input image. You can select the box by click once, then you can paste image from clipboard by Ctrl+V or delete image by DEL.
+- **FlipStreamLogBox**: A box for displaying logs on the screen.
 
-**Right Panel Buttons:**
+## Other Nodes
 
-- **T**: Toggle LoRA enable/disable.
-- **M**: Move the LoRA file.
-- **T**: Toggle tag enable/disable at the LoRA Input.
-- **R**: Add random tags at the LoRA Input.
-- **PV**: Preview video source.
-- **R**: Select random preset.
-- **P**: Load a preset prompt only.
-- **L**: Load a preset.
-- **M**: Move the Preset file.
-- **Save**: Save the current preset.
-- **C**: Capture an image from another browser tab or desktop and set it as the first frame; useful with W.
-- **W**: Apply WD14Tagger to the first frame; outputs tags to LoRA input.
-- **T**: Toggle tags in LoRA input. 
-- **U**: Update without reload; useful with T.
-- **Clr**: Clear the LoRA input.
+- **FlipStreamRunOnce**: Updates parameters, reloads pages, and run the current workflow once.
+- **FlipStreamSetMessage**: Sets a message for the message box at the bottom of the center panel.
+- **FlipStreamAnd(experimental)**: Multiple inputs AND operation node.
+- **FlipStreamOr(experimental)**: Multiple inputs OR operation node.
+- **FlipStreamSetState**: A node for setting internal state.
+- **FlipStreamGetState**: A node for getting internal state.
+- **FlipStreamSetParam**: A node for setting parameters. However, it needs a reload to update the value in the viewer UI.
+- **FlipStreamGetParam**: A node for getting parameters. 'lora' is a prepared parameter that contains text at the Lora input. Some prompts can contain '{lora}' and FlipStreamTextReplace can help find '{lora}' and replace it with the output of the get param node of 'lora'. 'b64dec' is true for 'lora' and other 'FlipStreamTextBox' parameters, because these multiline parameters are internally base64 encoded.
+- **FlipStreamGet(experimental)**: A node for getting internal state and parameters with independent node cache.
+- **FlipStreamGetFrame**: A node for get image from frame buffer. It can be use with FlipStreamButton - Capture button to get captured image.
+- **FlipStreamGetPreviewRoi**: A node for obtaining the preview ROI (Region of Interest) selection.
+- **FlipStreamImageSize**: A node for getting image size.
+- **FlipStreamTextReplace**: A node for replacing text. It will output the result of `text.replace(find, replace.format(value))`.
+- **FlipStreamTextConcat**: A node for concatenate text.
+- **FlipStreamScreenGrabber**: A node for grab multiframe screenshots.
+- **FlipStreamVideoInput**: A node for input video frames.
+- **FlipStreamSource**: A node for prepare image or latent source.
+- **FlipStreamSwitchImage**: A node for switching images.
+- **FlipStreamSwitchLatent**: A node for switching latents.
+- **FlipStreamGate(deprecated)**: A node for consolidating input timing to ensure that inputs to the sampler do not occur sequentially.
+- **FlipStreamRembg**: A node for remove background. It depends on ComfyUI-Inspyrenet-Rembg.
+- **FlipStreamSegMask(deprecated)**: Currently cannot use, so please use original node ComfyUI-Florence2.
+- **FlipStreamChat**: Loads an LLM (Large Language Model) and obtains chat responses. It only supports *.gguf files in the models/LLM folder and uses llama-cpp-python. The output format can be customized using the response_format parameter, which supports [JSON Schema](https://llama-cpp-python.readthedocs.io/en/latest/#json-schema-mode) for structured responses.
+- **FlipStreamParseJson**: Extracts values for multiple keys from a JSON string and joins them using a specified delimiter.
+- **FlipStreamChatJson**: Mix and simplified FlipStreamChat and FlipStreamParseJson using auto generated json scheme.
+- **FlipStreamBatchPrompt**: A node for simple batch prompting. Use the following format for the input prompt of this node. The 'pre text' and 'append text' sections apply to all frames. The separator `----` should consist of four hyphens, and each line of 'frame text' applies evenly to the number of frames specified in 'frames'. For example, if 'frames' is 8 and there are 2 'frame text' lines, they will be applied starting from frames 0 and 4.
+```
+pre text,
+----
+- frame text,
+- frame text,
+----
+append text
+```
+- **FlipStreamFilmVfi**: A node for video frame interpolation. It depends on ComfyUI-Frame-Interpolation.
+- **FlipStreamViewer**: A node for viewing content. The 'allowip' parameter allows you to set IP addresses that can access the viewer. ComfyUI commandline options "--listen 0.0.0.0" and appropriate firewall settings are also needed for that. The 'w14exc' parameter is used to set exclude_tags for the WD14 Tagger. 'fps' can control flip speed.
+- **FlipStreamViewerSimple**: Simplified FlipStreamViewer.
+- **FlipStreamViewerVideo**: Streams frames (and optional audio) to the viewer as a single MP4 instead of PNG frames, it is much faster for long or large videos. Audio is muted and the volume is 0 by default, the mute button and volume slider appear at the top left. FlipStreamGetFrame cannot get frames from the video stream.
+- **FlipStreamLogger**: Prints the input data to the console with a label.
+- **FlipStreamCurrent**: A node for setting current information text on the right panel status info.
+- **FlipStreamAllowIp**: A node for setting allowip.
+- **FlipStreamLoraMode**: A node for setting loraMode.
+- **FlipStreamSaveApiWorkflow**: A node for save current workflow as api workflow.
+- **FlipStreamRunApiWorkflow**: A node for run an api workflow.
+- **FlipStreamFree**: A node for free model and comfy caches.
+- **FlipStreamShutdown**: A node for scheduling shutdown (Windows only).
 
-## Additional Features
+## Workflow Examples
 
-- **LoRA Preview**: At the bottom of the right panel; click to jump to the Civitai LoRA page if found.
-- **Auto-hide**: Click to hide controls and images or the viewer auto-hides after 5 minutes of inactivity (reload).
+These workflow are sometimes old.
 
-## For More Quality
-
-- Use more higher-quality checkpoint models.
-- To enhance stability between frames, use a deliberate prompt in the 'pre text' to establish a consistent background, such as `ideal background, on desk`.
-- Select the appropriate words or LoRA in 'frameN text' to achieve the desired motion.
-
-## For More Speed
-
-- ComfyUI commandline options '--highvram --fast --fp8_e4m3fn-unet' are good for speed, however it depends on system and model.
-
-## Customizing Workflow
-
-You can customize the workflow for specific needs:
-
-- **FlipStreamLoader**: Use the 'mode' parameter to switch between sub-directories for checkpoints and LoRAs, useful for managing models like sd15, pony, etc.
-- **FlipStreamSetMode**: Only change the 'mode' parameter for the workflow without checkpoints.
-- **FlipStreamViewer**: The 'allowip' parameter allows you to set IP addresses that can access the viewer. The 'w14exc' parameter is used to set exclude_tags for the WD14 Tagger.
+- **simple.json**: Customized a ComfyUI default workflow for FlipStreamViewer UI nodes.
+- **dmd2_lora.json**: Using DMD2 and some LoRA, only needs 5 steps to generate an image.
+- **segmask.json**: A 2-step process for generating frames using batch prompts and segmentation mask, followed by interpolation.
+- **animate.json**: Using AnimateDiff with FlipStreamSource, FlipStreamBatchPrompt, and FlipStreamFilmVfi.
+- **grab_rembg_depth_animate.json**: Using ControlNet and AnimateDiff with FlipStreamScreenGrabber, FlipStreamRembg.
+- **quick_vid2vid.json**: Quick tuning workflow for vid2vid.
+- **quick_vid2vid_roi.json**: An example demonstrating the use of FlipStreamGetPreviewRoi and FlipStreamGate.
+- **visualnobel.json**: Visual nobel like UI using FlipStreamChat.

__init__.py

@@ -1,3 +1,2 @@
-from .flipstreamviewer import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
-
-__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']
+from .flipstreamviewer import WEB_DIRECTORY, NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
+__all__ = ['WEB_DIRECTORY', 'NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']

presetsamples/cat.json

@@ -1 +0,0 @@
-{"prompt": "aGlnaHJlcywgaWRlYWwgYmFja2dyb3VuZCwgb24gdGFibGUsIHNvbG8sIGJsYWNrIGNhdCwKe2xvcmF9Ci0tLS0KLSAKLSBzaXR0aW5nCi0tLS0=", "negativePrompt": "", "seed": 0, "keepSeed": 0, "steps": 12, "cfg": 4, "interval": 30, "sampler": "dpmpp_2m,sgm_uniform", "checkpoint": "", "lora": "bG9yYSBhbmQgZGV0YWlscw==", "startstep": 0, "frames": 2, "framewait": 1, "offsetX": 0, "offsetY": 0, "scale": 100}

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "comfyui_flipstreamviewer"
-description = "ComfyUI_FlipStreamViewer is a tool that provides a viewer interface for flipping images with frame interpolation, allowing you to watch high-fidelity pseudo-videos without needing AnimateDiff. Please note that this tool is not designed with security measures against third-party attacks. Therefore, it should only be used for personal purposes within a secure network."
-version = "2.1.0"
+description = "ComfyUI_FlipStreamViewer is a tool that provides a customizable viewer interface for flipping images with frame interpolation."
+version = "4.1.0"
 license = {file = "LICENSE.txt"}
 
 [project.urls]

requirements.txt

@@ -1 +1,2 @@
-opencv-python
+imageio
+mss

web/flipstreamviewer.js

@@ -0,0 +1,68 @@
+import { app } from "../../../scripts/app.js";
+import { api } from "../../../scripts/api.js";
+
+app.registerExtension({
+    name: "FlipStreamViewer.RunOnce",
+    async setup() {
+        api.addEventListener("FlipStreamViewer_run_once", async () => {
+            await api.interrupt();
+            await api.fetchApi("/queue", {
+                method: "POST",
+                body: JSON.stringify({ clear: true }),
+            });
+            await app.queuePrompt(0, 1);
+        });
+    }
+});
+
+app.registerExtension({
+    name: "FlipStreamViewer.FlipStreamChatJson",
+    async beforeRegisterNodeDef(t, d) {
+        const config = {
+            "FlipStreamGet":      { in: 0,  out: 2, range: 20, iname:'label', oname:'value', useo: true },
+            "FlipStreamChatJson": { in: 11, out: 3, range: 20, iname:'label', oname:'value', useo: true },
+            "FlipStreamTextConcat": { in: 2, out: 1, range: 20, iname:'text', oname:'', useo: false }
+        };
+        const CFG = config[d.name];
+        if (!CFG) return;
+
+        const sync = (n) => {
+            const w = n.widgets;
+            if (CFG.useo) {
+                while (n.outputs.length > w.length + CFG.out - CFG.in) {
+                    if (n.outputs.length < CFG.out || n.outputs[n.outputs.length - 1]?.links?.length) break;
+                    n.removeOutput(n.outputs.length - 1);
+                    n.outputs_dirty = true;
+                }
+            }
+            if (w.length < CFG.in + CFG.range && (w.at(-1)?.value || n.inputs[w.length - 1]?.link !== null)) {
+                if (CFG.useo) {
+                    n.addOutput(`${CFG.oname}${w.length - CFG.in}`, "*");
+                    n.outputs_dirty = true;
+                }
+                const inputName = `${CFG.iname}${w.length - CFG.in}`;
+                n.addWidget("string", inputName, "", () => {}, { forceInput: true });
+                n.inputs_dirty = true;
+            }
+            while (w.length > CFG.in + 1 && (!w.at(-1).value && n.inputs[w.length - 1]?.link === null) && (!w.at(-2).value && n.inputs[w.length - 2]?.link === null)) {
+                if (n.outputs[w.length + CFG.out - CFG.in - 1]?.links?.length) break;
+                if (CFG.useo) {
+                    n.removeOutput(w.length + CFG.out - CFG.in - 1);
+                    n.outputs_dirty = true;
+                }
+                w.pop();
+                n.inputs_dirty = true;
+            }
+            n.setSize(n.computeSize());
+            n.setDirtyCanvas(true, true);
+        };
+
+        t.prototype.onNodeCreated = function() { sync(this); };
+        t.prototype.onWidgetChanged = function() { sync(this); };
+        t.prototype.onConnectionsChange = function(type, slot, connected, info) { requestAnimationFrame(() => sync(this)); };
+
+        t.prototype.onConfigure = function(c) {
+            c.widgets_values.forEach((v, i) => { if (this.widgets[i]) this.widgets[i].value = v; sync(this);});  
+        };
+    }
+});