Lama 改成手动安装，新增JsonRepair

Update PyProject Toml - License

CNAME

@@ -0,0 +1 @@
+mixlabnodes.com

README.md

@@ -1,15 +1,27 @@
 ![](https://img.shields.io/github/release/shadowcz007/comfyui-mixlab-nodes)
 
-> 适配了最新版 comfyui 的 py3.11 ，torch 2.1.2+cu121
+> 适配了最新版 comfyui 的 py3.11 ，torch 2.3.1+cu121
 > [Mixlab nodes discord](https://discord.gg/cXs9vZSqeK)
 
 
 ##### `最新`：
 
-ChatGPT 节点支持 Local LLM（llama.cpp），Phi3、llama3 都可以直接一个节点运行了。模型下载后，放置到 `models/llamafile/`
+- App模式增加batch prompt，批量提示词，可以把动态提示词批量组成后运行
 
-- 右键菜单支持 text-to-text，方便对 prompt 词补全
+![alt text](./assets/1722517810720.png)
 
+- 增加 API Key Input 节点，用于管理LLM的Key,同时优化LLM相关节点，为后续agent模式做准备
+
+- 增加 SiliconflowLLM，可以使用由Siliconflow提供的免费LLM
+
+- 增加 Edit Mask，方便在生成的时候手动绘制 mask [workflow](./workflow/edit-mask-workflow.json)
+
+- LaMaInpainting 调整为手动安装
+
+<!-- - ChatGPT 节点支持 Local LLM（llama.cpp），Phi3、llama3 都可以直接一个节点运行了。模型下载后，放置到 `models/llamafile/` -->
+
+<!-- - 右键菜单支持 text-to-text，方便对 prompt 词补全 -->
+<!-- 
 强烈推荐：
 [Phi-3-mini-4k-instruct-function-calling-GGUF](https://huggingface.co/nold/Phi-3-mini-4k-instruct-function-calling-GGUF)
 
@@ -18,12 +30,16 @@ ChatGPT 节点支持 Local LLM（llama.cpp），Phi3、llama3 都可以直接一
 - 右键菜单支持 image-to-text，使用多模态模型，多模态使用 [llava-phi-3-mini-gguf](https://huggingface.co/xtuner/llava-phi-3-mini-gguf/tree/main)，注意需要把llava-phi-3-mini-mmproj-f16.gguf也下载
 
 ![](./assets/prompt_ai_setup.png)
-![](./assets/prompt-ai.png)
+![](./assets/prompt-ai.png) -->
 
 
 #### `相关插件推荐`
 
-<!-- [comfyui-sd-prompt-mixlab](https://github.com/shadowcz007/comfyui-sd-prompt-mixlab) -->
+[comfyui-liveportrait](https://github.com/shadowcz007/comfyui-liveportrait)
+
+[Comfyui-ChatTTS](https://github.com/shadowcz007/Comfyui-ChatTTS)
+
+[comfyui-sound-lab](https://github.com/shadowcz007/comfyui-sound-lab)
 
 [comfyui-Image-reward](https://github.com/shadowcz007/comfyui-Image-reward)
 
@@ -40,6 +56,8 @@ ChatGPT 节点支持 Local LLM（llama.cpp），Phi3、llama3 都可以直接一
 - 发布为 app 的 workflow，可以在右键里再次编辑了
 - web app 可以设置分类，在 comfyui 右键菜单可以编辑更新 web app
 - 支持动态提示
+- 支持把输出显示到comfyui背景（TouchDesigner 风格）
+- 如果转为web app打开是空白的，注意检查下插件目录的名字需要是：comfyui-mixlab-nodes(如果是zip包下载会多了个-main的后缀，需要去掉)
 
 ![](./assets/微信图片_20240421205440.png)
 
@@ -98,15 +116,20 @@ https://github.com/shadowcz007/comfyui-mixlab-nodes/assets/12645064/e7e77f90-e43
 
 [Voice + Real-time Face Swap Workflow](./workflow/语音+实时换脸workflow.json)
 
+- Preview Audio
+
+[text-to-audio](./workflow/text-to-audio-base-workflow.json)
+
 ### GPT
 
-> Support for calling multiple GPTs.Local LLM（llama.cpp）、 ChatGPT、ChatGLM3 、ChatGLM4 , Some code provided by rui. If you are using OpenAI's service, fill in https://api.openai.com/v1 . If you are using a local LLM service, fill in http://127.0.0.1:xxxx/v1 . Azure OpenAI:https://xxxx.openai.azure.com
+> Support for calling multiple GPTs.Local LLM 、 ChatGPT、ChatGLM3 、ChatGLM4 , Some code provided by rui. If you are using OpenAI's service, fill in https://api.openai.com/v1 . If you are using a local LLM service, fill in http://127.0.0.1:xxxx/v1 . Azure OpenAI:https://xxxx.openai.azure.com
 
-![gpt-workflow.svg](./assets/gpt-workflow.svg)
+[LLM_base_workflow](./workflow/LLM_base_workflow.json)
 
-[workflow-5](./workflow/5-gpt-workflow.json)
+- SiliconflowLLM
+- ChatGPTOpenAI
 
-最新：ChatGPT 节点支持 Local LLM（llama.cpp），Phi3、llama3 都可以直接一个节点运行了。
+<!-- 最新：ChatGPT 节点支持 Local LLM（llama.cpp），Phi3、llama3 都可以直接一个节点运行了。
 
 Model download,move to :`models/llamafile/`
 
@@ -134,7 +157,7 @@ pip install 'llama-cpp-python[server]'
 ```
 pip install llama-cpp-python \
   --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal
-```
+``` -->
 
 ## Prompt
 
@@ -161,6 +184,9 @@ pip install llama-cpp-python \
 
 > A new layer class node has been added, allowing you to separate the image into layers. After merging the images, you can input the controlnet for further processing.
 
+> The composite images node overlays a foreground image onto a background image at specified positions and scales, with optional blending modes and masking capabilities. position : 'overall',"center_center","left_bottom","center_bottom","right_bottom","left_top","center_top","right_top"
+
+
 ![layers](./assets/layers-workflow.svg)
 
 ![poster](./assets/poster-workflow.svg)
@@ -192,6 +218,12 @@ pip install llama-cpp-python \
 
 > Conveniently load images from a fixed address on the internet to ensure that default images in the workflow can be executed.
 
+#### TextImage
+
+> [下载字体](https://drxie.github.io/OSFCC/)放到 ```custom_nodes/comfyui-mixlab-nodes/assets/fonts```
+
+
+
 ### Style
 
 > Apply VisualStyle Prompting , Modified from [ComfyUI_VisualStylePrompting](https://github.com/ExponentialML/ComfyUI_VisualStylePrompting)
@@ -229,10 +261,14 @@ Add edges to an image.
 
 ![FeatheredMask](./assets/FlVou_Y6kaGWYoEj1Tn0aTd4AjMI.jpg)
 
-> LaMaInpainting
+> LaMaInpainting（需要手动安装）
+
+*  simple-lama-inpainting 里的pillow造成冲突，暂时从依赖里移除，如果有安装 simple-lama-inpainting ，节点会自动添加，没有，则不会自动添加。
 
 from [simple-lama-inpainting](https://github.com/enesmsahin/simple-lama-inpainting)
 
+* [问题汇总](https://github.com/shadowcz007/comfyui-mixlab-nodes/issues/294)
+
 > rembgNode
 
 "briarmbg","u2net","u2netp","u2net_human_seg","u2net_cloth_seg","silueta","isnet-general-use","isnet-anime"

__init__.py

@@ -1,36 +1,36 @@
+#
 import os
 import subprocess
 import importlib.util
 import sys,json
-import urllib
+import execution
+import uuid
 import hashlib
 import datetime
 import folder_paths
 import logging
 import base64,io,re
+import random
 from PIL import Image
 from comfy.cli_args import args
 python = sys.executable
 
-# print("sys.path", sys.path)
-
 #修复 sys.stdout.isatty()  object has no attribute 'isatty'
 try:
     sys.stdout.isatty()
 except:
-    # print('#fix sys.stdout.isatty')
+    print('#fix sys.stdout.isatty')
     sys.stdout.isatty = lambda: False
 
-llama_port=None
-llama_model=""
-llama_chat_format=""
+_URL_=None
 
-try:
-    from .nodes.ChatGPT import get_llama_models,get_llama_model_path,llama_cpp_client
-    llama_cpp_client("")
 
-except:
-    print("##nodes.ChatGPT ImportError")
+# try:
+#     from .nodes.ChatGPT import get_llama_models,get_llama_model_path,llama_cpp_client
+#     llama_cpp_client("")
+
+# except:
+#     print("##nodes.ChatGPT ImportError")
 
 
 from .nodes.RembgNode import get_rembg_models,U2NET_HOME,run_briarmbg,run_rembg
@@ -46,26 +46,35 @@ except ImportError:
     print("or")
     print("pip install -r requirements.txt")
     sys.exit()
-   
-def is_installed(package, package_overwrite=None):
+
+
+def is_installed(package, package_overwrite=None,auto_install=True):
+    is_has=False
     try:
         spec = importlib.util.find_spec(package)
+        is_has=spec is not None
     except ModuleNotFoundError:
         pass
 
     package = package_overwrite or package
 
     if spec is None:
-        print(f"Installing {package}...")
-        # 清华源 -i https://pypi.tuna.tsinghua.edu.cn/simple
-        command = f'"{python}" -m pip install {package}'
-  
-        result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, env=os.environ)
+        if auto_install==True:
+            print(f"Installing {package}...")
+            # 清华源 -i https://pypi.tuna.tsinghua.edu.cn/simple
+            command = f'"{python}" -m pip install {package}'
+    
+            result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, env=os.environ)
 
-        if result.returncode != 0:
-            print(f"Couldn't install\nCommand: {command}\nError code: {result.returncode}")
+            is_has=True
+
+            if result.returncode != 0:
+                print(f"Couldn't install\nCommand: {command}\nError code: {result.returncode}")
+                is_has=False
     else:
         print(package+'## OK')
+
+    return is_has
      
 try:
     import OpenSSL
@@ -88,7 +97,6 @@ except ImportError:
     sys.exit()
 
 
-
 def install_openai():
     # Helper function to install the OpenAI module if not already installed
     try:
@@ -172,15 +180,14 @@ def create_for_https():
         os.mkdir(https_key_path)
     if not os.path.exists(crt):
         create_key(key,crt)
-
-    print('https_key OK: ', crt,key)
+    # print('https_key OK: ', crt,key)
     return (crt,key)
 
 
 
 # workflow  目录下的所有json
 def read_workflow_json_files_all(folder_path):
-    print('#read_workflow_json_files_all',folder_path)
+    # print('#read_workflow_json_files_all',folder_path)
     json_files = []
     for root, dirs, files in os.walk(folder_path):
         for file in files:
@@ -310,31 +317,32 @@ def get_my_workflow_for_app(filename="my_workflow_app.json",category="",is_all=F
         print('app_workflow_path: ',app_workflow_path)
         try:
             with open(app_workflow_path) as json_file:
+                json_data=json.load(json_file)
                 apps = [{
                     'filename':filename,
-                    'data':json.load(json_file)
+                    'data':json_data
                 }]
         except Exception as e:
             print("发生异常：", str(e))
         
         # 这个代码不需要
         # if len(apps)==1 and category!='' and category!=None:
-            data=read_workflow_json_files(category_path)
+        data=read_workflow_json_files(category_path)
             
-            for item in data:
-                x=item["data"]
-                # print(apps[0]['filename'] ,item["filename"])
-                if apps[0]['filename']!=item["filename"]:
-                    category=''
-                    input=None
-                    output=None
-                    if 'category' in x['app']:
-                        category=x['app']['category']
-                    if 'input' in x['app']:
-                        input=x['app']['input']
-                    if 'output' in x['app']:
-                        output=x['app']['output']
-                    apps.append({
+        for item in data:
+            x=item["data"]
+            # print(apps[0]['filename'] ,item["filename"])
+            if apps[0]['filename']!=item["filename"]:
+                category=''
+                input=None
+                output=None
+                if 'category' in x['app']:
+                    category=x['app']['category']
+                if 'input' in x['app']:
+                    input=x['app']['input']
+                if 'output' in x['app']:
+                    output=x['app']['output']
+                apps.append({
                         "filename":item["filename"],
                         # "category":category,
                         "data":{
@@ -454,6 +462,7 @@ async def check_port_available(address, port):
 
 # https
 async def new_start(self, address, port, verbose=True, call_on_start=None):
+    global _URL_
     try:
         runner = web.AppRunner(self.app, access_log=None)
         await runner.setup()
@@ -522,10 +531,19 @@ async def new_start(self, address, port, verbose=True, call_on_start=None):
             logging.info("\n")
             logging.info("\n\nStarting server")
 
+            import socket
+
+            hostname = socket.gethostname()
+            ip_address = socket.gethostbyname(hostname)
+
+            # print(f"本机的IP地址是: {ip_address}")
+
+
             # print("\033[93mStarting server\n")
-            logging.info("\033[93mTo see the GUI go to: http://{}:{}".format(address, http_port))
-            logging.info("\033[93mTo see the GUI go to: https://{}:{}\033[0m".format(address, https_port))
- 
+            logging.info("\033[93mTo see the GUI go to: http://{}:{} or http://{}:{}".format(ip_address, http_port,address,http_port))
+            logging.info("\033[93mTo see the GUI go to: https://{}:{} or https://{}:{}\033[0m".format(ip_address, https_port,address,https_port))
+
+            _URL_="http://{}:{}".format(address,http_port)
             # print("\033[93mTo see the GUI go to: http://{}:{}".format(address, http_port))
             # print("\033[93mTo see the GUI go to: https://{}:{}\033[0m".format(address, https_port))
 
@@ -576,7 +594,7 @@ async def mixlab_app_handler(request):
             return web.Response(text=html_data, content_type='text/html')
     else:
         return web.Response(text="HTML file not found", status=404)
-
+    
 
 @routes.post('/mixlab/workflow')
 async def mixlab_workflow_hander(request):
@@ -609,13 +627,34 @@ async def mixlab_workflow_hander(request):
                     category=data['category']
                 if 'admin' in data:
                     admin=data['admin']
+
+                ds=get_my_workflow_for_app(filename,category,admin)
+                data=[]           
+                for json_data in ds:
+                    # 不传给前端
+                    if 'output' in json_data['data']:
+                        del json_data['data']['output']
+                    if 'workflow' in json_data['data']:
+                        del json_data['data']['workflow']
+                    data.append(json_data)
+
                 result={
-                    'data':get_my_workflow_for_app(filename,category,admin),
+                    'data':data,
                     'status':'success',
                 }
             elif data['task']=='list':
+                ds=get_workflows()
+                data=[]
+                for json_data in ds:
+                    # 不传给前端
+                    if 'output' in json_data['data']:
+                        del json_data['data']['output']
+                    if 'workflow' in json_data['data']:
+                        del json_data['data']['workflow']
+                    data.append(json_data)
+
                 result={
-                    'data':get_workflows(),
+                    'data':data,
                     'status':'success',
                 }
     except Exception as e:
@@ -649,11 +688,11 @@ async def get_checkpoints(request):
     except Exception as e:
         print('/mixlab/folder_paths',False,e)
 
-    try:
-        if data['type']=='llamafile':
-            names=get_llama_models()
-    except:
-        print("llamafile none")
+    # try:
+    #     if data['type']=='llamafile':
+    #         names=get_llama_models()
+    # except:
+    #     print("llamafile none")
 
     try:
         if data['type']=='rembg':
@@ -700,205 +739,141 @@ async def rembg_hander(request):
 
     return web.json_response(result)
 
-
-@routes.post("/mixlab/prompt_result")
-async def post_prompt_result(request):
-    data = await request.json()
-    res=None
-    # print(data)
-    try:
-        action=data['action']
-        if action=='save':
-            result=data['data']
-            res=save_prompt_result(result['prompt_id'],result)
-        elif action=='all':
-            res=get_prompt_result()
-    except Exception as e:
-        print('/mixlab/prompt_result',False,e)
+# 保存运行结果？暂时去掉
+# @routes.post("/mixlab/prompt_result")
+# async def post_prompt_result(request):
+#     data = await request.json()
+#     res=None
+#     # print(data)
+#     try:
+#         action=data['action']
+#         if action=='save':
+#             result=data['data']
+#             res=save_prompt_result(result['prompt_id'],result)
+#         elif action=='all':
+#             res=get_prompt_result()
+#     except Exception as e:
+#         print('/mixlab/prompt_result',False,e)
     
-    return web.json_response({"result":res})
+#     return web.json_response({"result":res})
 
+# 种子设置
+def random_seed(seed, data):
+    max_seed = 4294967295
 
-
-def start_local_live_thread(data):
-    import asyncio
-    from VoiceStreamAI.server import Server
-    from VoiceStreamAI.asr.asr_factory import ASRFactory
-    from VoiceStreamAI.vad.vad_factory import VADFactory
-
-    model="large-v3"
-    if "model" in data:
-        model=data['model']
-
-    vad_pipeline = VADFactory.create_vad_pipeline("pyannote")
-    #device 
-    asr_pipeline = ASRFactory.create_asr_pipeline("faster_whisper", **{"model_size":model})
-
-    port=8765
-    if 'port' in data:
-        port=data['port']
-
-    llm_port=9000
-    if 'llm_port' in data:
-        llm_port=data['llm_port']
-
-    server = Server(vad_pipeline, 
-                    asr_pipeline, 
-                    host="127.0.0.1", 
-                    port=port, 
-                    sampling_rate=16000, 
-                    samples_width=2,
-                    llm_port=llm_port
-                    )
-
-    loop = asyncio.new_event_loop()
-    asyncio.set_event_loop(loop)
-    loop.run_until_complete(server.start())
-    loop.run_forever()
-
-
-async def start_local_llm(data):
-    global llama_port,llama_model,llama_chat_format
-    if llama_port and llama_model and llama_chat_format:
-        return {"port":llama_port,"model":llama_model,"chat_format":llama_chat_format}
-
-    import threading
-    import uvicorn
-    from llama_cpp.server.app import create_app
-    from llama_cpp.server.settings import (
-                Settings,
-                ServerSettings,
-                ModelSettings,
-                ConfigFileSettings,
-            )
+    for id, value in data.items():
+        # print(seed,id)
+        if id in seed:
+            if 'seed' in value['inputs'] and not isinstance(value['inputs']['seed'], list) and seed[id] in ['increment', 'decrement', 'randomize']:
+                value['inputs']['seed'] = round(random.random() * max_seed)
+            
+            if 'noise_seed' in value['inputs'] and not isinstance(value['inputs']['noise_seed'], list) and seed[id] in ['increment', 'decrement', 'randomize']:
+                value['inputs']['noise_seed'] = round(random.random() * max_seed)
+            
+            if value.get('class_type') == "Seed_" and seed[id] in ['increment', 'decrement', 'randomize']:
+                value['inputs']['seed'] = round(random.random() * max_seed)
+            
+        print('new Seed', value)
     
-    if not "model" in data and "model_path" in data:
-        data['model']= os.path.basename(data["model_path"])
-        model=data["model_path"]
-
-    elif "model" in data:
-        model=get_llama_model_path(data['model'])
-
-    n_gpu_layers=-1
-
-    if "n_gpu_layers" in data:
-        n_gpu_layers=data['n_gpu_layers']
+    return data
 
 
-    chat_format="chatml"
-    if "model" in data and "function-calling" in data['model']:
-        chat_format="functionary-v2"
+# 运行工作流，代替官方的prompt接口
+@routes.post("/mixlab/prompt")
+async def mixlab_post_prompt(request):
+    p_intance=PromptServer.instance
+    logging.info("got prompt")
+    resp_code = 200
+    out_string = ""
+    json_data =  await request.json()
+    # json_data = p_intance.trigger_on_prompt(json_data)
+    # filename,category, client_id ,input
+    # workflow 的 filename,category
 
-    model_alias=os.path.basename(model)
+    # 输入的参数
+    input_data=json_data['input'] if "input" in json_data else []
+    # 种子
+    seed=json_data['seed'] if "seed" in json_data else {}
 
-    # 多模态
-    clip_model_path=None
+    apps=get_my_workflow_for_app(json_data['filename'],json_data['category'],False)
     
-    prefix = "llava-phi-3-mini"
-    file_name = prefix+"-mmproj-"
-    if model_alias.startswith(prefix):
-        for file in os.listdir(os.path.dirname(model)):
-            if file.startswith(file_name):
-                clip_model_path=os.path.join(os.path.dirname(model),file)
-                chat_format='llava-1-5'
-    print('#clip_model_path',chat_format,clip_model_path)
+    prompt=json_data['prompt'] if 'prompt' in json_data else None
+
+    if len(apps)==1:
+        # 取到prompt
+        prompt=apps[0]['data']['output']
+        # 更新input_data到prompt里
+        '''
+          {
+                "inputs": {
+                    "number": 512,
+                    "min_value": 512,
+                    "max_value": 2048,
+                    "step": 1
+                },
+                "class_type": "IntNumber",
+                "id": "22"
+            },
+        '''
+
+        for inp in input_data:
+            id=inp['id']
+            if prompt[id]['class_type']==inp['class_type']:
+                prompt[id]['inputs'].update(inp['inputs']) 
 
 
-    address="127.0.0.1"
-    port=9090
-    success = False
-    for i in range(11):  # 尝试最多11次
-        if await check_port_available(address, port + i):
-            port = port + i
-            success = True
-            break
+    if prompt==None:
+        return web.json_response({"error": "no prompt", "node_errors": []}, status=400)
+    else:
+        # 种子更新 
+        '''
+            "seed": {
+                    "45": "randomize",
+                    "46": "randomize"
+                }
+            '''
+        json_data["prompt"]=random_seed(seed,prompt)
 
-    if success == False:
-        return {"port":None,"model":""}
+    # print("#json_data",prompt)
+    # 需要把apps处理成 prompt
+    # 注意seed的处理
     
-    
-    server_settings=ServerSettings(host=address,port=port)
+    if "number" in json_data:
+        number = float(json_data['number'])
+    else:
+        number = p_intance.number
+        if "front" in json_data:
+            if json_data['front']:
+                number = -number
 
-    name, ext = os.path.splitext(os.path.basename(model))
-    print('#model',name)
-    app = create_app(
-                server_settings=server_settings,
-                model_settings=[
-                    ModelSettings(
-                    model=model,
-                    model_alias=name,
-                    n_gpu_layers=n_gpu_layers,
-                    n_ctx=4098,
-                    chat_format=chat_format,
-                    embedding=False,
-                    clip_model_path=clip_model_path
-                    )])
+        p_intance.number += 1
 
-    def run_uvicorn():
-        uvicorn.run(
-                app,
-                host=os.getenv("HOST", server_settings.host),
-                port=int(os.getenv("PORT", server_settings.port)),
-                ssl_keyfile=server_settings.ssl_keyfile,
-                ssl_certfile=server_settings.ssl_certfile,
-            )
+    if "prompt" in json_data:
+        prompt = json_data["prompt"]
+        valid = execution.validate_prompt(prompt)
+        extra_data = {}
+        if "extra_data" in json_data:
+            extra_data = json_data["extra_data"]
 
-    # 创建一个子线程
-    thread = threading.Thread(target=run_uvicorn)
-
-    # 启动子线程
-    thread.start()
-
-    llama_port=port
-    llama_model=data['model']
-    llama_chat_format=chat_format
-
-    return  {"port":llama_port,"model":llama_model,"chat_format":llama_chat_format}
-
-# llam服务的开启
-@routes.post('/mixlab/start_llama')
-async def my_hander_method(request):
-    data =await request.json()
-    # print(data)
-    if llama_port and llama_model and llama_chat_format:
-        return web.json_response({"port":llama_port,"model":llama_model,"chat_format":llama_chat_format} )
-    try:
-        result=await start_local_llm(data)
-    except:
-        result= {"port":None,"model":"","llama_cpp_error":True}
-        print('start_local_llm error')
-
-    return web.json_response(result)
+        if "client_id" in json_data:
+            extra_data["client_id"] = json_data["client_id"]
+        if valid[0]:
+            prompt_id = str(uuid.uuid4())
+            outputs_to_execute = valid[2]
+            p_intance.prompt_queue.put((number, prompt_id, prompt, extra_data, outputs_to_execute))
+            response = {"prompt_id": prompt_id, "number": number, "node_errors": valid[3]}
+            return web.json_response(response)
+        else:
+            logging.warning("invalid prompt: {}".format(valid[1]))
+            return web.json_response({"error": valid[1], "node_errors": valid[3]}, status=400)
+    else:
+        return web.json_response({"error": "no prompt", "node_errors": []}, status=400)
 
 
-@routes.post('/mixlab/start_live')
-async def mixlab_live_start_handler(request):
-    import threading
-    llm=await start_local_llm({
-        "model":"Phi-3-mini-4k-instruct-Q5_K_S.gguf",
-        "n_gpu_layers":2
-    })
-
-    # {"port":llama_port,"model":llama_model,"chat_format":llama_chat_format}
-
-    os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com' #hf_hub_download 里的下载地址修改
-    os.environ['PYANNOTE_AUTH_TOKEN'] = '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'
-
-    # Create and start the thread
-    data = {
-        "llm_port":llm['port'],
-        "port":8725,
-        "model":"large-v3"
-    }  # Replace with your actual data if needed
-    thread = threading.Thread(target=start_local_live_thread, args=(data,))
-    thread.start()
-
-    return web.json_response(data)
-
-
-@routes.get('/mixlab/live')
-async def mixlab_live_handler(request):
-    html_file = os.path.join(current_path, "web/live.html")
+# AR页面
+# @routes.get('/mixlab/AR')
+async def handle_ar_page(request):
+    html_file = os.path.join(current_path, "web/ar.html")
     if os.path.exists(html_file):
         with open(html_file, 'r', encoding='utf-8', errors='ignore') as f:
             html_data = f.read()
@@ -906,6 +881,118 @@ async def mixlab_live_handler(request):
     else:
         return web.Response(text="HTML file not found", status=404)
 
+
+# async def start_local_llm(data):
+#     global llama_port,llama_model,llama_chat_format
+#     if llama_port and llama_model and llama_chat_format:
+#         return {"port":llama_port,"model":llama_model,"chat_format":llama_chat_format}
+
+#     import threading
+#     import uvicorn
+#     from llama_cpp.server.app import create_app
+#     from llama_cpp.server.settings import (
+#                 Settings,
+#                 ServerSettings,
+#                 ModelSettings,
+#                 ConfigFileSettings,
+#             )
+    
+#     if not "model" in data and "model_path" in data:
+#         data['model']= os.path.basename(data["model_path"])
+#         model=data["model_path"]
+
+#     elif "model" in data:
+#         model=get_llama_model_path(data['model'])
+
+#     n_gpu_layers=-1
+
+#     if "n_gpu_layers" in data:
+#         n_gpu_layers=data['n_gpu_layers']
+
+
+#     chat_format="chatml"
+
+#     model_alias=os.path.basename(model)
+
+#     # 多模态
+#     clip_model_path=None
+    
+#     prefix = "llava-phi-3-mini"
+#     file_name = prefix+"-mmproj-"
+#     if model_alias.startswith(prefix):
+#         for file in os.listdir(os.path.dirname(model)):
+#             if file.startswith(file_name):
+#                 clip_model_path=os.path.join(os.path.dirname(model),file)
+#                 chat_format='llava-1-5'
+#     # print('#clip_model_path',chat_format,clip_model_path,model)
+
+#     address="127.0.0.1"
+#     port=9090
+#     success = False
+#     for i in range(11):  # 尝试最多11次
+#         if await check_port_available(address, port + i):
+#             port = port + i
+#             success = True
+#             break
+
+#     if success == False:
+#         return {"port":None,"model":""}
+    
+    
+#     server_settings=ServerSettings(host=address,port=port)
+
+#     name, ext = os.path.splitext(os.path.basename(model))
+#     if name:
+#         # print('#model',name)
+#         app = create_app(
+#                     server_settings=server_settings,
+#                     model_settings=[
+#                         ModelSettings(
+#                         model=model,
+#                         model_alias=name,
+#                         n_gpu_layers=n_gpu_layers,
+#                         n_ctx=4098,
+#                         chat_format=chat_format,
+#                         embedding=False,
+#                         clip_model_path=clip_model_path
+#                         )])
+
+#         def run_uvicorn():
+#             uvicorn.run(
+#                     app,
+#                     host=os.getenv("HOST", server_settings.host),
+#                     port=int(os.getenv("PORT", server_settings.port)),
+#                     ssl_keyfile=server_settings.ssl_keyfile,
+#                     ssl_certfile=server_settings.ssl_certfile,
+#                 )
+
+#         # 创建一个子线程
+#         thread = threading.Thread(target=run_uvicorn)
+
+#         # 启动子线程
+#         thread.start()
+
+#         llama_port=port
+#         llama_model=data['model']
+#         llama_chat_format=chat_format
+
+#     return  {"port":llama_port,"model":llama_model,"chat_format":llama_chat_format}
+
+# llam服务的开启
+# @routes.post('/mixlab/start_llama')
+# async def my_hander_method(request):
+#     data =await request.json()
+#     # print(data)
+#     if llama_port and llama_model and llama_chat_format:
+#         return web.json_response({"port":llama_port,"model":llama_model,"chat_format":llama_chat_format} )
+#     try:
+#         result=await start_local_llm(data)
+#     except:
+#         result= {"port":None,"model":"","llama_cpp_error":True}
+#         print('start_local_llm error')
+
+#     return web.json_response(result)
+
 # 重启服务
 @routes.post('/mixlab/re_start')
 def re_start(request):
@@ -915,24 +1002,24 @@ def re_start(request):
         pass
     return os.execv(sys.executable, [sys.executable] + sys.argv)
 
-
+# 状态
+@routes.get('/mixlab/status')
+def mix_status(request):
+    return web.Response(text="running#"+_URL_)
 
 # 导入节点
 from .nodes.PromptNode import GLIGENTextBoxApply_Advanced,EmbeddingPrompt,RandomPrompt,PromptSlide,PromptSimplification,PromptImage,JoinWithDelimiter
-from .nodes.ImageNode import ComparingTwoFrames,LoadImages_,CompositeImages,GridDisplayAndSave,GridInput,ImagesPrompt,SaveImageAndMetadata,SaveImageToLocal,SplitImage,GridOutput,GetImageSize_,MirroredImage,ImageColorTransfer,NoiseImage,TransparentImage,GradientImage,LoadImagesFromPath,LoadImagesFromURL,ResizeImage,TextImage,SvgImage,Image3D,ShowLayer,NewLayer,MergeLayers,CenterImage,AreaToMask,SmoothMask,SplitLongMask,ImageCropByAlpha,EnhanceImage,FaceToMask
+from .nodes.ImageNode import ImageBatchToList_,ImageListToBatch_,ComparingTwoFrames,LoadImages_,CompositeImages,GridDisplayAndSave,GridInput,ImagesPrompt,SaveImageAndMetadata,SaveImageToLocal,SplitImage,GridOutput,GetImageSize_,MirroredImage,ImageColorTransfer,NoiseImage,TransparentImage,GradientImage,LoadImagesFromPath,LoadImagesFromURL,ResizeImage,TextImage,SvgImage,Image3D,ShowLayer,NewLayer,MergeLayers,CenterImage,AreaToMask,SmoothMask,SplitLongMask,ImageCropByAlpha,EnhanceImage,FaceToMask
 # from .nodes.Vae import VAELoader,VAEDecode
 from .nodes.ScreenShareNode import ScreenShareNode,FloatingVideo
 
-from .nodes.ChatGPT import ChatGPTNode,ShowTextForGPT,CharacterInText,TextSplitByDelimiter
-from .nodes.Audio import GamePal,SpeechRecognition,SpeechSynthesis
-from .nodes.Utils import IncrementingListNode,ListSplit,CreateLoraNames,CreateSampler_names,CreateCkptNames,CreateSeedNode,TESTNODE_,TESTNODE_TOKEN,AppInfo,IntNumber,FloatSlider,TextInput,ColorInput,FontInput,TextToNumber,DynamicDelayProcessor,LimitNumber,SwitchByIndex,MultiplicationNode
+from .nodes.Audio import AudioPlayNode,SpeechRecognition,SpeechSynthesis
+from .nodes.Utils import KeyInput,IncrementingListNode,ListSplit,CreateLoraNames,CreateSampler_names,CreateCkptNames,CreateSeedNode,TESTNODE_,TESTNODE_TOKEN,AppInfo,IntNumber,FloatSlider,TextInput,ColorInput,FontInput,TextToNumber,DynamicDelayProcessor,LimitNumber,SwitchByIndex,MultiplicationNode
 from .nodes.Mask import PreviewMask_,MaskListReplace,MaskListMerge,OutlineMask,FeatheredMask
 
 from .nodes.Style import ApplyVisualStylePrompting,StyleAlignedReferenceSampler,StyleAlignedBatchAlign,StyleAlignedSampleReferenceLatents
 
-from .nodes.Video import VideoCombine_Adv,LoadVideoAndSegment,ImageListReplace,VAEEncodeForInpaint_Frames
-
-from .nodes.TripoSR import LoadTripoSRModel,TripoSRSampler,SaveTripoSRMesh
+from .nodes.P5 import P5Input
 
 
 # 要导出的所有节点及其名称的字典
@@ -963,6 +1050,8 @@ NODE_CLASS_MAPPINGS = {
     "ImageColorTransfer":ImageColorTransfer,
     "ShowLayer":ShowLayer,
     "NewLayer":NewLayer,
+    "ImageListToBatch_":ImageListToBatch_,
+    "ImageBatchToList_":ImageBatchToList_,
     "CompositeImages_":CompositeImages,
     "SplitImage":SplitImage,
     "CenterImage":CenterImage,
@@ -984,12 +1073,10 @@ NODE_CLASS_MAPPINGS = {
     # "VAEDecodeConsistencyDecoder":VAEDecode,
     "ScreenShare":ScreenShareNode,
     "FloatingVideo":FloatingVideo,
-    "ChatGPTOpenAI":ChatGPTNode,
-    "ShowTextForGPT":ShowTextForGPT,
-    "CharacterInText":CharacterInText,
-    "TextSplitByDelimiter":TextSplitByDelimiter,
+   
     "SpeechRecognition":SpeechRecognition,
     "SpeechSynthesis":SpeechSynthesis,
+    "KeyInput":KeyInput,
     "Color":ColorInput,
     "FloatSlider":FloatSlider,
     "IntNumber":IntNumber,
@@ -1011,26 +1098,26 @@ NODE_CLASS_MAPPINGS = {
     "ApplyVisualStylePrompting_":ApplyVisualStylePrompting,
     "StyleAlignedReferenceSampler_": StyleAlignedReferenceSampler,
     "StyleAlignedSampleReferenceLatents_": StyleAlignedSampleReferenceLatents,
-    "StyleAlignedBatchAlign_": StyleAlignedBatchAlign,
-    "LoadVideoAndSegment_":LoadVideoAndSegment,
-    "VideoCombine_Adv":VideoCombine_Adv,
+    "StyleAlignedBatchAlign_": StyleAlignedBatchAlign, 
     "ListSplit_":ListSplit,
-    "MaskListReplace_":MaskListReplace,
-    "ImageListReplace_":ImageListReplace,
-    "VAEEncodeForInpaint_Frames":VAEEncodeForInpaint_Frames,
+    "MaskListReplace_":MaskListReplace, 
     "IncrementingListNode_":IncrementingListNode,
     "PreviewMask_":PreviewMask_,
-     "LoadTripoSRModel_": LoadTripoSRModel,
-    "TripoSRSampler_": TripoSRSampler,
-    "SaveTripoSRMesh": SaveTripoSRMesh
-    # "GamePal":GamePal
+    "AudioPlay":AudioPlayNode,
+
+    "P5Input":P5Input
 }
 
 # 一个包含节点友好/可读的标题的字典
 NODE_DISPLAY_NAME_MAPPINGS = {
     "AppInfo":"App Info ♾️MixlabApp",
+    "ScreenShare":"Screen Share ♾️Mixlab",
+    "FloatingVideo":"Floating Video ♾️Mixlab",
+    "TextImage":"Text Image ♾️Mixlab",
+    
     "Color":"Color Input ♾️MixlabApp",
     "TextInput_":"Text Input ♾️MixlabApp",
+    "KeyInput":"API Key Input ♾️MixlabApp",
     "FloatSlider":"Float Slider Input ♾️MixlabApp",
     "IntNumber":"Int Input ♾️MixlabApp",
     "ImagesPrompt_":"Images Input ♾️MixlabApp",
@@ -1042,14 +1129,14 @@ NODE_DISPLAY_NAME_MAPPINGS = {
     "SplitLongMask":"Splitting a long image into sections",
     "VAELoaderConsistencyDecoder":"Consistency Decoder Loader",
     "VAEDecodeConsistencyDecoder":"Consistency Decoder Decode",
-    "ScreenShare":"Screen Share ♾️Mixlab",
-    "FloatingVideo":"FloatingVideo ♾️Mixlab",
-    "ChatGPTOpenAI":"ChatGPT & Local LLM ♾️Mixlab",
-    "ShowTextForGPT":"Show Text ♾️MixlabApp",
+    
+    
     "MergeLayers":"Merge Layers ♾️Mixlab",
     "SpeechSynthesis":"SpeechSynthesis ♾️Mixlab",
     "SpeechRecognition":"SpeechRecognition ♾️Mixlab",
     "3DImage":"3DImage ♾️Mixlab",
+    "ImageListToBatch_":"Image List To Batch",
+    "ImageBatchToList_":"Image Batch To List",
     "CompositeImages_":"Composite Images ♾️Mixlab",
     "DynamicDelayProcessor":"DynamicDelayByText ♾️Mixlab",
     "LaMaInpainting":"LaMaInpainting ♾️Mixlab",
@@ -1075,28 +1162,69 @@ NODE_DISPLAY_NAME_MAPPINGS = {
     "GridInput":"Grid Input ♾️Mixlab",
     "GridOutput":"Grid Output ♾️Mixlab",
     "GetImageSize_":"Get Image Size ♾️Mixlab",
-    "VAEEncodeForInpaint_Frames":"VAE Encode For Inpaint Frames ♾️Mixlab",
     "IncrementingListNode_":"Create Incrementing Number List ♾️Mixlab",
     "LoadImagesToBatch":"Load Images(base64) ♾️Mixlab",
     "PreviewMask_":"Preview Mask",
-    "LoadTripoSRModel_": "Load TripoSR Model",
-    "TripoSRSampler_": "TripoSR Sampler",
-    "SaveTripoSRMesh": "Save TripoSR Mesh"
+    "AudioPlay":"Preview Audio ♾️Mixlab",
+
+     "MultiplicationNode":"Math Operation ♾️Mixlab",
+
+     "P5Input":"P5 Input ♾️Mixlab for test"
 }
 
 # web ui的节点功能
 WEB_DIRECTORY = "./web"
 
-
 logging.info('--------------')
 logging.info('\033[91m ### Mixlab Nodes: \033[93mLoaded')
 # print('\033[91m ### Mixlab Nodes: \033[93mLoaded')
 
 try:
-    from .nodes.Lama import LaMaInpainting
-    logging.info('LaMaInpainting.available {}'.format(LaMaInpainting.available))
-    if LaMaInpainting.available:
-        NODE_CLASS_MAPPINGS['LaMaInpainting']=LaMaInpainting
+    from .nodes.ChatGPT import JsonRepair,ChatGPTNode,ShowTextForGPT,CharacterInText,TextSplitByDelimiter,SiliconflowFreeNode
+    logging.info('ChatGPT.available True')
+
+    NODE_CLASS_MAPPINGS_V = { 
+       "ChatGPTOpenAI":ChatGPTNode,
+       "SiliconflowLLM":SiliconflowFreeNode,
+        "ShowTextForGPT":ShowTextForGPT,
+        "CharacterInText":CharacterInText,
+        "TextSplitByDelimiter":TextSplitByDelimiter,
+        "JsonRepair":JsonRepair
+    }
+
+    # 一个包含节点友好/可读的标题的字典
+    NODE_DISPLAY_NAME_MAPPINGS_V = { 
+        "ChatGPTOpenAI":"ChatGPT & Local LLM ♾️Mixlab",
+        "SiliconflowLLM":"LLM Siliconflow ♾️Mixlab",
+        "ShowTextForGPT":"Show Text ♾️MixlabApp",
+        "CharacterInText":"Character In Text",
+        "TextSplitByDelimiter":"Text Split By Delimiter",
+        "JsonRepair":"Json Repair"
+    }
+
+
+    NODE_CLASS_MAPPINGS.update(NODE_CLASS_MAPPINGS_V)
+    NODE_DISPLAY_NAME_MAPPINGS.update(NODE_DISPLAY_NAME_MAPPINGS_V)
+
+except Exception as e:
+    logging.info('ChatGPT.available False')
+
+
+try:
+    from .nodes.edit_mask import EditMask
+    logging.info('edit_mask.available True')
+    NODE_CLASS_MAPPINGS['EditMask']=EditMask
+    NODE_DISPLAY_NAME_MAPPINGS['EditMask']="Edit Mask ♾️Mixlab"
+except Exception as e:
+    logging.info('edit_mask.available False')
+
+try:
+    is_has=is_installed('simple_lama_inpainting',None,False)
+    if is_has:
+        from .nodes.Lama import LaMaInpainting
+        logging.info('LaMaInpainting.available {}'.format(LaMaInpainting.available))
+        if LaMaInpainting.available:
+            NODE_CLASS_MAPPINGS['LaMaInpainting']=LaMaInpainting
 except Exception as e:
     logging.info('LaMaInpainting.available False')
 
@@ -1127,4 +1255,57 @@ try:
 except Exception as e:
     logging.info('RembgNode_.available False' )
 
+
+try:
+    from .nodes.Video import GenerateFramesByCount,scenesNode_,CombineAudioVideo,VideoCombine_Adv,LoadVideoAndSegment,ImageListReplace,VAEEncodeForInpaint_Frames,LoadAndCombinedAudio_
+    
+    NODE_CLASS_MAPPINGS_V = {
+        "VAEEncodeForInpaint_Frames":VAEEncodeForInpaint_Frames,
+        "ImageListReplace_":ImageListReplace,
+        "LoadVideoAndSegment_":LoadVideoAndSegment,
+        "VideoCombine_Adv":VideoCombine_Adv,
+        "LoadAndCombinedAudio_":LoadAndCombinedAudio_,
+        "CombineAudioVideo":CombineAudioVideo,
+        "ScenesNode_":scenesNode_,
+        "GenerateFramesByCount":GenerateFramesByCount 
+    }
+
+    # 一个包含节点友好/可读的标题的字典
+    NODE_DISPLAY_NAME_MAPPINGS_V = {
+        "VAEEncodeForInpaint_Frames":"VAE Encode For Inpaint Frames ♾️Mixlab",
+        "ImageListReplace_":"Image List Replace",
+        "LoadVideoAndSegment_":"Load Video And Segment",
+        "VideoCombine_Adv":"Video Combine",
+        "LoadAndCombinedAudio_":"Load And Combined Audio",
+        "CombineAudioVideo":"Combine Audio Video",
+        "ScenesNode_":"Select Scene", 
+        "GenerateFramesByCount":"Generate Frames By Count"
+    }
+
+
+    NODE_CLASS_MAPPINGS.update(NODE_CLASS_MAPPINGS_V)
+    NODE_DISPLAY_NAME_MAPPINGS.update(NODE_DISPLAY_NAME_MAPPINGS_V)
+
+except:
+    logging.info('Video.available False')
+
+
+try:
+    from .nodes.TripoSR import LoadTripoSRModel,TripoSRSampler,SaveTripoSRMesh
+    logging.info('TripoSR.available')
+    # logging.info( folder_paths.get_temp_directory())
+    NODE_CLASS_MAPPINGS['LoadTripoSRModel_']=LoadTripoSRModel
+    NODE_DISPLAY_NAME_MAPPINGS["LoadTripoSRModel_"]= "Load TripoSR Model"
+    
+    NODE_CLASS_MAPPINGS['TripoSRSampler_']=TripoSRSampler
+    NODE_DISPLAY_NAME_MAPPINGS["TripoSRSampler_"]= "TripoSR Sampler"
+
+    NODE_CLASS_MAPPINGS['SaveTripoSRMesh']=SaveTripoSRMesh
+    NODE_DISPLAY_NAME_MAPPINGS["SaveTripoSRMesh"]= "Save TripoSR Mesh"
+    
+
+except Exception as e:
+    logging.info('TripoSR.available False' )
+
+
 logging.info('\033[93m -------------- \033[0m')

install.bat

@@ -11,9 +11,9 @@ if exist "%python_exec%" (
         %python_exec% -s -m pip install "%%i" -i https://pypi.tuna.tsinghua.edu.cn/simple
     )
 
-    %python_exec% -s -m pip install --upgrade --force llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121
+    @REM %python_exec% -s -m pip install --upgrade --force llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121
 
-    %python_exec% -s -m pip install --upgrade --force llama-cpp-python[server]
+    @REM %python_exec% -s -m pip install --upgrade --force llama-cpp-python[server]
     
 
 ) else (

nodes/Audio.py

@@ -1,6 +1,7 @@
 
-
-
+import os
+import folder_paths
+import torchaudio
 
 class SpeechRecognition:
     @classmethod
@@ -55,46 +56,65 @@ class SpeechSynthesis:
         return {"ui": {"text": text}, "result": (text,)}
     
 
-#
-class GamePal:
+
+class AudioPlayNode:
+    def __init__(self):
+        self.output_dir = folder_paths.get_temp_directory()
+        self.type = "temp"
+        self.prefix_append = ""
+        self.compress_level = 4
     @classmethod
     def INPUT_TYPES(s):
-        return {
-            "required": {
-                "input_text": ("STRING",{"multiline": True,"default": ""}),
-            },
-            "optional": {
-                 
-                "input_num": ("INT",{
-                                "default":100, 
-                                "min": -1, #Minimum value
-                                "max": 0xffffffffffffffff, #Maximum value
-                                "step": 1, #Slider's step
-                                "display": "slider" # Cosmetic only: display as "number" or "slider"
-                                }), 
-                "python_code": ("STRING",{"multiline": True,"default": "result= 1 if 'Mixlab' in input_text else 0"}),
-            }
-        }
-
-    INPUT_IS_LIST = False
-    RETURN_TYPES = ("INT",)
+        return {"required": {
+            "audio": ("AUDIO",),
+              }, 
+                }
+    
+    RETURN_TYPES = ()
+  
     FUNCTION = "run"
-    OUTPUT_NODE = True
-    OUTPUT_IS_LIST = (False,)
 
     CATEGORY = "♾️Mixlab/Audio"
 
-    def run(self, input_text,input_num,python_code):
-        exec(python_code)
-        res=None
-        try:
-            # 可能会引发异常的代码
-            res=result
-        except:
-            # 处理异常的代码
-            print('')
+    INPUT_IS_LIST = False
+    OUTPUT_IS_LIST = ()
 
-        print(res)
+    OUTPUT_NODE = True
+  
+    def run(self,audio):
 
-        # print(session_history)
-        return {"ui": {"text": [input_text],"num":[input_num]}, "result": (res,)}
+        # 判断是否是 Tensor 类型
+        is_tensor = not isinstance(audio, dict)
+        # print('#判断是否是 Tensor 类型',is_tensor,audio)
+        if not is_tensor and 'waveform' in audio and 'sample_rate' in audio:
+            # {'waveform': tensor([], size=(1, 1, 0)), 'sample_rate': 44100}
+            is_tensor=True
+
+        if is_tensor and (not 'audio_path' in audio):
+            filename_prefix=""
+            # 保存
+            filename_prefix += self.prefix_append
+            full_output_folder, filename, counter, subfolder, filename_prefix = folder_paths.get_save_image_path(filename_prefix, self.output_dir)
+            results = list()
+            
+            filename_with_batch_num = filename.replace("%batch_num%", str(1))
+            file = f"{filename_with_batch_num}_{counter:05}_.wav"
+            
+            torchaudio.save(os.path.join(full_output_folder, file), audio['waveform'].squeeze(0), audio["sample_rate"])
+            results.append({
+                    "filename": file,
+                    "subfolder": subfolder,
+                    "type": self.type
+                })
+            
+        else:
+            results=[{
+                    "filename": audio['filename'],
+                    "subfolder":audio['subfolder'],
+                    "type": audio['type'],
+                    "audio_path":audio['audio_path']
+                }]
+                
+
+        # print(audio)
+        return {"ui": {"audio":results}}

nodes/ChatGPT.py

@@ -6,14 +6,69 @@ import folder_paths
 import hashlib
 import codecs,sys
 import importlib.util
+import subprocess
+
+python = sys.executable
+
+# 从文本中提取json
+def extract_json_strings(text):
+    json_strings = []
+    brace_level = 0
+    json_str = ''
+    in_json = False
+    
+    for char in text:
+        if char == '{':
+            brace_level += 1
+            in_json = True
+        if in_json:
+            json_str += char
+        if char == '}':
+            brace_level -= 1
+        if in_json and brace_level == 0:
+            json_strings.append(json_str)
+            json_str = ''
+            in_json = False
+
+    return json_strings[0] if len(json_strings)>0 else "{}"
 
 
-def is_installed(package):
+def is_installed(package, package_overwrite=None,auto_install=True):
+    is_has=False
     try:
         spec = importlib.util.find_spec(package)
+        is_has=spec is not None
     except ModuleNotFoundError:
-        return False
-    return spec is not None
+        pass
+
+    package = package_overwrite or package
+
+    if spec is None:
+        if auto_install==True:
+            print(f"Installing {package}...")
+            # 清华源 -i https://pypi.tuna.tsinghua.edu.cn/simple
+            command = f'"{python}" -m pip install {package}'
+    
+            result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, env=os.environ)
+
+            is_has=True
+
+            if result.returncode != 0:
+                print(f"Couldn't install\nCommand: {command}\nError code: {result.returncode}")
+                is_has=False
+    else:
+        print(package+'## OK')
+
+    return is_has
+  
+
+
+# def is_installed(package):
+#     try:
+#         spec = importlib.util.find_spec(package)
+#     except ModuleNotFoundError:
+#         return False
+#     return spec is not None
 
 
 def get_unique_hash(string):
@@ -53,30 +108,14 @@ def azure_client(key,url):
 
 def openai_client(key,url):
     client = openai.OpenAI(
-    api_key=key,
-    base_url=url
+        api_key=key,
+        base_url=url
     )
     return client
 
 def ZhipuAI_client(key):
-
     try:
-        if is_installed('zhipuai')==False:
-            import subprocess
-
-            # 安装
-            print('#pip install zhipuai')
-
-            result = subprocess.run([sys.executable, '-s', '-m', 'pip', 'install', 'zhipuai'], capture_output=True, text=True)
-
-            #检查命令执行结果
-            if result.returncode == 0:
-                print("#install success")
-                from zhipuai import ZhipuAI
-            else:
-                print("#install error")
-            
-        else:
+        if is_installed('zhipuai')==True:
             from zhipuai import ZhipuAI
     except:
         print("#install zhipuai error")
@@ -97,73 +136,76 @@ def get_llama_path():
     except:
         return os.path.join(folder_paths.models_dir, "llamafile")
 
-def get_llama_models():
-    res=[]
+# def get_llama_models():
+#     res=[]
 
-    model_path=get_llama_path()
-    if os.path.exists(model_path):
-        files = os.listdir(model_path)
-        for file in files:
-            if os.path.isfile(os.path.join(model_path, file)):
-                res.append(file)
-        res=phi_sort(res)
-    return res
+#     model_path=get_llama_path()
+#     if os.path.exists(model_path):
+#         files = os.listdir(model_path)
+#         for file in files:
+#             if os.path.isfile(os.path.join(model_path, file)):
+#                 res.append(file)
+#         res=phi_sort(res)
+#     return res
 
-llama_modes_list=get_llama_models()
+# llama_modes_list=get_llama_models()
+# llama_modes_list=[]
 
-def get_llama_model_path(file_name):
-    model_path=get_llama_path()
-    mp=os.path.join(model_path,file_name)
-    return mp
+# def get_llama_model_path(file_name):
+#     model_path=get_llama_path()
+#     mp=os.path.join(model_path,file_name)
+#     return mp
 
-def llama_cpp_client(file_name):
-    try:
-        if is_installed('llama_cpp')==False:
-            import subprocess
+# def llama_cpp_client(file_name):
+#     try:
+#         if is_installed('llama_cpp')==False:
+#             import subprocess
 
-            # 安装
-            print('#pip install llama-cpp-python')
+#             # 安装
+#             print('#pip install llama-cpp-python')
  
-            result = subprocess.run([sys.executable, '-s', '-m', 'pip', 
-                                     'install', 
-                                     'llama-cpp-python',
-                                     '--extra-index-url',
-                                     'https://abetlen.github.io/llama-cpp-python/whl/cu121'
-                                     ], capture_output=True, text=True)
+#             result = subprocess.run([sys.executable, '-s', '-m', 'pip', 
+#                                      'install', 
+#                                      'llama-cpp-python',
+#                                      '--extra-index-url',
+#                                      'https://abetlen.github.io/llama-cpp-python/whl/cu121'
+#                                      ], capture_output=True, text=True)
 
-            #检查命令执行结果
-            if result.returncode == 0:
-                print("#install success")
-                from llama_cpp import Llama
+#             #检查命令执行结果
+#             if result.returncode == 0:
+#                 print("#install success")
+#                 from llama_cpp import Llama
 
-                subprocess.run([sys.executable, '-s', '-m', 'pip', 
-                                     'install', 
-                                     'llama-cpp-python[server]'
-                                     ], capture_output=True, text=True)
+#                 subprocess.run([sys.executable, '-s', '-m', 'pip', 
+#                                      'install', 
+#                                      'llama-cpp-python[server]'
+#                                      ], capture_output=True, text=True)
 
-            else:
-                print("#install error")
+#             else:
+#                 print("#install error")
             
-        else:
-            from llama_cpp import Llama
-    except:
-        print("#install llama-cpp-python error")
+#         else:
+#             from llama_cpp import Llama
+#     except:
+#         print("#install llama-cpp-python error")
 
-    if file_name:
-        mp=get_llama_model_path(file_name)
-        # file_name=get_llama_models()[0]
-        # model_path=os.path.join(folder_paths.models_dir, "llamafile")
-        # mp=os.path.join(model_path,file_name)
+#     if file_name:
+#         mp=get_llama_model_path(file_name)
+#         # file_name=get_llama_models()[0]
+#         # model_path=os.path.join(folder_paths.models_dir, "llamafile")
+#         # mp=os.path.join(model_path,file_name)
 
-        llm = Llama(model_path=mp, chat_format="chatml",n_gpu_layers=-1,n_ctx=512)
+#         llm = Llama(model_path=mp, chat_format="chatml",n_gpu_layers=-1,n_ctx=512)
 
-        return llm
+#         return llm
 
-    
+
+if is_installed('json_repair'):
+    from json_repair import repair_json
 
 
 def chat(client, model_name,messages ):
-
+        print('#chat',model_name,messages)
         try_count = 0
         while True:
             try_count += 1
@@ -206,6 +248,36 @@ def chat(client, model_name,messages ):
         return content
 
 
+llm_apis=[
+            {
+                "value": "https://api.openai.com/v1",
+                "label": "openai"
+            },
+            {
+                "value": "https://openai.api2d.net/v1",
+                "label": "api2d"
+            },
+            # {
+            #     "value": "https://docs-test-001.openai.azure.com",
+            #     "label": "https://docs-test-001.openai.azure.com"
+            # },
+             
+            {
+                "value": "https://api.moonshot.cn/v1",
+                "label": "Kimi"
+            },
+            {
+                "value": "https://api.deepseek.com/v1",
+                "label": "DeepSeek-V2"
+            },
+            {
+                "value": "https://api.siliconflow.cn/v1",
+                "label": "SiliconCloud"
+            }]
+        
+llm_apis_dict = {api["label"]: api["value"] for api in llm_apis}
+
+
 class ChatGPTNode:
     def __init__(self):
         # self.__client = OpenAI()
@@ -215,35 +287,60 @@ class ChatGPTNode:
 
     @classmethod
     def INPUT_TYPES(cls):
-        model_list=llama_modes_list+[
-                    "gpt-3.5-turbo",
-                    "gpt-3.5-turbo-0125",
-                    "gpt-35-turbo",
-                    "gpt-3.5-turbo-16k", 
-                    "gpt-3.5-turbo-16k-0613", 
-                    "gpt-4-0613",
-                    "gpt-4-1106-preview",
-                    "glm-4"
+
+        model_list=[ 
+            "gpt-3.5-turbo",
+            "gpt-3.5-turbo-16k",
+            "gpt-4o",
+            "gpt-4o-2024-05-13",
+            "gpt-4",
+            "gpt-4-0314",
+            "gpt-4-0613",
+            "gpt-3.5-turbo-0301",
+            "gpt-3.5-turbo-0613",
+            "gpt-3.5-turbo-16k-0613",
+            "qwen-turbo",
+            "qwen-plus",
+            "qwen-long",
+            "qwen-max",
+            "qwen-max-longcontext",
+            "glm-4",
+            "glm-3-turbo",
+            "moonshot-v1-8k",
+            "moonshot-v1-32k",
+            "moonshot-v1-128k",
+            "deepseek-chat",
+            "Qwen/Qwen2-7B-Instruct",
+            "THUDM/glm-4-9b-chat",
+            "01-ai/Yi-1.5-9B-Chat-16K",
+            "meta-llama/Meta-Llama-3.1-8B-Instruct"
                     ]
+        
         return {
             "required": {
-                "api_key":("KEY", {"default": "", "multiline": True,"dynamicPrompts": False}),
-                "api_url":("URL", {"default": "", "multiline": True,"dynamicPrompts": False}),
+                # "api_key":("KEY", {"default": "", "multiline": True,"dynamicPrompts": False}),
+                # "api_key":("STRING", {"forceInput": True,}),
+               
                 "prompt": ("STRING", {"multiline": True,"dynamicPrompts": False}),
                 "system_content": ("STRING", 
                                    {
                                        "default": "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.", 
                                        "multiline": True,"dynamicPrompts": False
                                        }),
+                 
                 "model": ( model_list, 
                     {"default": model_list[0]}),
                 "seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff, "step": 1}),
                 "context_size":("INT", {"default": 1, "min": 0, "max":30, "step": 1}),
+                "api_url":(list(llm_apis_dict.keys()), 
+                    {"default": list(llm_apis_dict.keys())[0]}),
             },
-             "hidden": {
-                "unique_id": "UNIQUE_ID",
-                "extra_pnginfo": "EXTRA_PNGINFO",
-            },
+             "optional":{
+                    "api_key":("STRING", {"forceInput": True,}),
+                    "custom_model_name":("STRING", {"forceInput": True,}), #适合自定义model
+                     "custom_api_url":("STRING", {"forceInput": True,}), #适合自定义model
+                },
+             
         }
 
     RETURN_TYPES = ("STRING","STRING","STRING",)
@@ -255,12 +352,29 @@ class ChatGPTNode:
 
     
     def generate_contextual_text(self,
-                                 api_key,
-                                 api_url, 
+                                #  api_key,
                                  prompt, 
                                  system_content,
-                                model, 
-                                seed,context_size,unique_id = None, extra_pnginfo=None):
+                                 model, 
+                                seed,
+                                context_size,
+                                api_url,
+                                api_key=None,
+                                custom_model_name=None,
+                                custom_api_url=None,
+                                ):
+        
+        if custom_model_name!=None:
+            model=custom_model_name
+
+        api_url=llm_apis_dict[api_url] if api_url in llm_apis_dict else ""
+
+        if custom_api_url!=None:
+            api_url=custom_api_url
+
+        if api_key==None:
+            api_key="lm_studio"
+
         # print(api_key!='',api_url,prompt,system_content,model,seed)
         # 可以选择保留会话历史以维持上下文记忆
         # 或者在此处清除会话历史 self.session_history.clear()
@@ -273,7 +387,7 @@ class ChatGPTNode:
             self.system_content=system_content
             # self.session_history=[]
             # self.session_history.append({"role": "system", "content": system_content})
-        
+        print("api_key,api_url",api_key,api_url)
         # 
         if is_azure_url(api_url):
             client=azure_client(api_key,api_url)
@@ -282,12 +396,12 @@ class ChatGPTNode:
             if model == "glm-4" :
                 client = ZhipuAI_client(api_key)  # 使用 Zhipuai 的接口
                 print('using Zhipuai interface')
-            elif model in llama_modes_list:
-                #
-                client=llama_cpp_client(model)
+            # elif model in llama_modes_list:
+            #     #
+            #     client=llama_cpp_client(model)
             else :
                 client = openai_client(api_key,api_url)  # 使用 ChatGPT  的接口
-                print('using ChatGPT interface')
+                # print('using ChatGPT interface',api_key,api_url)
 
         # 把用户的提示添加到会话历史中
         # 调用API时传递整个会话历史
@@ -303,6 +417,7 @@ class ChatGPTNode:
         session_history=crop_list_tail(self.session_history,context_size)
 
         messages=[{"role": "system", "content": self.system_content}]+session_history+[{"role": "user", "content": prompt}]
+
         response_content = chat(client,model,messages)
         
         self.session_history=self.session_history+[{"role": "user", "content": prompt}]+[{'role':'assistant',"content":response_content}]
@@ -323,6 +438,93 @@ class ChatGPTNode:
         return (response_content,json.dumps(messages, indent=4),json.dumps(self.session_history, indent=4),)
 
 
+class SiliconflowFreeNode:
+    def __init__(self):
+        # self.__client = OpenAI()
+        self.session_history = []  # 用于存储会话历史的列表
+        # self.seed=0
+        self.system_content="You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible."
+
+    @classmethod
+    def INPUT_TYPES(cls):
+        model_list= [ 
+            "Qwen/Qwen2-7B-Instruct",
+            "THUDM/glm-4-9b-chat",
+            "01-ai/Yi-1.5-9B-Chat-16K",
+            "meta-llama/Meta-Llama-3.1-8B-Instruct"
+            ]
+        return {
+            "required": {
+                "api_key":("STRING", {"forceInput": True,}),
+                "prompt": ("STRING", {"multiline": True,"dynamicPrompts": False}),
+                "system_content": ("STRING", 
+                                   {
+                                       "default": "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.", 
+                                       "multiline": True,"dynamicPrompts": False
+                                       }),
+                "model": ( model_list, 
+                    {"default": model_list[0]}),
+                "seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff, "step": 1}),
+                "context_size":("INT", {"default": 1, "min": 0, "max":30, "step": 1}),
+            },
+               "optional":{
+                    "custom_model_name":("STRING", {"forceInput": True,}), #适合自定义model
+                }, 
+        }
+
+    RETURN_TYPES = ("STRING","STRING","STRING",)
+    RETURN_NAMES = ("text","messages","session_history",)
+    FUNCTION = "generate_contextual_text"
+    CATEGORY = "♾️Mixlab/GPT"
+    INPUT_IS_LIST = False
+    OUTPUT_IS_LIST = (False,False,False,)
+
+    
+    def generate_contextual_text(self,
+                                 api_key,
+                                 prompt, 
+                                 system_content,
+                                model, 
+                                seed,context_size,custom_model_name=None):
+
+        if custom_model_name!=None:
+            model=custom_model_name
+
+        api_url="https://api.siliconflow.cn/v1"
+        
+        # 把系统信息和初始信息添加到会话历史中
+        if system_content:
+            self.system_content=system_content
+            # self.session_history=[]
+            # self.session_history.append({"role": "system", "content": system_content})
+        
+        # 
+        client = openai_client(api_key,api_url)  # 使用 ChatGPT  的接口
+        # print('using ChatGPT interface',api_key,api_url)
+
+        # 把用户的提示添加到会话历史中
+        # 调用API时传递整个会话历史
+
+        def crop_list_tail(lst, size):
+            if size >= len(lst):
+                return lst
+            elif size==0:
+                return []
+            else:
+                return lst[-size:]
+            
+        session_history=crop_list_tail(self.session_history,context_size)
+
+        messages=[{"role": "system", "content": self.system_content}]+session_history+[{"role": "user", "content": prompt}]
+
+        response_content = chat(client,model,messages)
+        
+        self.session_history=self.session_history+[{"role": "user", "content": prompt}]+[{'role':'assistant',"content":response_content}]
+
+        return (response_content,json.dumps(messages, indent=4),json.dumps(self.session_history, indent=4),)
+
+
+
 
 class ShowTextForGPT:
     @classmethod
@@ -484,3 +686,35 @@ class TextSplitByDelimiter:
         arr= arr[start_index:start_index + max_count * (skip_every+1):(skip_every+1)]
 
         return (arr,)
+
+
+class JsonRepair:
+    @classmethod
+    def INPUT_TYPES(s):
+        return {
+            "required": {
+                 "json_string":("STRING", {"forceInput": True,}), 
+            }
+        }
+
+    INPUT_IS_LIST = False
+    RETURN_TYPES = ("STRING",)
+    FUNCTION = "run"
+    # OUTPUT_NODE = True
+    OUTPUT_IS_LIST = (False,)
+
+    CATEGORY = "♾️Mixlab/GPT"
+
+    def run(self, json_string):
+
+        json_string=extract_json_strings(json_string)
+        # print(json_string)
+        good_json_string = repair_json(json_string)
+
+        # 将 JSON 字符串解析为 Python 对象
+        data = json.loads(good_json_string)
+
+        # 将 Python 对象转换回 JSON 字符串，确保中文字符不被转义
+        json_str_with_chinese = json.dumps(data, ensure_ascii=False)
+
+        return (json_str_with_chinese,)

nodes/ClipInterrogator.py

@@ -79,7 +79,7 @@ def get_clip_interrogator_path():
 
 cache_path=get_clip_interrogator_path()
 
-caption_model_path=os.path.join(cache_path, "Salesforce/blip-image-captioning-base")
+caption_model_path=os.path.join(cache_path, "Salesforce","blip-image-captioning-base")
 if not os.path.exists(caption_model_path):
     print(f"## clip_interrogator_model not found: {caption_model_path}, pls download from https://huggingface.co/Salesforce/blip-image-captioning-base")
     caption_model_path='Salesforce/blip-image-captioning-base'

nodes/ImageNode.py

@@ -1,6 +1,7 @@
 import numpy as np
 import requests
 import torch
+import torchvision.transforms.v2 as T
 # from PIL import Image, ImageDraw
 from PIL import Image, ImageOps,ImageFilter,ImageEnhance,ImageDraw,ImageSequence, ImageFont
 from PIL.PngImagePlugin import PngInfo
@@ -14,8 +15,8 @@ import cv2
 import string
 import math,glob
 from .Watcher import FolderWatcher
-import hashlib
 
+from itertools import product
 
 
 # 将PIL图片转换为OpenCV格式
@@ -28,142 +29,105 @@ def opencv_to_pil(image):
     pil_image = Image.fromarray(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
     return pil_image
 
+# 列出目录下面的所有文件
+def get_files_with_extension(directory, extensions):
+    file_list = []
+    # 确保extensions参数是一个list，即使只有一个元素
+    if not isinstance(extensions, (tuple, list)):
+        extensions = [extensions]
+    for root, dirs, files in os.walk(directory):
+        # print(f"Files at {root}: {files}")  # 确认files是一个字符串列表
+        for file in files:
+            # 检查文件是否以任何一个提供的扩展名结尾
+            if any(file.endswith(ext) for ext in extensions):
+                # 直接将文件名添加到列表中
+                file_list.append(file)
+    return file_list
+	
+def composite_images(foreground, background, mask, is_multiply_blend=False, position="overall", scale=0.25):
+    width, height = foreground.size
+    bg_image = background
+    bwidth, bheight = bg_image.size
 
-def composite_images(foreground, background, mask,is_multiply_blend=False,position="overall"):
-    width,height=foreground.size
- 
-    bg_image=background
+    scale=max(scale,1/bwidth)
+    scale=max(scale,1/bheight)
 
-    bwidth,bheight=bg_image.size
+    def determine_scale_option(width, height):
+        return 'height' if height > width else 'width'
 
-    # 按z-index排序
-    if position=="overall":
+    if position == "overall":
         layer = {
-                "x":0,
-                "y":0,
-                "width":bwidth,
-                "height":bheight,
-                "z_index":88,
-                "scale_option":'overall',
-                "image":foreground,
-                "mask":mask
+            "x": 0,
+            "y": 0,
+            "width": bwidth,
+            "height": bheight,
+            "z_index": 88,
+            "scale_option": 'overall',
+            "image": foreground,
+            "mask": mask
         }
+    else:
+        scale_option = determine_scale_option(width, height)
+        if scale_option == 'height':
+            scale = int(bheight * scale) / height
+        else:
+            scale = int(bwidth * scale) / width
 
-    elif position=='center_bottom':
-        
-        scale = int(bwidth*0.25) / width
+        new_width = int(width * scale)
         new_height = int(height * scale)
 
-        layer = {
-                "x":int(bwidth*0.75*0.5),
-                "y":bheight-new_height-24,
-                "width":int(bwidth*0.25),
-                "height":int(bheight*0.25),
-                "z_index":88,
-                "scale_option":'width',
-                "image":foreground,
-                "mask":mask
-        }
-
-    elif position=='right_bottom':
-        
-        scale = int(bwidth*0.25) / width
-        new_height = int(height * scale)
+        if position == 'center_bottom':
+            x_position = int((bwidth - new_width) * 0.5)
+            y_position = bheight - new_height - 24
+        elif position == 'right_bottom':
+            x_position = bwidth - new_width - 24
+            y_position = bheight - new_height - 24
+        elif position == 'center_top':
+            x_position = int((bwidth - new_width) * 0.5)
+            y_position = 24
+        elif position == 'right_top':
+            x_position = bwidth - new_width - 24
+            y_position = 24
+        elif position == 'left_top':
+            x_position = 24
+            y_position = 24
+        elif position == 'left_bottom':
+            x_position = 24
+            y_position = bheight - new_height - 24
+        elif position == 'center_center':
+            x_position = int((bwidth - new_width) * 0.5)
+            y_position = int((bheight - new_height) * 0.5)
 
         layer = {
-                "x":bwidth-int(bwidth*0.25)-24,
-                "y":bheight-new_height-24,
-                "width":int(bwidth*0.25),
-                "height":int(bheight*0.25),
-                "z_index":88,
-                "scale_option":'width',
-                "image":foreground,
-                "mask":mask
+            "x": x_position,
+            "y": y_position,
+            "width": new_width,
+            "height": new_height,
+            "z_index": 88,
+            "scale_option": scale_option,
+            "image": foreground,
+            "mask": mask
         }
 
+    layer_image = layer['image']
+    layer_mask = layer['mask']
 
-    elif position=='center_top':
-        
-        scale = int(bwidth*0.25) / width
-        new_height = int(height * scale)
+    bg_image = merge_images(bg_image,
+                            layer_image,
+                            layer_mask,
+                            layer['x'],
+                            layer['y'],
+                            layer['width'],
+                            layer['height'],
+                            layer['scale_option'],
+                            is_multiply_blend)
 
-        layer = {
-                "x":int( bwidth*0.75*0.5),
-                "y":24,
-                "width":int(bwidth*0.25),
-                "height":int(bheight*0.25),
-                "z_index":88,
-                "scale_option":'width',
-                "image":foreground,
-                "mask":mask
-        }
+    bg_image = bg_image.convert('RGB')
 
-    elif position=='right_top':
-        
-        scale = int(bwidth*0.25) / width
-        new_height = int(height * scale)
-
-        layer = {
-                "x":bwidth-int(bwidth*0.25)-24,
-                "y":24,
-                "width":int(bwidth*0.25),
-                "height":int(bheight*0.25),
-                "z_index":88,
-                "scale_option":'width',
-                "image":foreground,
-                "mask":mask
-        }
-    elif position=='left_top':
-        
-        scale = int(bwidth*0.25) / width
-        new_height = int(height * scale)
-
-        layer = {
-                "x":24,
-                "y":24,
-                "width":int(bwidth*0.25),
-                "height":int(bheight*0.25),
-                "z_index":88,
-                "scale_option":'width',
-                "image":foreground,
-                "mask":mask
-        }
-    elif position=='left_bottom':
-        
-        scale = int(bwidth*0.25) / width
-        new_height = int(height * scale)
-
-        layer = {
-                "x":24,
-                "y":bheight-new_height-24,
-                "width":int(bwidth*0.25),
-                "height":int(bheight*0.25),
-                "z_index":88,
-                "scale_option":'width',
-                "image":foreground,
-                "mask":mask
-        }
-                
-    # width, height = bg_image.size
-
-    layer_image=layer['image']
-    layer_mask=layer['mask']
-    
-    bg_image=merge_images(bg_image,
-                        layer_image,
-                        layer_mask,
-                        layer['x'],
-                        layer['y'],
-                        layer['width'],
-                        layer['height'],
-                        layer['scale_option'],
-                        is_multiply_blend )
-                   
-    bg_image=bg_image.convert('RGB')
-    
     return bg_image
 
 
+
 def count_files_in_directory(directory):
     file_count = 0
     for _, _, files in os.walk(directory):
@@ -200,7 +164,8 @@ class AnyType(str):
 any_type = AnyType("*")
 
 
-FONT_PATH= os.path.abspath(os.path.join(os.path.dirname(__file__),'../assets/王汉宗颜楷体繁.ttf'))
+FONT_PATH= os.path.abspath(os.path.join(os.path.dirname(__file__),"..","assets","fonts"))
+
 
 MAX_RESOLUTION=8192
 
@@ -802,85 +767,78 @@ def multiply_blend(image1, image2):
 
 # cv2.imwrite('result.jpg', result)
 
+# 使用gpt4o优化代码
+# 为了消除图像合并时出现的灰色描边，可以使用以下方法：
+# 调整透明度：确保透明像素不会引入不需要的颜色。
+# 预处理图像：在缩放图像之前，可以先将图像的边缘进行预处理，例如扩展边缘颜色，减少抗锯齿带来的过渡效果。
 
-def merge_images(bg_image, layer_image, mask, x, y, width, height, scale_option,is_multiply_blend=False):
+def merge_images(bg_image, layer_image, mask, x, y, width, height, scale_option, is_multiply_blend=False):
     # 打开底图
     bg_image = bg_image.convert("RGBA")
 
     # 打开图层
     layer_image = layer_image.convert("RGBA")
-    # layer_image = layer_image.resize((width, height))
-
+   
     # 根据缩放选项调整图像大小
     if scale_option == "height":
         # 按照高度比例缩放
         original_width, original_height = layer_image.size
         scale = height / original_height
         new_width = int(original_width * scale)
-        layer_image = layer_image.resize((new_width, height))
+        layer_image = layer_image.resize((new_width, height), Image.NEAREST)
     elif scale_option == "width":
         # 按照宽度比例缩放
         original_width, original_height = layer_image.size
         scale = width / original_width
         new_height = int(original_height * scale)
-        layer_image = layer_image.resize((width, new_height))
+        layer_image = layer_image.resize((width, new_height), Image.NEAREST)
     elif scale_option == "overall":
         # 整体缩放
-        layer_image = layer_image.resize((width, height))
-
+        layer_image = layer_image.resize((width, height), Image.NEAREST)
     elif scale_option == "longest":
         original_width, original_height = layer_image.size
         if original_width > original_height:
-            new_width=width
+            new_width = width
             scale = width / original_width
             new_height = int(original_height * scale)
-            x=0
-            y=int((height-new_height)*0.5)
+            x = 0
+            y = int((height - new_height) * 0.5)
         else:
-            new_height=height
+            new_height = height
             scale = height / original_height
             new_width = int(original_height * scale)
-            x=int((width-new_width)*0.5)
-            y=0
-    # elif side == "shortest":
-    #         if width < height:
-    #              
-    #         else:
-    #              
-    
+            x = int((width - new_width) * 0.5)
+            y = 0
 
     # 调整mask的大小
     nw, nh = layer_image.size
-    mask = mask.resize((nw, nh))
+    mask = mask.resize((nw, nh), Image.NEAREST)
 
-    # # 分离出a通道
-    # r, g, b, alpha = layer_image.split()
-    # alpha = ImageOps.invert(alpha)
-    # # 创建一个新的RGB图像
-    # new_rgb_image = Image.new("RGB", layer_image.size)
-    # # 将透明通道粘贴到新的RGB图像上
-    # new_rgb_image.paste(layer_image, (0, 0), mask=alpha)
-   
-    # new_rgb_image.paste(layer_image, (x, y), mask=mask)
-    # mask=new_rgb_image.convert('L')
-    # mask = ImageOps.invert(mask)
+    # 预处理图像边缘以减少灰色描边
+    layer_image = layer_image.filter(ImageFilter.SMOOTH)
 
     if is_multiply_blend:
-        bg_image_white=Image.new("RGB", bg_image.size,(255, 255, 255))
+        bg_image_white = Image.new("RGB", bg_image.size, (255, 255, 255))
 
         bg_image_white.paste(layer_image, (x, y), mask=mask)
-        bg_image=multiply_blend(bg_image_white,bg_image)
-        bg_image=bg_image.convert("RGBA")
+        bg_image = multiply_blend(bg_image_white, bg_image)
+        bg_image = bg_image.convert("RGBA")
     else:
-        transparent_img = Image.new("RGBA",layer_image.size, (255, 255, 255, 0))
-        transparent_img.paste(layer_image,(0, 0), mask)
-        # transparent_img.save('test.png')
-        bg_image.paste(transparent_img, (x, y), transparent_img)
+        transparent_img = Image.new("RGBA", layer_image.size, (255, 255, 255, 0))
+        # 调整透明度处理
+        for i in range(transparent_img.size[0]):
+            for j in range(transparent_img.size[1]):
+                r, g, b, a = transparent_img.getpixel((i, j))
+                if a > 0:
+                    transparent_img.putpixel((i, j), (r, g, b, 255))
 
+        transparent_img.paste(layer_image, (0, 0), mask)
+        bg_image.paste(transparent_img, (x, y), transparent_img)
 
     # 输出合成后的图片
     return bg_image
 
+#MixCopilot
 
 def resize_2(img):
     # 检查图像的高度是否是2的倍数，如果不是，则调整高度
@@ -954,53 +912,13 @@ def resize_image(layer_image, scale_option, width, height,color="white"):
     return layer_image
 
 
-
-
-
-
-# def generate_text_image(text_list, font_path, font_size, text_color, vertical=True, spacing=0):
-#     # Load Chinese font
-#     font = ImageFont.truetype(font_path, font_size)
-
-#     # Calculate image size based on the number of characters and orientation
-#     if vertical:
-#         width = font_size + 100
-#         height = font_size * len(text_list) + (len(text_list) - 1) * spacing + 100
-#     else:
-#         width = font_size * len(text_list) + (len(text_list) - 1) * spacing + 100
-#         height = font_size + 100
-
-#     # Create a blank image
-#     image = Image.new('RGBA', (width, height), (255, 255, 255,0))
-#     draw = ImageDraw.Draw(image)
-
-#     # Draw text
-#     if vertical:
-#         for i, char in enumerate(text_list):
-#             char_position = (50, 50 + i * font_size)
-#             draw.text(char_position, char, font=font, fill=text_color)
-#     else:
-#         for i, char in enumerate(text_list):
-#             char_position = (50 + i * (font_size + spacing), 50)
-#             draw.text(char_position, char, font=font, fill=text_color)
-
-#     # Save the image
-#     # image.save(output_image_path)
-
-#     # 分离alpha通道
-#     alpha_channel = image.split()[3]
-
-#     # 创建一个只有alpha通道的新图像
-#     alpha_image = Image.new('L', image.size)
-#     alpha_image.putdata(alpha_channel.getdata())
-
-#     image=image.convert('RGB')
-
-#     return (image,alpha_image)
-def generate_text_image(text, font_path, font_size, text_color, vertical=True, stroke=False, stroke_color=(0, 0, 0), stroke_width=1, spacing=0):
+def generate_text_image(text, font_path, font_size, text_color, vertical=True, stroke=False, stroke_color=(0, 0, 0), stroke_width=1, spacing=0, line_spacing=0,padding=4):
     # Split text into lines based on line breaks
     lines = text.split("\n")
 
+    # Load font
+    font = ImageFont.truetype(font_path, font_size)
+
     # 1. Determine layout direction
     if vertical:
         layout = "vertical"
@@ -1009,49 +927,54 @@ def generate_text_image(text, font_path, font_size, text_color, vertical=True, s
 
     # 2. Calculate absolute coordinates for each character
     char_coordinates = []
-    if layout == "vertical":
-        x = 0
-        y = 0
-        for i in range(len(lines)):
-            line = lines[i]
-            for char in line:
-                char_coordinates.append((x, y))
-                y += font_size + spacing
-            x += font_size + spacing
-            y = 0
-    else:
-        x = 0
-        y = 0
-        for line in lines:
-            for char in line:
-                char_coordinates.append((x, y))
-                x += font_size + spacing
-            y += font_size + spacing
-            x = 0
+    x, y = padding, padding
+    max_width, max_height = 0, 0
 
-    # 3. Calculate image width and height
     if layout == "vertical":
-        width = (len(lines) * (font_size + spacing)) - spacing
-        height = ((len(max(lines, key=len)) + 1) * (font_size + spacing)) + spacing
+        for line in lines:
+            max_char_width = max(font.getsize(char)[0] for char in line)
+            for char in line:
+                char_width, char_height = font.getsize(char)
+                char_coordinates.append((x, y))
+                y += char_height + spacing
+                max_height = max(max_height, y + padding)
+            x += max_char_width + line_spacing
+            y = padding
+        max_width = x
+        total_line_width = sum(font.getsize(line)[1] for line in lines)
+        total_spacing = line_spacing * (len(lines) - 1)
+        # 确保左边和右边的padding都被计入max_width
+        max_width = total_line_width + total_spacing + padding * 2
     else:
-        width = (len(max(lines, key=len)) * (font_size + spacing)) - spacing
-        height = ((len(lines) - 1) * (font_size + spacing)) + font_size
+        for line in lines:
+            line_width, line_height = font.getsize(line)
+            for char in line:
+                char_width, char_height = font.getsize(char)
+                char_coordinates.append((x, y))
+                x += char_width + spacing
+                max_width = max(max_width, x + padding)
+            y += line_height + line_spacing
+            x = padding
+        # max_height = y
+        total_line_heights = sum(font.getsize(line)[1] for line in lines)
+        total_spacing = line_spacing * (len(lines) - 1)
+        # 确保顶部和底部的padding都被计入max_height
+        max_height = total_line_heights + total_spacing + padding * 2
+
+    # 3. Create image with calculated width and height
+    image = Image.new('RGBA', (max_width, max_height), (255, 255, 255, 0))
+    draw = ImageDraw.Draw(image)
 
     # 4. Draw each character on the image
-    image = Image.new('RGBA', (width, height), (255, 255, 255, 0))
-    draw = ImageDraw.Draw(image)
-    font = ImageFont.truetype(font_path, font_size)
-
     index = 0
-    for i, line in enumerate(lines):
-        for j, char in enumerate(line):
+    for line in lines:
+        for char in line:
             x, y = char_coordinates[index]
-            
             if stroke:
-                draw.text((x-stroke_width, y), char, font=font, fill=stroke_color)
-                draw.text((x+stroke_width, y), char, font=font, fill=stroke_color)
-                draw.text((x, y-stroke_width), char, font=font, fill=stroke_color)
-                draw.text((x, y+stroke_width), char, font=font, fill=stroke_color)
+                draw.text((x-stroke_width, y), char, font=font, fill=text_color)
+                draw.text((x+stroke_width, y), char, font=font, fill=text_color)
+                draw.text((x, y-stroke_width), char, font=font, fill=text_color)
+                draw.text((x, y+stroke_width), char, font=font, fill=text_color)
             
             draw.text((x, y), char, font=font, fill=text_color)
             index += 1
@@ -1376,6 +1299,9 @@ class LoadImages_:
             image=pil2tensor(image)
             ims.append(image)
 
+        if len(ims)==0:
+            image1 = Image.new('RGB', (512, 512), color='black')
+            return (pil2tensor(image1),)
         image1 = ims[0]
         for image2 in ims[1:]:
             if image1.shape[1:] != image2.shape[1:]:
@@ -1578,7 +1504,7 @@ class ImageCropByAlpha:
 
 
 
-
+# get_files_with_extension(FONT_PATH,'.ttf')
 
 class TextImage:
     @classmethod
@@ -1586,18 +1512,32 @@ class TextImage:
         return {"required": { 
             
                     "text": ("STRING",{"multiline": True,"default": "龍馬精神迎新歲","dynamicPrompts": False}),
-                    "font_path": ("STRING",{"multiline": False,"default": FONT_PATH,"dynamicPrompts": False}),
+                    "font": (get_files_with_extension(FONT_PATH,['.ttf','.otf']),),#后缀为 ttf
                     "font_size": ("INT",{
                                 "default":100, 
-                                "min": 100, #Minimum value
-                                "max": 1000, #Maximum value
+                                "min": 1, #Minimum value
+                                "max": 10000000, #Maximum value
                                 "step": 1, #Slider's step
                                 "display": "number" # Cosmetic only: display as "number" or "slider"
                                 }), 
                     "spacing": ("INT",{
                                 "default":12, 
-                                "min": -200, #Minimum value
-                                "max": 200, #Maximum value
+                                "min": -2000000000, #Minimum value
+                                "max": 2000000000, #Maximum value
+                                "step": 1, #Slider's step
+                                "display": "number" # Cosmetic only: display as "number" or "slider"
+                                }), 
+                    "line_spacing": ("INT",{
+                                "default":12, 
+                                "min": -2000000000, #Minimum value
+                                "max": 2000000000, #Maximum value
+                                "step": 1, #Slider's step
+                                "display": "number" # Cosmetic only: display as "number" or "slider"
+                                }), 
+                    "padding": ("INT",{
+                                "default":8, 
+                                "min": 0, #Minimum value
+                                "max": 2000000000, #Maximum value
                                 "step": 1, #Slider's step
                                 "display": "number" # Cosmetic only: display as "number" or "slider"
                                 }), 
@@ -1608,7 +1548,7 @@ class TextImage:
                 }
     
     RETURN_TYPES = ("IMAGE","MASK",)
-    # RETURN_NAMES = ("WIDTH","HEIGHT","X","Y",)
+    RETURN_NAMES = ("image","mask",)
 
     FUNCTION = "run"
 
@@ -1617,11 +1557,14 @@ class TextImage:
     INPUT_IS_LIST = False
     OUTPUT_IS_LIST = (False,False,)
 
-    def run(self,text,font_path,font_size,spacing,text_color,vertical,stroke):
+    def run(self,text,font,font_size,spacing,line_spacing,padding,text_color,vertical,stroke):
         
-        # text_list=list(text)
+        font_path=os.path.join(FONT_PATH,font)
+
+        if text=="":
+            text=" "
         # stroke=False, stroke_color=(0, 0, 0), stroke_width=1, spacing=0
-        img,mask=generate_text_image(text,font_path,font_size,text_color,vertical,stroke,(0, 0, 0),1,spacing)
+        img,mask=generate_text_image(text,font_path,font_size,text_color,vertical,stroke,(0, 0, 0),1,spacing,line_spacing,padding)
         
         img=pil2tensor(img)
         mask=pil2tensor(mask)
@@ -1655,7 +1598,7 @@ class LoadImagesFromURL:
 
     def run(self,url,seed=0):
         global urls_image
-        print(urls_image)
+        # print(urls_image)
         def filter_http_urls(urls):
             filtered_urls = []
             for url in urls.split('\n'):
@@ -1854,10 +1797,16 @@ class CompositeImages:
                     "mask":("MASK",),
                     "background": ("IMAGE",),
                     },
-             "optional":{
-                           
+             "optional":{ 
                   "is_multiply_blend":  ("BOOLEAN", {"default": False}),
-                  "position":  (['overall',"center_bottom","center_top","right_bottom","left_bottom","right_top","left_top"],),    
+                  "position":  (['overall',"center_center","left_bottom","center_bottom","right_bottom","left_top","center_top","right_top"],),
+                   "scale": ("FLOAT",{
+                                "default":0.35, 
+                                "min": 0.01, #Minimum value
+                                "max": 1, #Maximum value
+                                "step": 0.01, #Slider's step
+                                "display": "number" # Cosmetic only: display as "number" or "slider"
+                            }), 
                     }
                 }
     
@@ -1870,15 +1819,30 @@ class CompositeImages:
 
     # OUTPUT_IS_LIST = (True,)
 
-    def run(self, foreground,mask,background,is_multiply_blend,position):
-        foreground= tensor2pil(foreground)
-        mask= tensor2pil(mask)
-        background= tensor2pil(background)
-        res=composite_images(foreground,background,mask,is_multiply_blend,position)
+    # def run(self, foreground,mask,background,is_multiply_blend,position,scale):
+    #     foreground= tensor2pil(foreground)
+    #     mask= tensor2pil(mask)
+    #     background= tensor2pil(background)
+    #     res=composite_images(foreground,background,mask,is_multiply_blend,position,scale)
         
-        return (pil2tensor(res),)
+    #     return (pil2tensor(res),)
 
+    def run(self, foreground,mask,background, is_multiply_blend, position, scale):
+        results = []
+        f1=[]
+        for fg, mask in zip(foreground, mask ):
+            f1.append([fg,mask])
+        for f, bg in product(f1, background):
+            [fg,mask]=f
+            fg_pil = tensor2pil(fg)
+            mask_pil = tensor2pil(mask)
+            bg_pil = tensor2pil(bg)
+            res = composite_images(fg_pil, bg_pil, mask_pil, is_multiply_blend, position, scale)
+            results.append(pil2tensor(res))
+        
+        output_image = torch.cat(results, dim=0)
 
+        return (output_image,)
 
 
 class EmptyLayer:
@@ -3207,3 +3171,52 @@ class SaveImageToLocal:
             counter += 1
 
         return ()
+
+
+class ImageBatchToList_:
+    @classmethod
+    def INPUT_TYPES(s):
+        return {"required": {"image_batch": ("IMAGE",), }}
+
+    RETURN_TYPES = ("IMAGE",)
+    RETURN_NAMES = ("image_list",)
+    OUTPUT_IS_LIST = (True,)
+    FUNCTION = "run"
+
+    CATEGORY = "♾️Mixlab/Image"
+
+    def run(self, image_batch):
+        images = [image_batch[i:i + 1, ...] for i in range(image_batch.shape[0])]
+        return (images, )
+
+class ImageListToBatch_:
+    @classmethod
+    def INPUT_TYPES(s):
+        return {
+            "required": {
+                "images": ("IMAGE",),
+            }
+        }
+
+    RETURN_TYPES = ("IMAGE",)
+    FUNCTION = "run"
+    INPUT_IS_LIST = True
+    CATEGORY = "♾️Mixlab/Image"
+
+    def run(self, images):
+        shape = images[0].shape[1:3]
+        out = []
+
+        for i in range(len(images)):
+            img = images[i].permute([0,3,1,2])
+            if images[i].shape[1:3] != shape:
+                transforms = T.Compose([
+                    T.CenterCrop(min(img.shape[2], img.shape[3])),
+                    T.Resize((shape[0], shape[1]), interpolation=T.InterpolationMode.BICUBIC),
+                ])
+                img = transforms(img)
+            out.append(img.permute([0,2,3,1]))
+
+        out = torch.cat(out, dim=0)
+
+        return (out,)

nodes/Lama.py

@@ -85,8 +85,6 @@ class LaMaInpainting:
             "image": ("IMAGE",),
             "mask": ("MASK",),
                              },
-
-            
                 }
     
     RETURN_TYPES = ("IMAGE",)

nodes/P5.py

@@ -0,0 +1,104 @@
+import torch
+import numpy as np
+from PIL import Image,ImageSequence,ImageOps
+import base64
+import io
+import comfy.utils
+import folder_paths
+import node_helpers
+
+
+# Tensor to PIL
+def tensor2pil(image):
+    return Image.fromarray(np.clip(255. * image.cpu().numpy().squeeze(), 0, 255).astype(np.uint8))
+
+# Convert PIL to Tensor
+def pil2tensor(image):
+    return torch.from_numpy(np.array(image).astype(np.float32) / 255.0).unsqueeze(0)
+
+def load_image( image):
+    image_path = folder_paths.get_annotated_filepath(image)
+        
+    img = node_helpers.pillow(Image.open, image_path)
+        
+    output_images = []
+    output_masks = []
+    w, h = None, None
+
+    excluded_formats = ['MPO']
+        
+    for i in ImageSequence.Iterator(img):
+        i = node_helpers.pillow(ImageOps.exif_transpose, i)
+
+        if i.mode == 'I':
+            i = i.point(lambda i: i * (1 / 255))
+        image = i.convert("RGB")
+
+        if len(output_images) == 0:
+            w = image.size[0]
+            h = image.size[1]
+            
+        if image.size[0] != w or image.size[1] != h:
+            continue
+            
+        image = np.array(image).astype(np.float32) / 255.0
+        image = torch.from_numpy(image)[None,]
+        if 'A' in i.getbands():
+            mask = np.array(i.getchannel('A')).astype(np.float32) / 255.0
+            mask = 1. - torch.from_numpy(mask)
+        else:
+            mask = torch.zeros((64,64), dtype=torch.float32, device="cpu")
+        output_images.append(image)
+        output_masks.append(mask.unsqueeze(0))
+
+    if len(output_images) > 1 and img.format not in excluded_formats:
+        output_image = torch.cat(output_images, dim=0)
+        output_mask = torch.cat(output_masks, dim=0)
+    else:
+        output_image = output_images[0]
+        output_mask = output_masks[0]
+
+    return (output_image, output_mask)
+
+          
+
+class P5Input:
+    @classmethod
+    def INPUT_TYPES(s):
+        return {"required": { 
+                    "frames":("IMAGEBASE64",), 
+                             },
+                }
+    
+    RETURN_TYPES = ("IMAGE",)
+    RETURN_NAMES = ("frames",)
+
+    FUNCTION = "run"
+
+    CATEGORY = "♾️Mixlab/Input"
+
+    OUTPUT_NODE = True
+    INPUT_IS_LIST = False
+    OUTPUT_IS_LIST = (False,)
+
+    def run(self, frames):
+        ims=[]
+        for im in frames['images']:
+            # print(im)
+            if 'type' in im and (not f"[{im['type']}]" in im['name']):
+                im['name']=im['name']+" "+f"[{im['type']}]"
+                
+            output_image, output_mask = load_image(im['name'])
+            ims.append(output_image)
+
+        if len(ims)==0:
+            image1 = Image.new('RGB', (512, 512), color='black')
+            return (pil2tensor(image1),)
+        image1 = ims[0]
+        for image2 in ims[1:]:
+            if image1.shape[1:] != image2.shape[1:]:
+                image2 = comfy.utils.common_upscale(image2.movedim(-1, 1), image1.shape[2], image1.shape[1], "bilinear", "center").movedim(1, -1)
+            image1 = torch.cat((image1, image2), dim=0)
+
+        # 用于节点提示:p5节点提示有多少帧
+        return {"ui": {"_info": [len(frames['images'])]}, "result": (image1,)}

nodes/ScreenShareNode.py

@@ -90,7 +90,7 @@ class ScreenShareNode:
                 } }
     
     RETURN_TYPES = ('IMAGE','STRING','FLOAT',"INT")
-    RETURN_NAMES = ("IMAGE","PROMPT","FLOAT","INT")
+    RETURN_NAMES = ("current frame (image)","prompt","denoise (float)","seed (int)")
     FUNCTION = "run"
 
     CATEGORY = "♾️Mixlab/Screen"
@@ -109,7 +109,7 @@ class FloatingVideo:
     @classmethod
     def INPUT_TYPES(s):
         return { "required":{
-            "images": ("IMAGE",)
+            "image": ("IMAGE",)
         }, }
     
     # RETURN_TYPES = ('IMAGE','MASK')
@@ -124,16 +124,16 @@ class FloatingVideo:
     # OUTPUT_IS_LIST = (False,False,)
   
     # 运行的函数
-    def run(self,images):
+    def run(self,image):
        
         results = list()
     
-        for image in images:
-            image=tensor2pil(image)
+        for im in image:
+            im=tensor2pil(im)
             # image_base64 = base64.b64encode(image.tobytes())
 
             buffered = BytesIO()
-            image.save(buffered, format="JPEG")
+            im.save(buffered, format="JPEG")
             image_base64 = base64.b64encode(buffered.getvalue()).decode("utf-8")
 
             results.append(image_base64)

nodes/Utils.py

@@ -133,7 +133,7 @@ def get_font_files(directory):
 
     return font_files
 
-r_directory = os.path.join(os.path.dirname(__file__), '../assets/')
+r_directory = os.path.join(os.path.dirname(__file__), '..','assets','/')
 
 font_files = get_font_files(r_directory)
 # print(font_files)
@@ -181,6 +181,28 @@ class ColorInput:
         return (h,r,g,b,a,)
 
 
+class KeyInput:
+    @classmethod
+    def INPUT_TYPES(s):
+        return {"required": { 
+                    "key":("KEY",), 
+                             },
+                }
+    
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("key",)
+
+    FUNCTION = "run"
+
+    CATEGORY = "♾️Mixlab/Input"
+
+    INPUT_IS_LIST = False
+    OUTPUT_IS_LIST = (False,)
+
+    def run(self,key):
+        return (key,)
+
+
 
 class FontInput:
     @classmethod
@@ -566,7 +588,7 @@ class AppInfo:
                              },
 
                 "optional":{
-                    "IMAGE": ("IMAGE",),
+                    "image": ("IMAGE",),
                     "description":("STRING",{"multiline": True,"default": "","dynamicPrompts": False}),
                     "version":("INT", {
                         "default": 1, 
@@ -594,12 +616,12 @@ class AppInfo:
     INPUT_IS_LIST = True
     # OUTPUT_IS_LIST = (True,)
 
-    def run(self,name,input_ids,output_ids,IMAGE,description,version,share_prefix,link,category,auto_save):
+    def run(self,name,input_ids,output_ids,image,description,version,share_prefix,link,category,auto_save):
         name=name[0]
         
         im=None
-        if IMAGE:
-            im=IMAGE[0][0]
+        if image:
+            im=image[0][0]
             #TODO batch 的方式需要处理
             im=create_temp_file(im)
         # image [img,] img[batch,w,h,a] 列表里面是batch，

nodes/Video.py

@@ -17,9 +17,128 @@ import folder_paths
 from comfy.k_diffusion.utils import FolderOfImages
 from comfy.utils import common_upscale
 
+import torchaudio
+import base64
+
+import mimetypes
 
 
 
+def get_frames(frame_count, frames, revert=False):
+    if not revert:
+        if frame_count <= len(frames):
+            return frames[:frame_count]
+        else:
+            return [frames[i % len(frames)] for i in range(frame_count)]
+    else:
+        extended_frames = frames + frames[-2:0:-1]  # 正向加反向中间部分
+        if frame_count <= len(extended_frames):
+            return extended_frames[:frame_count]
+        else:
+            return [extended_frames[i % len(extended_frames)] for i in range(frame_count)]
+
+# # 示例用法
+# frames = ["frame1", "frame2", "frame3"]
+# frame_count = 2
+
+# result = get_frames(frame_count, frames, revert=False)
+# print(result)  # 输出: ['frame1', 'frame2', 'frame3', 'frame1', 'frame2', 'frame3', 'frame1']
+
+# result = get_frames(frame_count, frames, revert=True)
+# print(result)  # 输出: ['frame1', 'frame2', 'frame3', 'frame2', 'frame1', 'frame2', 'frame3']
+
+
+
+
+def get_mime_type(file_path):
+    # 获取文件的 MIME 类型
+    mime_type, _ = mimetypes.guess_type(file_path)
+    
+    # 如果无法猜测类型，返回默认类型
+    if mime_type is None:
+        return 'application/octet-stream'
+    
+    return mime_type
+# import subprocess
+# from imageio_ffmpeg import get_ffmpeg_exe
+
+
+def save_audio_base64s_to_file(base64_audios, output_folder, file_name):
+    # Ensure the output folder exists
+    if not os.path.exists(output_folder):
+        os.makedirs(output_folder)
+
+    decoded_audios=[]
+    for a in base64_audios:
+    
+        # If the base64 string contains a header, remove it
+        if ',' in a:
+            a = a.split(',')[1]
+        
+        # 解码 base64 数据
+        a=base64.b64decode(a)
+        decoded_audios.append(a)
+
+    # 拼接音频数据
+    combined_audio = b''.join(decoded_audios)
+    
+    # Create the full file path
+    file_path = os.path.join(output_folder, file_name)
+    
+    # Write the decoded audio to the file
+    with open(file_path, 'wb') as audio_file:
+        audio_file.write(combined_audio)
+    
+    return file_path
+
+# Example usage
+# base64_audio = "data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAIlYAAESsAAACABAAZGF0YQAAAAA="
+# output_folder = "audio_files"
+# file_name = "output.wav"
+
+# file_path = save_audio_base64_to_file(base64_audio, output_folder, file_name)
+# print(f"Audio saved to: {file_path}")
+
+# 写一个python文件，用来 判断文件夹内命名为 所有chat_tts开头的文件数量（chat_tts_00001），并输出新的编号
+def get_new_counter(full_output_folder, filename_prefix):
+    # 获取目录中的所有文件
+    files = os.listdir(full_output_folder)
+    
+    # 过滤出以 filename_prefix 开头并且后续部分为数字的文件
+    filtered_files = []
+    for f in files:
+        if f.startswith(filename_prefix):
+            # 去掉文件名中的前缀和后缀，只保留中间的数字部分
+            base_name = f[len(filename_prefix)+1:]
+            number_part = base_name.split('.')[0]  # 假设文件名中只有一个点，即扩展名
+            if number_part.isdigit():
+                filtered_files.append(int(number_part))
+
+    if not filtered_files:
+        return 1
+
+    # 获取最大的编号
+    max_number = max(filtered_files)
+    
+    # 新的编号
+    return max_number + 1
+
+def crop_audio(input_file, start_time, duration):
+    # Load the audio file
+    audio_tensor, sample_rate = torchaudio.load(input_file)
+    
+    # Convert start_time and duration from seconds to sample indices
+    start_sample = int(start_time * sample_rate)
+    end_sample = start_sample + int(duration * sample_rate)
+    
+    # Perform the slicing
+    cropped_audio_tensor = audio_tensor[:, start_sample:end_sample]
+    
+    # Save the cropped audio to a new file
+    torchaudio.save(input_file, cropped_audio_tensor, sample_rate)
+    
+    return input_file
+
 def generate_folder_name(directory,video_path):
     # Get the directory and filename from the video path
     _, filename = os.path.split(video_path)
@@ -60,6 +179,9 @@ def split_video(video_path, video_segment_frames, transition_frames, output_dir)
         
         # 打印当前片段的起始帧和结束帧
         print(f"Segment {i+1}: Start Frame {start_frame}, End Frame {end_frame}")
+
+        if end_frame<start_frame:
+            break
         
         # 保存当前片段为一个视频文件
         segment_video_path = f"{output_dir}/segment_{i+1}.avi"
@@ -68,6 +190,7 @@ def split_video(video_path, video_segment_frames, transition_frames, output_dir)
         segment_video = cv2.VideoWriter(segment_video_path, fourcc, fps, (int(video_capture.get(cv2.CAP_PROP_FRAME_WIDTH)),
                                                               int(video_capture.get(cv2.CAP_PROP_FRAME_HEIGHT))))
         
+
         for frame_num in range(start_frame, end_frame):
             ret, frame = video_capture.read()
             if ret:
@@ -101,6 +224,25 @@ if ffmpeg_path is None:
     except:
         print("ffmpeg could not be found. Outputs that require it have been disabled")
 
+
+def combine_audio_video(audio_path, video_path, output_path):
+   
+    command = [
+        ffmpeg_path,
+        '-i', video_path,
+        '-i', audio_path,
+        '-c:v', 'copy',
+        '-c:a', 'aac',
+        '-shortest',
+        output_path
+    ]
+    
+    subprocess.run(command, check=True)
+    return output_path
+
+
+
+
 # Tensor to PIL
 def tensor2pil(image):
     return Image.fromarray(np.clip(255. * image.cpu().numpy().squeeze(), 0, 255).astype(np.uint8))
@@ -262,7 +404,7 @@ class LoadVideoAndSegment:
                     files.append(f)
         return {"required": {
                     "video": (sorted(files), {"video_upload": True}),
-                     "video_segment_frames": ("INT", {"default": 10, "min": 1, "step": 1}),
+                     "video_segment_frames": ("INT", {"default": 10, "min": -1, "step": 1}),
                      "transition_frames": ("INT", {"default": 0, "min": 0, "step": 1}), 
                      },}
 
@@ -332,63 +474,6 @@ class LoadVideoAndSegment:
         
         video_path = folder_paths.get_annotated_filepath(video)
         
-        # check if video is a gif - will need to use cv fallback to read frames
-        # use cv fallback if ffmpeg not installed or gif
-        # if ffmpeg_path is None:
-        #     return self.load_video_cv_fallback(video, frame_load_cap, skip_first_frames)
-        # otherwise, continue with ffmpeg
-        
-        # args_dummy = [ffmpeg_path, "-i", video_path, "-f", "null", "-"]
-        # try:
-        #     with subprocess.Popen(args_dummy, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE) as proc:
-        #         for line in proc.stderr.readlines():
-        #             match = re.search(", ([1-9]|\\d{2,})x(\\d+)",line.decode('utf-8'))
-        #             if match is not None:
-        #                 size = [int(match.group(1)), int(match.group(2))]
-        #                 break
-        # except Exception as e:
-        #     print(f"Retrying with opencv due to ffmpeg error: {e}")
-        #     return self.load_video_cv_fallback(video, frame_load_cap, skip_first_frames)
-        # args_all_frames = [ffmpeg_path, "-i", video_path, "-v", "error",
-        #                      "-pix_fmt", "rgb24"]
-
-        # vfilters = []
-       
-        # if skip_first_frames > 0:
-        #     vfilters.append(f"select=gt(n\\,{skip_first_frames-1})")
-        # if frame_load_cap > 0:
-        #     vfilters.append(f"select=gt({frame_load_cap}\\,n)")
-        # #manually calculate aspect ratio to ensure reads remain aligned
-        
-        # if len(vfilters) > 0:
-        #     args_all_frames += ["-vf", ",".join(vfilters)]
-
-        # args_all_frames += ["-f", "rawvideo", "-"]
-        # images = []
-        # try:
-        #     with subprocess.Popen(args_all_frames, stdout=subprocess.PIPE) as proc:
-        #         #Manually buffer enough bytes for an image
-        #         bpi = size[0]*size[1]*3
-        #         current_bytes = bytearray(bpi)
-        #         current_offset=0
-        #         while True:
-        #             bytes_read = proc.stdout.read(bpi - current_offset)
-        #             if bytes_read is None:#sleep to wait for more data
-        #                 time.sleep(.2)
-        #                 continue
-        #             if len(bytes_read) == 0:#EOF
-        #                 break
-        #             current_bytes[current_offset:len(bytes_read)] = bytes_read
-        #             current_offset+=len(bytes_read)
-        #             if current_offset == bpi:
-        #                 images.append(np.array(current_bytes, dtype=np.float32).reshape(size[1], size[0], 3) / 255.0)
-        #                 current_offset = 0
-        # except Exception as e:
-        #     print(f"Retrying with opencv due to ffmpeg error: {e}")
-        #     return self.load_video_cv_fallback(video, frame_load_cap, skip_first_frames)
-
-        # imgs=split_list(images,video_segment_frames,transition_frames)
-
         # temp path 
         tp=folder_paths.get_temp_directory()
         basename = os.path.basename(video_path)  # 获取文件名
@@ -396,15 +481,22 @@ class LoadVideoAndSegment:
 
         folder_path = create_folder(tp,name_without_extension)
         
- 
-        # 导出的数据
-        scenes_video,total_frames,fps=split_video(video_path,video_segment_frames,
-                                 transition_frames,folder_path)
+        if video_segment_frames==-1:
+            # 不切割视频
+            scenes_video=[video_path]
+            # 读取视频文件
+            video_capture = cv2.VideoCapture(video_path)
+            
+            # 获取视频的总帧数和帧率
+            total_frames = int(video_capture.get(cv2.CAP_PROP_FRAME_COUNT))
+            fps = video_capture.get(cv2.CAP_PROP_FPS)
+
+        else:
+            # 导出的数据
+            scenes_video,total_frames,fps=split_video(video_path,video_segment_frames,
+                                    transition_frames,folder_path)
       
 
-        # imgs=[torch.from_numpy(np.stack(im)) for im in imgs]
-
-        # images = torch.from_numpy(np.stack(images))
 
         return (scenes_video,len(scenes_video), total_frames,fps,)
 
@@ -422,7 +514,113 @@ class LoadVideoAndSegment:
             return "Invalid image file: {}".format(video)
 
         return True
-    
+
+
+
+class LoadAndCombinedAudio_:
+    @classmethod
+    def INPUT_TYPES(s):
+       
+        return {"required": {
+                    "audios": ("AUDIOBASE64",),
+                    "start_time": ("FLOAT" , {"default": 0, "min": 0, "max": 10000000, "step": 0.01}),
+                    "duration": ("FLOAT" , {"default": 10, "min": -1, "max": 10000000, "step": 0.01}),
+                     },
+                }
+
+    CATEGORY = "♾️Mixlab/Audio"
+
+    RETURN_TYPES = ("STRING","AUDIO",)
+    RETURN_NAMES = ("audio_file_path","audio",)
+    FUNCTION = "run"
+
+    def run(self,audios, start_time, duration):
+        output_dir = folder_paths.get_output_directory()
+        counter=get_new_counter(output_dir,'audio_')
+
+        audio_file_name = f"audio_{counter:05}.wav"
+
+        audio_file=save_audio_base64s_to_file(audios['base64'],output_dir,audio_file_name)
+        # duration == -1 则不裁切
+        if duration > -1:
+            crop_audio(audio_file, start_time, duration)
+
+        waveform, sample_rate = torchaudio.load(audio_file)
+        audio = {
+            "filename": audio_file_name,
+            "subfolder": "",
+            "type": "output",
+            "audio_path":audio_file,
+            "waveform": waveform.unsqueeze(0), 
+            "sample_rate": sample_rate}
+
+        return (audio_file,audio ,)
+
+class CombineAudioVideo:
+    @classmethod
+    def INPUT_TYPES(s):
+       
+        return {"required": {
+                     "video": ("SCENE_VIDEO",),
+                     "audio": ("AUDIO", ), 
+                     },
+                }
+
+    CATEGORY = "♾️Mixlab/Video"
+
+    OUTPUT_NODE = True
+    FUNCTION = "run" 
+    RETURN_TYPES = ("SCENE_VIDEO",)
+    RETURN_NAMES = ("SCENE_VIDEO",)
+
+    def run(self,video, audio):
+
+        output_dir = folder_paths.get_output_directory()
+
+        # 判断是否是 Tensor 类型
+        is_tensor = not isinstance(audio, dict)
+        # print('#判断是否是 Tensor 类型',is_tensor,audio)
+        if not is_tensor and 'waveform' in audio and 'sample_rate' in audio:
+            # {'waveform': tensor([], size=(1, 1, 0)), 'sample_rate': 44100}
+            is_tensor=True
+
+        if "audio_path" in audio:
+            is_tensor=False
+            audio_file_path=audio["audio_path"]
+
+        if is_tensor:
+            filename_prefix="audio_tmp"
+            full_output_folder, filename, counter, subfolder, filename_prefix = folder_paths.get_save_image_path(
+                filename_prefix, 
+                folder_paths.get_temp_directory())
+            
+            filename_with_batch_num = filename.replace("%batch_num%", str(1))
+            file = f"{filename_with_batch_num}_{counter:05}_.wav"
+            
+            audio_file_path=os.path.join(full_output_folder, file)
+
+            torchaudio.save(audio_file_path, audio['waveform'].squeeze(0), audio["sample_rate"])
+            
+        # 获取文件名和扩展名
+        base, ext = os.path.splitext(video)
+        counter=get_new_counter(output_dir,'video_final_')
+
+        v_file = f"video_final_{counter:05}{ext}"
+        
+        v_file_path=os.path.join(output_dir, v_file)
+
+        combine_audio_video(audio_file_path,video,v_file_path)
+
+        previews = [
+            {
+                "filename": v_file,
+                "subfolder": "",
+                "type": "output",
+                "format": get_mime_type(v_file),
+            }
+        ]
+
+        return {"ui": {"gifs": previews},"result":(v_file_path,)}
 
 # The code is based on ComfyUI-VideoHelperSuite modification.
 class VideoCombine_Adv:
@@ -454,7 +652,8 @@ class VideoCombine_Adv:
             },
         }
 
-    RETURN_TYPES = ()
+    RETURN_TYPES = ("SCENE_VIDEO",)
+    RETURN_NAMES = ("scenes_video",)
     OUTPUT_NODE = True
     CATEGORY = "♾️Mixlab/Video"
     FUNCTION = "run"
@@ -623,7 +822,7 @@ class VideoCombine_Adv:
                 "format": format,
             }
         ]
-        return {"ui": {"gifs": previews}}
+        return {"ui": {"gifs": previews},"result":(file_path,)}
 
 
 class VAEEncodeForInpaint_Frames:
@@ -690,4 +889,113 @@ class VAEEncodeForInpaint_Frames:
             result.append({"samples":t, "noise_mask": (mask_erosion[:,:,:x,:y].round())})
 
 
-        return (result, )
+        return (result, )
+    
+
+
+class GenerateFramesByCount:
+    @classmethod
+    def INPUT_TYPES(cls):
+        
+        return {"required": {
+                    "frames": ('IMAGE',),
+                    "frame_count": ("INT", {"default": 72, "min": 1, "step": 1}), 
+                    "revert" :("BOOLEAN", {"default": True},),
+                     },}
+
+    RETURN_TYPES = ('IMAGE',)
+    RETURN_NAMES = ("frames",)
+
+    FUNCTION = "r"
+    CATEGORY = "♾️Mixlab/Video"
+    # INPUT_IS_LIST = True
+
+    def r(self, frames, frame_count, revert):
+        
+        image_list = [frames[i:i + 1, ...] for i in range(frames.shape[0])]
+
+        image_list=get_frames(frame_count,image_list,revert)
+
+        images = torch.cat(image_list, dim=0)
+
+        return (images,)
+
+
+class scenesNode_:
+    @classmethod
+    def INPUT_TYPES(cls):
+        
+        return {"required": {
+                    "scenes_video": ('SCENE_VIDEO',),
+                    "index": ("INT", {"default": 0, "min": 0, "step": 1}),
+                     
+                     },}
+
+    RETURN_TYPES = ('IMAGE','INT',)
+    RETURN_NAMES = ("video frames (batch)","count",)
+    # OUTPUT_IS_LIST = (False,)
+
+    FUNCTION = "run"
+    CATEGORY = "♾️Mixlab/Video"
+    INPUT_IS_LIST = True
+
+    def load_video_cv_fallback(self, video, frame_load_cap, skip_first_frames):
+        # print('#video',video)
+        try:
+            video_cap = cv2.VideoCapture(video)
+            if not video_cap.isOpened():
+                raise ValueError(f"{video} could not be loaded with cv fallback.")
+            # set video_cap to look at start_index frame
+            images = []
+            total_frame_count = 0
+            frames_added = 0
+            base_frame_time = 1/video_cap.get(cv2.CAP_PROP_FPS)
+            
+            target_frame_time = base_frame_time
+
+            time_offset=0.0
+            while video_cap.isOpened():
+                if time_offset < target_frame_time:
+                    is_returned, frame = video_cap.read()
+                    # if didn't return frame, video has ended
+                    if not is_returned:
+                        break
+                    time_offset += base_frame_time
+                if time_offset < target_frame_time:
+                    continue
+                time_offset -= target_frame_time
+                # if not at start_index, skip doing anything with frame
+                total_frame_count += 1
+                if total_frame_count <= skip_first_frames:
+                    continue
+                # TODO: do whatever operations need to happen, like force_size, etc
+
+                # opencv loads images in BGR format (yuck), so need to convert to RGB for ComfyUI use
+                # follow up: can videos ever have an alpha channel?
+                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+                # convert frame to comfyui's expected format (taken from comfy's load image code)
+                image = Image.fromarray(frame)
+                image = ImageOps.exif_transpose(image)
+                image = np.array(image, dtype=np.float32) / 255.0
+                image = torch.from_numpy(image)[None,]
+                images.append(image)
+                frames_added += 1
+                # if cap exists and we've reached it, stop processing frames
+                if frame_load_cap > 0 and frames_added >= frame_load_cap:
+                    break
+        finally:
+            video_cap.release()
+        
+        images = torch.cat(images, dim=0)
+       
+        return (images, frames_added,)
+
+    def run(self, scenes_video,index):
+        print('#scenes_video',index,scenes_video)
+        index=index[0]
+        if len(scenes_video) > index:
+            vp=scenes_video[index]
+        else:
+            vp=scenes_video[-1]
+        
+        return self.load_video_cv_fallback(vp,0,0)

nodes/VoiceStreamAI/asr/asr_factory.py

@@ -1,12 +0,0 @@
-from VoiceStreamAI.asr.whisper_asr import WhisperASR
-from VoiceStreamAI.asr.faster_whisper_asr import FasterWhisperASR
-
-class ASRFactory:
-    @staticmethod
-    def create_asr_pipeline(type, **kwargs):
-        if type == "whisper":
-            return WhisperASR(**kwargs)
-        if type == "faster_whisper":
-            return FasterWhisperASR(**kwargs)
-        else:
-            raise ValueError(f"Unknown ASR pipeline type: {type}")

nodes/VoiceStreamAI/asr/asr_interface.py

@@ -1,9 +0,0 @@
-class ASRInterface:
-    async def transcribe(self, client):
-        """
-        Transcribe the given audio data.
-
-        :param client: The client object with all the member variables including the buffer
-        :return: The transcription structure, see for example the faster_whisper_asr.py file.
-        """
-        raise NotImplementedError("This method should be implemented by subclasses.")

nodes/VoiceStreamAI/asr/faster_whisper_asr.py

@@ -1,142 +0,0 @@
-import os
-from faster_whisper import WhisperModel
-
-from VoiceStreamAI.asr.asr_interface import ASRInterface
-from VoiceStreamAI.audio_utils import save_audio_to_file
-
-import folder_paths
-
-language_codes = {
-    "afrikaans": "af",
-    "amharic": "am",
-    "arabic": "ar",
-    "assamese": "as",
-    "azerbaijani": "az",
-    "bashkir": "ba",
-    "belarusian": "be",
-    "bulgarian": "bg",
-    "bengali": "bn",
-    "tibetan": "bo",
-    "breton": "br",
-    "bosnian": "bs",
-    "catalan": "ca",
-    "czech": "cs",
-    "welsh": "cy",
-    "danish": "da",
-    "german": "de",
-    "greek": "el",
-    "english": "en",
-    "spanish": "es",
-    "estonian": "et",
-    "basque": "eu",
-    "persian": "fa",
-    "finnish": "fi",
-    "faroese": "fo",
-    "french": "fr",
-    "galician": "gl",
-    "gujarati": "gu",
-    "hausa": "ha",
-    "hawaiian": "haw",
-    "hebrew": "he",
-    "hindi": "hi",
-    "croatian": "hr",
-    "haitian": "ht",
-    "hungarian": "hu",
-    "armenian": "hy",
-    "indonesian": "id",
-    "icelandic": "is",
-    "italian": "it",
-    "japanese": "ja",
-    "javanese": "jw",
-    "georgian": "ka",
-    "kazakh": "kk",
-    "khmer": "km",
-    "kannada": "kn",
-    "korean": "ko",
-    "latin": "la",
-    "luxembourgish": "lb",
-    "lingala": "ln",
-    "lao": "lo",
-    "lithuanian": "lt",
-    "latvian": "lv",
-    "malagasy": "mg",
-    "maori": "mi",
-    "macedonian": "mk",
-    "malayalam": "ml",
-    "mongolian": "mn",
-    "marathi": "mr",
-    "malay": "ms",
-    "maltese": "mt",
-    "burmese": "my",
-    "nepali": "ne",
-    "dutch": "nl",
-    "norwegian nynorsk": "nn",
-    "norwegian": "no",
-    "occitan": "oc",
-    "punjabi": "pa",
-    "polish": "pl",
-    "pashto": "ps",
-    "portuguese": "pt",
-    "romanian": "ro",
-    "russian": "ru",
-    "sanskrit": "sa",
-    "sindhi": "sd",
-    "sinhalese": "si",
-    "slovak": "sk",
-    "slovenian": "sl",
-    "shona": "sn",
-    "somali": "so",
-    "albanian": "sq",
-    "serbian": "sr",
-    "sundanese": "su",
-    "swedish": "sv",
-    "swahili": "sw",
-    "tamil": "ta",
-    "telugu": "te",
-    "tajik": "tg",
-    "thai": "th",
-    "turkmen": "tk",
-    "tagalog": "tl",
-    "turkish": "tr",
-    "tatar": "tt",
-    "ukrainian": "uk",
-    "urdu": "ur",
-    "uzbek": "uz",
-    "vietnamese": "vi",
-    "yiddish": "yi",
-    "yoruba": "yo",
-    "chinese": "zh",
-    "cantonese": "yue",
-}
-
-
-class FasterWhisperASR(ASRInterface):
-    def __init__(self, **kwargs):
-        model_size = kwargs.get('model_size', "large-v3")
-        device = kwargs.get('device', "cuda")
-        model_root = os.path.join(folder_paths.models_dir, "whisper") 
-        # Run on GPU with FP16
-        self.asr_pipeline = WhisperModel(model_size, device=device, compute_type="float16",download_root=model_root)
-
-    async def transcribe(self, client):
-        file_path = await save_audio_to_file(client.scratch_buffer, client.get_file_name())
-
-        language = None if client.config['language'] is None else language_codes.get(client.config['language'].lower())
-        segments, info = self.asr_pipeline.transcribe(file_path, word_timestamps=True, language=language)
-
-        segments = list(segments)  # The transcription will actually run here.
-        os.remove(file_path)
-
-        flattened_words = [word for segment in segments for word in segment.words]
-
-        to_return = {
-        "language": info.language,
-        "language_probability": info.language_probability,
-        "text": ' '.join([s.text.strip() for s in segments]),
-        "words":
-            [
-                {"word": w.word, "start": w.start, "end": w.end, "probability":w.probability} for w in flattened_words
-            ]
-        }
-        return to_return
-

nodes/VoiceStreamAI/asr/whisper_asr.py

@@ -1,27 +0,0 @@
-from transformers import pipeline
-from VoiceStreamAI.asr.asr_interface import ASRInterface
-from VoiceStreamAI.audio_utils import save_audio_to_file
-import os
-
-class WhisperASR(ASRInterface):
-    def __init__(self, **kwargs):
-        model_name = kwargs.get('model_name', "openai/whisper-large-v3")
-        self.asr_pipeline = pipeline("automatic-speech-recognition", model=model_name)
-
-    async def transcribe(self, client):
-        file_path = await save_audio_to_file(client.scratch_buffer, client.get_file_name())
-        
-        if client.config['language'] is not None:
-            to_return = self.asr_pipeline(file_path, generate_kwargs={"language": client.config['language']})['text']
-        else:
-            to_return = self.asr_pipeline(file_path)['text']
-
-        os.remove(file_path)
-
-        to_return = {
-            "language": "UNSUPPORTED_BY_HUGGINGFACE_WHISPER",
-            "language_probability": None,
-            "text": to_return.strip(),
-            "words": "UNSUPPORTED_BY_HUGGINGFACE_WHISPER"
-        }
-        return to_return

nodes/VoiceStreamAI/audio_utils.py

@@ -1,26 +0,0 @@
-import wave
-import os
-
-async def save_audio_to_file(audio_data, file_name, audio_dir="audio_files", audio_format="wav"):
-    """
-    Saves the audio data to a file.
-
-    :param client_id: Unique identifier for the client.
-    :param audio_data: The audio data to save.
-    :param file_counters: Dictionary to keep track of file counts for each client.
-    :param audio_dir: Directory where audio files will be saved.
-    :param audio_format: Format of the audio file.
-    :return: Path to the saved audio file.
-    """
-
-    os.makedirs(audio_dir, exist_ok=True)
-    
-    file_path = os.path.join(audio_dir, file_name)
-
-    with wave.open(file_path, 'wb') as wav_file:
-        wav_file.setnchannels(1)  # Assuming mono audio
-        wav_file.setsampwidth(2)
-        wav_file.setframerate(16000)
-        wav_file.writeframes(audio_data)
-
-    return file_path

nodes/VoiceStreamAI/buffering_strategy/buffering_strategies.py

@@ -1,142 +0,0 @@
-import os
-import asyncio
-import json
-import time
-
-from VoiceStreamAI.buffering_strategy.buffering_strategy_interface import BufferingStrategyInterface
-from openai import OpenAI
-
-class SilenceAtEndOfChunk(BufferingStrategyInterface):
-    """
-    A buffering strategy that processes audio at the end of each chunk with silence detection.
-
-    This class is responsible for handling audio chunks, detecting silence at the end of each chunk,
-    and initiating the transcription process for the chunk.
-
-    Attributes:
-        client (Client): The client instance associated with this buffering strategy.
-        chunk_length_seconds (float): Length of each audio chunk in seconds.
-        chunk_offset_seconds (float): Offset time in seconds to be considered for processing audio chunks.
-    """
-
-    def __init__(self, client, **kwargs):
-        """
-        Initialize the SilenceAtEndOfChunk buffering strategy.
-
-        Args:
-            client (Client): The client instance associated with this buffering strategy.
-            **kwargs: Additional keyword arguments, including 'chunk_length_seconds' and 'chunk_offset_seconds'.
-        """
-        self.client = client
-
-        self.chunk_length_seconds = os.environ.get('BUFFERING_CHUNK_LENGTH_SECONDS')
-        if not self.chunk_length_seconds:
-            self.chunk_length_seconds = kwargs.get('chunk_length_seconds')
-        self.chunk_length_seconds = float(self.chunk_length_seconds)
-
-        self.chunk_offset_seconds = os.environ.get('BUFFERING_CHUNK_OFFSET_SECONDS')
-        if not self.chunk_offset_seconds:
-            self.chunk_offset_seconds = kwargs.get('chunk_offset_seconds')
-        self.chunk_offset_seconds = float(self.chunk_offset_seconds)
-
-        self.error_if_not_realtime = os.environ.get('ERROR_IF_NOT_REALTIME')
-        if not self.error_if_not_realtime:
-            self.error_if_not_realtime = kwargs.get('error_if_not_realtime', False)
-        
-        self.processing_flag = False
-
-        self.messages=[]
-
-    def process_audio(self, websocket, vad_pipeline, asr_pipeline,llm_port):
-        """
-        Process audio chunks by checking their length and scheduling asynchronous processing.
-
-        This method checks if the length of the audio buffer exceeds the chunk length and, if so,
-        it schedules asynchronous processing of the audio.
-
-        Args:
-            websocket (Websocket): The WebSocket connection for sending transcriptions.
-            vad_pipeline: The voice activity detection pipeline.
-            asr_pipeline: The automatic speech recognition pipeline.
-        """
-        chunk_length_in_bytes = self.chunk_length_seconds * self.client.sampling_rate * self.client.samples_width
-        if len(self.client.buffer) > chunk_length_in_bytes:
-            if self.processing_flag:
-                exit("Error in realtime processing: tried processing a new chunk while the previous one was still being processed")
-
-            self.client.scratch_buffer += self.client.buffer
-            self.client.buffer.clear()
-            self.processing_flag = True
-            # Schedule the processing in a separate task
-            asyncio.create_task(self.process_audio_async(websocket, vad_pipeline, asr_pipeline,llm_port))
-    
-    async def process_audio_async(self, websocket, vad_pipeline, asr_pipeline,llm_port):
-        """
-        Asynchronously process audio for activity detection and transcription.
-
-        This method performs heavy processing, including voice activity detection and transcription of
-        the audio data. It sends the transcription results through the WebSocket connection.
-
-        Args:
-            websocket (Websocket): The WebSocket connection for sending transcriptions.
-            vad_pipeline: The voice activity detection pipeline.
-            asr_pipeline: The automatic speech recognition pipeline.
-        """   
-        start = time.time()
-        vad_results = await vad_pipeline.detect_activity(self.client)
-
-        if len(vad_results) == 0:
-            self.client.scratch_buffer.clear()
-            self.client.buffer.clear()
-            self.processing_flag = False
-            return
-
-        last_segment_should_end_before = ((len(self.client.scratch_buffer) / (self.client.sampling_rate * self.client.samples_width)) - self.chunk_offset_seconds)
-        if vad_results[-1]['end'] < last_segment_should_end_before:
-            transcription = await asr_pipeline.transcribe(self.client)
-            if transcription['text'] != '':
-                end = time.time()
-                transcription['processing_time'] = end - start
-
-                transcription['status']="chat_start"
-
-                json_transcription = json.dumps(transcription)
-
-                await websocket.send(json_transcription)
-
-                # Point to the local server
-                client = OpenAI(base_url=f"http://localhost:{llm_port}/v1", api_key="lm-studio")
-
-               
-                messages=[
-                    {"role": "system", "content": "You are a friendly and engaging AI designed to interact with users in a conversational manner. Your personality is that of a sophisticated and polite young professional who is both a designer and a programmer. You are well-mannered, articulate, and possess a good sense of humor. Your goal is to provide helpful and insightful responses while maintaining a pleasant and enjoyable conversation. Be sure to use your knowledge in design and programming to enrich the dialogue and offer relevant advice or information when appropriate. Always be respectful and considerate of the user's feelings and perspectives. Additionally, you are fluent in both English and Chinese, and can seamlessly switch between the two languages to best assist users."}, 
-                ]+self.messages[-10:0]+[{"role": "user", "content":transcription['text']}]
-                # print('#messages',messages)
-                
-                completion = client.chat.completions.create(
-                    model="model-identifier",
-                    messages=messages,
-                    temperature=0.7,
-                    )
-
-                transcription['asistant'] = completion.choices[0].message.content
-
-                transcription['status']="chat_end"
-
-                json_transcription = json.dumps(transcription) 
-
-                self.messages.append({
-                    "role": "user", 
-                    "content":transcription['text']})
-                
-                self.messages.append({
-                    "role": "asistant", 
-                    "content": transcription['asistant']
-                                      })
-                # print('#messages',completion.choices[0].message.content)
-
-                await websocket.send(json_transcription)
-            self.client.scratch_buffer.clear()
-            self.client.increment_file_counter()
-        
-        self.processing_flag = False

nodes/VoiceStreamAI/buffering_strategy/buffering_strategy_factory.py

@@ -1,41 +0,0 @@
-from VoiceStreamAI.buffering_strategy.buffering_strategies import SilenceAtEndOfChunk
-
-class BufferingStrategyFactory:
-    """
-    A factory class for creating instances of different buffering strategies.
-
-    This factory provides a centralized way to instantiate various buffering strategies
-    based on the type specified. It abstracts the creation logic, making it easier to
-    manage and extend with new buffering strategy types.
-
-    Methods:
-        create_buffering_strategy: Creates and returns an instance of a specified buffering strategy.
-    """
-
-    @staticmethod
-    def create_buffering_strategy(type, client, **kwargs):
-        """
-        Creates an instance of a buffering strategy based on the specified type.
-
-        This method acts as a factory for creating buffering strategy objects. It returns
-        an instance of the strategy corresponding to the given type. If the type is not
-        recognized, it raises a ValueError.
-
-        Args:
-            type (str): The type of buffering strategy to create. Currently supports 'silence_at_end_of_chunk'.
-            client (Client): The client instance to be associated with the buffering strategy.
-            **kwargs: Additional keyword arguments specific to the buffering strategy being created.
-
-        Returns:
-            An instance of the specified buffering strategy.
-
-        Raises:
-            ValueError: If the specified type is not recognized or supported.
-
-        Example:
-            strategy = BufferingStrategyFactory.create_buffering_strategy("silence_at_end_of_chunk", client)
-        """
-        if type == "silence_at_end_of_chunk":
-            return SilenceAtEndOfChunk(client, **kwargs)
-        else:
-            raise ValueError(f"Unknown buffering strategy type: {type}")

nodes/VoiceStreamAI/buffering_strategy/buffering_strategy_interface.py

@@ -1,31 +0,0 @@
-class BufferingStrategyInterface:
-    """
-    An interface class for buffering strategies in audio processing systems.
-
-    This class defines the structure for buffering strategies used in handling
-    and processing audio data. It serves as a template for creating custom buffering
-    strategies that fit specific requirements of an audio processing pipeline.
-
-    Subclasses should implement the methods defined in this interface to ensure
-    consistency and compatibility with the system's audio processing framework.
-
-    Methods:
-        process_audio: Process audio data. This method should be implemented by subclasses.
-    """
-
-    def process_audio(self, websocket, vad_pipeline, asr_pipeline):
-        """
-        Process audio data using the given WebSocket connection, VAD pipeline, and ASR pipeline.
-
-        This method is intended to be overridden in subclasses to provide specific logic
-        for handling and processing audio data in different buffering strategies.
-
-        Args:
-            websocket (Websocket): The WebSocket connection for communication with clients.
-            vad_pipeline: The Voice Activity Detection (VAD) pipeline used for detecting speech in the audio.
-            asr_pipeline: The Automatic Speech Recognition (ASR) pipeline used for transcribing speech in the audio.
-
-        Raises:
-            NotImplementedError: If the method is not implemented in the subclass.
-        """
-        raise NotImplementedError("This method should be implemented by subclasses.")

nodes/VoiceStreamAI/client.py

@@ -1,54 +0,0 @@
-from VoiceStreamAI.buffering_strategy.buffering_strategy_factory import BufferingStrategyFactory
-
-class Client:
-    """
-    Represents a client connected to the VoiceStreamAI server.
-
-    This class maintains the state for each connected client, including their
-    unique identifier, audio buffer, configuration, and a counter for processed audio files.
-
-    Attributes:
-        client_id (str): A unique identifier for the client.
-        buffer (bytearray): A buffer to store incoming audio data.
-        config (dict): Configuration settings for the client, like chunk length and offset.
-        file_counter (int): Counter for the number of audio files processed.
-        total_samples (int): Total number of audio samples received from this client.
-        sampling_rate (int): The sampling rate of the audio data in Hz.
-        samples_width (int): The width of each audio sample in bits.
-    """
-    def __init__(self, client_id, sampling_rate, samples_width):
-        self.client_id = client_id
-        self.buffer = bytearray()
-        self.scratch_buffer = bytearray()
-        self.config = {"language": None,
-                       "processing_strategy": "silence_at_end_of_chunk", 
-                       "processing_args": {
-                           "chunk_length_seconds": 5, 
-                           "chunk_offset_seconds": 0.1
-                           }
-                       }
-        self.file_counter = 0
-        self.total_samples = 0
-        self.sampling_rate = sampling_rate
-        self.samples_width = samples_width
-        self.buffering_strategy = BufferingStrategyFactory.create_buffering_strategy(self.config['processing_strategy'], self, **self.config['processing_args'])
-
-    def update_config(self, config_data):
-        self.config.update(config_data)
-        self.buffering_strategy = BufferingStrategyFactory.create_buffering_strategy(self.config['processing_strategy'], self, **self.config['processing_args'])
-
-    def append_audio_data(self, audio_data):
-        self.buffer.extend(audio_data)
-        self.total_samples += len(audio_data) / self.samples_width
-
-    def clear_buffer(self):
-        self.buffer.clear()
-
-    def increment_file_counter(self):
-        self.file_counter += 1
-
-    def get_file_name(self):
-        return f"{self.client_id}_{self.file_counter}.wav"
-    
-    def process_audio(self, websocket, vad_pipeline, asr_pipeline,llm_port):
-        self.buffering_strategy.process_audio(websocket, vad_pipeline, asr_pipeline,llm_port)

nodes/VoiceStreamAI/main.py

@@ -1,54 +0,0 @@
-import argparse
-import asyncio
-import json
-
-import os
-import sys
-from pathlib import Path
-
-
-# 获取当前文件的绝对路径
-current_file_path = os.path.abspath(__file__)
-
-# 获取当前文件的目录
-current_directory = os.path.dirname(current_file_path)
-sys.path.append(str(Path(current_directory).parent))
-# print("sys.path", current_directory)
-
-
-from VoiceStreamAI.server import Server
-from VoiceStreamAI.asr.asr_factory import ASRFactory
-from VoiceStreamAI.vad.vad_factory import VADFactory
-
-def parse_args():
-    parser = argparse.ArgumentParser(description="VoiceStreamAI Server: Real-time audio transcription using self-hosted Whisper and WebSocket")
-    parser.add_argument("--vad-type", type=str, default="pyannote", help="Type of VAD pipeline to use (e.g., 'pyannote')")
-    parser.add_argument("--vad-args", type=str, default='{"auth_token": "huggingface_token"}', help="JSON string of additional arguments for VAD pipeline")
-    parser.add_argument("--asr-type", type=str, default="faster_whisper", help="Type of ASR pipeline to use (e.g., 'whisper')")
-    parser.add_argument("--asr-args", type=str, default='{"model_size": "large-v3"}', help="JSON string of additional arguments for ASR pipeline")
-    parser.add_argument("--host", type=str, default="127.0.0.1", help="Host for the WebSocket server")
-    parser.add_argument("--port", type=int, default=8765, help="Port for the WebSocket server")
-    parser.add_argument("--certfile", type=str, default=None, help="The path to the SSL certificate (cert file) if using secure websockets")
-    parser.add_argument("--keyfile", type=str, default=None, help="The path to the SSL key file if using secure websockets")
-    return parser.parse_args()
-
-def main():
-    args = parse_args()
-
-    try:
-        vad_args = json.loads(args.vad_args)
-        asr_args = json.loads(args.asr_args)
-    except json.JSONDecodeError as e:
-        print(f"Error parsing JSON arguments: {e}")
-        return
-
-    vad_pipeline = VADFactory.create_vad_pipeline(args.vad_type, **vad_args)
-    asr_pipeline = ASRFactory.create_asr_pipeline(args.asr_type, **asr_args)
-
-    server = Server(vad_pipeline, asr_pipeline, host=args.host, port=args.port, sampling_rate=16000, samples_width=2, certfile=args.certfile, keyfile=args.keyfile)
-
-    asyncio.get_event_loop().run_until_complete(server.start())
-    asyncio.get_event_loop().run_forever()
-
-if __name__ == "__main__":
-    main()

nodes/VoiceStreamAI/requirements.txt

@@ -1,7 +0,0 @@
-websockets
-speechbrain
-pyannote-audio
-asyncio
-sentence-transformers
-transformers
-faster-whisper

nodes/VoiceStreamAI/server.py

@@ -1,88 +0,0 @@
-import websockets
-import uuid
-import json
-import asyncio
-import ssl
-
-from VoiceStreamAI.audio_utils import save_audio_to_file
-from VoiceStreamAI.client import Client
-
-class Server:
-    """
-    Represents the WebSocket server for handling real-time audio transcription.
-
-    This class manages WebSocket connections, processes incoming audio data,
-    and interacts with VAD and ASR pipelines for voice activity detection and
-    speech recognition.
-
-    Attributes:
-        vad_pipeline: An instance of a voice activity detection pipeline.
-        asr_pipeline: An instance of an automatic speech recognition pipeline.
-        host (str): Host address of the server.
-        port (int): Port on which the server listens.
-        sampling_rate (int): The sampling rate of audio data in Hz.
-        samples_width (int): The width of each audio sample in bits.
-        connected_clients (dict): A dictionary mapping client IDs to Client objects.
-    """
-    def __init__(self, vad_pipeline, asr_pipeline, host='localhost', port=8765, sampling_rate=16000, samples_width=2, certfile = None, keyfile = None,llm_port=9000):
-        self.vad_pipeline = vad_pipeline
-        self.asr_pipeline = asr_pipeline
-        self.host = host
-        self.port = port
-        self.sampling_rate = sampling_rate
-        self.samples_width = samples_width
-        self.certfile = certfile
-        self.keyfile = keyfile
-        self.connected_clients = {}
-
-        self.llm_port=llm_port
-
-    async def handle_audio(self, client, websocket):
-        while True:
-            message = await websocket.recv()
-
-            if isinstance(message, bytes):
-                client.append_audio_data(message)
-            elif isinstance(message, str):
-                config = json.loads(message)
-                if config.get('type') == 'config':
-                    client.update_config(config['data'])
-                    continue
-            else:
-                print(f"Unexpected message type from {client.client_id}")
-
-            # this is synchronous, any async operation is in BufferingStrategy
-            client.process_audio(websocket, self.vad_pipeline, self.asr_pipeline,self.llm_port)
-
-
-    async def handle_websocket(self, websocket, path):
-        client_id = str(uuid.uuid4())
-        client = Client(client_id, self.sampling_rate, self.samples_width)
-        self.connected_clients[client_id] = client
-
-        print(f"Client {client_id} connected")
-
-        try:
-            await self.handle_audio(client, websocket)
-        except websockets.ConnectionClosed as e:
-            print(f"Connection with {client_id} closed: {e}")
-        finally:
-            del self.connected_clients[client_id]
-
-    def start(self):
-        if self.certfile:
-            # Create an SSL context to enforce encrypted connections
-            ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
-            
-            # Load your server's certificate and private key
-            # Replace 'your_cert_path.pem' and 'your_key_path.pem' with the actual paths to your files
-            ssl_context.load_cert_chain(certfile=self.certfile, keyfile=self.keyfile)
-
-            print(f"WebSocket server ready to accept secure connections on {self.host}:{self.port}")
-            
-            # Pass the SSL context to the serve function along with the host and port
-            # Ensure the secure flag is set to True if using a secure WebSocket protocol (wss://)
-            return websockets.serve(self.handle_websocket, self.host, self.port, ssl=ssl_context)
-        else:
-            print(f"WebSocket server ready to accept secure connections on {self.host}:{self.port}")
-            return websockets.serve(self.handle_websocket, self.host, self.port)

nodes/VoiceStreamAI/vad/pyannote_vad.py

@@ -1,50 +0,0 @@
-from os import remove
-import os
-
-from pyannote.core import Segment
-from pyannote.audio import Model
-from pyannote.audio.pipelines import VoiceActivityDetection
-
-from VoiceStreamAI.vad.vad_interface import VADInterface
-from VoiceStreamAI.audio_utils import save_audio_to_file
-
-
-class PyannoteVAD(VADInterface):
-    """
-    Pyannote-based implementation of the VADInterface.
-    """
-
-    def __init__(self, **kwargs):
-        """
-        Initializes Pyannote's VAD pipeline.
-
-        Args:
-            model_name (str): The model name for Pyannote.
-            auth_token (str, optional): Authentication token for Hugging Face.
-        """
-        
-        model_name = kwargs.get('model_name', "pyannote/segmentation")
-
-        auth_token = os.environ.get('PYANNOTE_AUTH_TOKEN')
-        if not auth_token:
-            auth_token = kwargs.get('auth_token')
-        
-        if auth_token is None:
-            raise ValueError("Missing required env var in PYANNOTE_AUTH_TOKEN or argument in --vad-args: 'auth_token'")
-        
-        pyannote_args = kwargs.get('pyannote_args', {"onset": 0.5, "offset": 0.5, "min_duration_on": 0.3, "min_duration_off": 0.3})
-        self.model = Model.from_pretrained(model_name, use_auth_token=auth_token)
-        self.vad_pipeline = VoiceActivityDetection(segmentation=self.model)
-        self.vad_pipeline.instantiate(pyannote_args)
-
-    async def detect_activity(self, client):
-        audio_file_path = await save_audio_to_file(client.scratch_buffer, client.get_file_name())
-        vad_results = self.vad_pipeline(audio_file_path)
-        remove(audio_file_path)
-        vad_segments = []
-        if len(vad_results) > 0:
-            vad_segments = [
-                {"start": segment.start, "end": segment.end, "confidence": 1.0}
-                for segment in vad_results.itersegments()
-            ]
-        return vad_segments

nodes/VoiceStreamAI/vad/vad_factory.py

@@ -1,23 +0,0 @@
-from VoiceStreamAI.vad.pyannote_vad import PyannoteVAD
-
-class VADFactory:
-    """
-    Factory for creating instances of VAD systems.
-    """
-
-    @staticmethod
-    def create_vad_pipeline(type, **kwargs):
-        """
-        Creates a VAD pipeline based on the specified type.
-
-        Args:
-            type (str): The type of VAD pipeline to create (e.g., 'pyannote').
-            kwargs: Additional arguments for the VAD pipeline creation.
-
-        Returns:
-            VADInterface: An instance of a class that implements VADInterface.
-        """
-        if type == "pyannote":
-            return PyannoteVAD(**kwargs)
-        else:
-            raise ValueError(f"Unknown VAD pipeline type: {type}")

nodes/VoiceStreamAI/vad/vad_interface.py

@@ -1,16 +0,0 @@
-class VADInterface:
-    """
-    Interface for voice activity detection (VAD) systems.
-    """
-
-    async def detect_activity(self, client):
-        """
-        Detects voice activity in the given audio data.
-
-        Args:
-            client (src.Client): The client to detect on
-
-        Returns:
-            List: VAD result, a list of objects containing "start", "end", "confidence"
-        """
-        raise NotImplementedError("This method should be implemented by subclasses.")

nodes/edit_mask.py

@@ -0,0 +1,172 @@
+import torch
+from PIL import Image, ImageOps, ImageSequence, ImageFile
+from PIL.PngImagePlugin import PngInfo
+
+import numpy as np
+import os
+import folder_paths 
+import node_helpers
+import hashlib
+
+# Tensor to PIL
+def tensor2pil(image):
+    return Image.fromarray(np.clip(255. * image.cpu().numpy().squeeze(), 0, 255).astype(np.uint8))
+
+# tensor 取hash值
+def tensor_to_hash(tensor):
+    # 将 Tensor 转换为 NumPy 数组
+    np_array = tensor.cpu().numpy()
+    
+    # 将 NumPy 数组转换为字节数据
+    byte_data = np_array.tobytes()
+    
+    # 计算哈希值
+    hash_value = hashlib.md5(byte_data).hexdigest()
+    
+    return hash_value
+
+
+def create_temp_file(image):
+    output_dir = folder_paths.get_temp_directory()
+
+    (
+            full_output_folder,
+            filename,
+            counter,
+            subfolder,
+            _,
+        ) = folder_paths.get_save_image_path('material', output_dir)
+
+    
+    image=tensor2pil(image)
+ 
+    image_file = f"{filename}_{counter:05}.png"
+     
+    image_path=os.path.join(full_output_folder, image_file)
+
+    image.save(image_path,compress_level=4)
+
+    return (image_path,[{
+    "filename": image_file,
+    "subfolder": subfolder,
+    "type": "temp"
+    }])
+
+
+# image - tensor - 文件路径
+# loadImage的方法（ 文件路径 - image-mask ）
+class EditMask:
+
+    def __init__(self):
+        self.image_id = None
+
+    @classmethod
+    def INPUT_TYPES(s):
+        return {"required":
+                    {"image": ("IMAGE",), # 表示一个张量
+                     
+                     },
+
+                    "optional":{
+                            "image_update": ("IMAGE_FILE",)
+                        },
+                   
+                }
+
+    CATEGORY = "♾️Mixlab/Mask"
+
+    RETURN_TYPES = ("IMAGE", "MASK")
+    RETURN_NAMES = ("image", "mask")
+
+    FUNCTION = "edit"
+
+    OUTPUT_NODE = True
+
+    def edit(self, image,image_update=None):
+
+        # 根据image输入来判断是否是新的图片
+        if self.image_id==None:
+            self.image_id=tensor_to_hash(image)
+            image_update=None
+        else:
+            image_id=tensor_to_hash(image)
+            if image_id!=self.image_id:
+                image_update=None
+                self.image_id=image_id
+
+
+        image_path=None
+        # print('#image_update',self.image_id,image_update)
+        if image_update==None:
+            print('--')
+        else:
+            if 'images' in image_update:
+                images=image_update['images']
+                filename=images[0]['filename']
+                subfolder=images[0]['subfolder']
+                type=images[0]['type']
+                name, base_dir=folder_paths.annotated_filepath(filename)
+                if type.endswith("output"):
+                    base_dir = folder_paths.get_output_directory() 
+                elif type.endswith("input"):
+                    base_dir = folder_paths.get_input_directory() 
+                elif type.endswith("temp"):
+                    base_dir = folder_paths.get_temp_directory() 
+                #base_dir = folder_paths.get_input_directory()  
+                # print(base_dir,subfolder, name)
+                image_path = os.path.join(base_dir,subfolder, name)
+        
+        if image_path==None:
+            image_path,images=create_temp_file(image)
+
+        print('#image_path',os.path.exists(image_path),image_path)
+        # image_path = folder_paths.get_annotated_filepath(image) #文件名
+        
+        if not os.path.exists(image_path):
+            image_path,images=create_temp_file(image)
+
+
+        img = node_helpers.pillow(Image.open, image_path)
+        
+        output_images = []
+        output_masks = []
+        w, h = None, None
+
+        excluded_formats = ['MPO']
+        
+        for i in ImageSequence.Iterator(img):
+            i = node_helpers.pillow(ImageOps.exif_transpose, i)
+
+            if i.mode == 'I':
+                i = i.point(lambda i: i * (1 / 255))
+            image = i.convert("RGB")
+
+            if len(output_images) == 0:
+                w = image.size[0]
+                h = image.size[1]
+            
+            if image.size[0] != w or image.size[1] != h:
+                continue
+            
+            image = np.array(image).astype(np.float32) / 255.0
+            image = torch.from_numpy(image)[None,]
+            if 'A' in i.getbands():
+                mask = np.array(i.getchannel('A')).astype(np.float32) / 255.0
+                mask = 1. - torch.from_numpy(mask)
+            else:
+                # 尺寸不对，需要按照image来
+                mask = torch.zeros((h, w), dtype=torch.float32, device="cpu")
+
+            output_images.append(image)
+            output_masks.append(mask.unsqueeze(0))
+
+        if len(output_images) > 1 and img.format not in excluded_formats:
+            output_image = torch.cat(output_images, dim=0)
+            output_mask = torch.cat(output_masks, dim=0)
+        else:
+            output_image = output_images[0]
+            output_mask = output_masks[0]
+
+        return {"ui":{"images": images},"result": (output_image, output_mask)}
+
+        # return (output_image, output_mask)

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "comfyui-mixlab-nodes"
 description = "3D, ScreenShareNode & FloatingVideoNode, SpeechRecognition & SpeechSynthesis, GPT, LoadImagesFromLocal, Layers, Other Nodes, ..."
-version = "0.28.3"
+version = "0.35.2"
 license = "MIT"
 dependencies = ["numpy", "pyOpenSSL", "watchdog", "opencv-python-headless", "matplotlib", "openai", "simple-lama-inpainting", "clip-interrogator==0.6.0", "transformers>=4.36.0", "lark-parser", "imageio-ffmpeg", "rembg[gpu]", "omegaconf==2.3.0", "Pillow>=9.5.0", "einops==0.7.0", "trimesh>=4.0.5", "huggingface-hub", "scikit-image"]
 

requirements.txt

@@ -4,7 +4,7 @@ watchdog
 opencv-python-headless
 matplotlib
 openai
-simple-lama-inpainting
+# simple-lama-inpainting
 clip-interrogator==0.6.0
 transformers>=4.36.0
 lark-parser
@@ -15,4 +15,7 @@ Pillow>=9.5.0
 einops==0.7.0
 trimesh>=4.0.5
 huggingface-hub
-scikit-image
+scikit-image
+torchaudio
+soundfile>=0.12.1
+json-repair

web/ar.html

@@ -0,0 +1,22 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Mixlab AR</title>
+</head>
+
+<body>
+
+    <script type="module">
+
+        import { api } from "../../../scripts/api.js";
+        import Command from '/extensions/comfyui-mixlab-nodes/javascript/command.js'
+
+
+    </script>
+
+</body>
+
+</html>

web/javascript/app_mixlab.js

@@ -2,6 +2,9 @@ import { app } from '../../../scripts/app.js'
 import { $el } from '../../../scripts/ui.js'
 import { api } from '../../../scripts/api.js'
 
+import { td_bg } from './td_background.js'
+// console.log('td_bg', td_bg)
+
 //本机安装的插件节点全集
 window._nodesAll = null
 
@@ -185,6 +188,25 @@ async function extractInputAndOutputData (
         if (node.type == 'Color') {
         }
 
+        // 语音输入的支持
+        if (node.type == 'LoadAndCombinedAudio_') {
+          // if (
+          //   data[id].widgets_values &&
+          //   data[id].widgets_values[0] &&
+          //   data[id].widgets_values[0].base64 &&
+          //   data[id].widgets_values[0].base64.length > 0
+          // ) {
+          //   options.defaultBase64 = data[id].widgets_values[0].base64
+          // }
+
+          input[inputIds.indexOf(id)] = {
+            ...data[id],
+            title: node.title,
+            id,
+            options
+          }
+        }
+
         if (node.type === 'LoadImage') {
           // loadImage的mask支持
           let output = node.outputs.filter(ot => ot.type == 'MASK')[0]
@@ -235,7 +257,9 @@ async function extractInputAndOutputData (
         node.type === 'KSampler' ||
         node.type == 'SamplerCustom' ||
         node.type === 'ChinesePrompt_Mix' ||
-        node.type === 'Seed_'
+        node.type === 'Seed_'||
+        node.type==='SiliconflowLLM'||
+        node.type==='ChatGPTOpenAI'
       ) {
         // seed 的类型收集
         try {
@@ -397,11 +421,11 @@ async function save (json, download = false, showInfo = true) {
 
 function getInputsAndOutputs () {
   const inputs =
-      `LoadImage LoadImagesToBatch ImagesPrompt_ VHS_LoadVideo CLIPTextEncode PromptSlide TextInput_ Color FloatSlider IntNumber CheckpointLoaderSimple LoraLoader`.split(
+      `LoadImage LoadImagesToBatch ImagesPrompt_ LoadAndCombinedAudio_ LoadVideoAndSegment_ VHS_LoadVideo CLIPTextEncode PromptSlide TextInput_ Color FloatSlider IntNumber CheckpointLoaderSimple LoraLoader`.split(
         ' '
       ),
     outputs =
-      `SaveTripoSRMesh,PreviewImage,SaveImage,TransparentImage,ShowTextForGPT,VHS_VideoCombine,VideoCombine_Adv,Image Save,SaveImageAndMetadata_,ClipInterrogator`.split(
+      `SaveTripoSRMesh,PreviewImage,SaveImage,TransparentImage,ShowTextForGPT,CombineAudioVideo,VHS_VideoCombine,VideoCombine_Adv,Image Save,SaveImageAndMetadata_,ClipInterrogator`.split(
         ','
       )
 
@@ -445,19 +469,18 @@ app.registerExtension({
         const { input, output } = getInputsAndOutputs()
         input_ids.value = input.join('\n')
         output_ids.value = output.join('\n')
-
         const widget = {
           type: 'div',
           name: 'AppInfoRun',
           draw (ctx, node, widget_width, y, widget_height) {
             Object.assign(
               this.div.style,
-              get_position_style(
+              {...get_position_style(
                 ctx,
                 widget_width,
                 node.size[1] - widget_height,
                 node.size[1]
-              )
+              ),zIndex:1}
             )
           }
         }
@@ -503,6 +526,21 @@ app.registerExtension({
           }
         })
 
+        //td bg
+        const tdBG = document.createElement('button')
+        tdBG.innerText = 'Canvas Mode'
+        tdBG.style = style
+        tdBG.style.marginLeft = '12px'
+
+        tdBG.addEventListener('click', () => {
+          td_bg.toggle()
+          if (td_bg.running) {
+            tdBG.style.background = 'yellow'
+          } else {
+            tdBG.style.background = 'transparent'
+          }
+        })
+
         // author
         let author = document.createElement('div')
         // author.style=`display: flex`
@@ -659,6 +697,7 @@ app.registerExtension({
 
         btns.appendChild(btn)
         btns.appendChild(download)
+        btns.appendChild(tdBG)
 
         document.body.appendChild(widget.div)
         this.addCustomWidget(widget)
@@ -672,6 +711,7 @@ app.registerExtension({
         this.serialize_widgets = true //需要保存参数
 
         window._mixlab_app_json = null
+ 
       }
 
       const onExecuted = nodeType.prototype.onExecuted
@@ -687,9 +727,8 @@ app.registerExtension({
           }
 
           const div = this.widgets.filter(w => w.div)[0].div
-          Array.from(
-            div.querySelectorAll('button'),
-            b => (b.style.background = 'yellow')
+          Array.from(div.querySelectorAll('button'), b =>
+            b.innerText != 'Canvas Mode' ? (b.style.background = 'yellow') : ''
           )
         } catch (error) {}
       }

web/javascript/audio_mixlab.js

@@ -396,3 +396,218 @@ app.registerExtension({
     }
   }
 })
+
+// 上传音频转为base64
+async function uploadAndConvertAudio (file) {
+  if (!file) {
+    alert('Please select a WAV file.')
+    return
+  }
+
+  if (file.type !== 'audio/wav') {
+    alert('Only WAV files are supported.')
+    return
+  }
+
+  try {
+    const base64Audio = await readFileAsDataURL(file)
+    return base64Audio
+  } catch (error) {
+    console.error('Error reading file:', error)
+    alert('Error reading file.')
+  }
+}
+
+function readFileAsDataURL (file) {
+  return new Promise((resolve, reject) => {
+    const reader = new FileReader()
+
+    reader.onload = function (event) {
+      resolve(event.target.result)
+    }
+
+    reader.onerror = function (error) {
+      reject(error)
+    }
+
+    reader.readAsDataURL(file)
+  })
+}
+
+const createInputAudioForBatch = (base64, widget) => {
+  // Create an audio element
+  let audio = document.createElement('audio')
+  audio.src = base64
+  audio.controls = true
+  audio.style = 'width: 120px; display: block'
+
+  // Create a delete button
+  let deleteButton = document.createElement('button')
+  deleteButton.textContent = 'Delete'
+
+  deleteButton.style = `cursor: pointer;
+        font-weight: 300;
+        margin: 2px; 
+        margin-left: 10px;
+        color: var(--descrip-text);
+        background-color: var(--comfy-input-bg);
+        border-radius: 8px;
+        border-color: var(--border-color);
+        border-style: solid;height: 30px;min-width: 122px;
+       `
+
+  // Create a container for the audio and delete button
+  let container = document.createElement('div')
+  container.appendChild(audio)
+  container.appendChild(deleteButton)
+  container.style = `display: flex;margin-top: 12px;`
+
+  // Add event listener for the delete button
+  deleteButton.addEventListener('click', e => {
+    let newValue = []
+    let items = widget.value?.base64 || []
+    for (const v of items) {
+      if (v != base64) newValue.push(v)
+    }
+    widget.value.base64 = newValue
+    container.remove()
+  })
+
+  return container
+}
+
+app.registerExtension({
+  name: 'Mixlab.Comfy.LoadAndCombinedAudio_',
+  async getCustomWidgets (app) {
+    return {
+      AUDIOBASE64 (node, inputName, inputData, app) {
+        // console.log('##node', node)
+        const widget = {
+          value: {
+            base64: []
+          }, // 不能[x,x,x]
+          type: inputData[0], // the type
+          name: inputName, // the name, slice
+          size: [128, 32], // a default size
+          draw (ctx, node, width, y) {},
+          computeSize (...args) {
+            return [128, 122] // a method to compute the current size of the widget
+          }
+          // serializeValue (nodeId, widgetIndex) {
+          //   return widget.value
+          // },
+        }
+        //  widget.something = something;          // maybe adds stuff to it
+        node.addCustomWidget(widget) // adds it to the node
+        return widget // and returns it.
+      }
+    }
+  },
+
+  async beforeRegisterNodeDef (nodeType, nodeData, app) {
+    if (nodeType.comfyClass == 'LoadAndCombinedAudio_') {
+      const orig_nodeCreated = nodeType.prototype.onNodeCreated
+
+      nodeType.prototype.onNodeCreated = function () {
+        orig_nodeCreated?.apply(this, arguments)
+
+        let audiosWidget = this.widgets.filter(w => w.name == 'audios')[0]
+
+        const widget = {
+          type: 'div',
+          name: 'audio_base64',
+          draw (ctx, node, widget_width, y, widget_height) {
+            Object.assign(
+              this.div.style,
+              get_position_style(ctx, widget_width, 44, node.size[1])
+            )
+          },
+          serialize: false
+        }
+
+        widget.div = $el('div', {})
+
+        document.body.appendChild(widget.div)
+
+        let audioPreview = document.createElement('div')
+        let audiosDiv = document.createElement('div') //显示图片
+        audiosDiv.className = 'audios_preview'
+        audiosDiv.style = `width: calc(100% - 14px);
+        display: flex;
+        flex-wrap: wrap;
+        padding: 7px;    justify-content: space-between;
+        align-items: center;`
+
+        const btn = document.createElement('button')
+        btn.innerText = 'Upload Audio'
+
+        btn.style = `cursor: pointer;
+        font-weight: 300;
+        margin: 2px; 
+        color: var(--descrip-text);
+        background-color: var(--comfy-input-bg);
+        border-radius: 8px;
+        border-color: var(--border-color);
+        border-style: solid;height: 30px;min-width: 122px;
+       `
+
+        btn.addEventListener('click', e => {
+          e.preventDefault()
+          let inputAudio = document.createElement('input')
+          inputAudio.type = 'file'
+          inputAudio.accept = "audio/*"
+          inputAudio.style.display = 'none'
+          inputAudio.addEventListener('change', async e => {
+            e.preventDefault()
+            const file = e.target.files[0]
+            let base64 = await uploadAndConvertAudio(file)
+            if (!audiosWidget.value) audiosWidget.value = { base64: [] }
+            audiosWidget.value.base64.push(base64)
+
+            let a = createInputAudioForBatch(base64, audiosWidget)
+            audiosDiv.appendChild(a)
+          })
+
+          inputAudio.click()
+          inputAudio.remove()
+        })
+
+        widget.div.appendChild(audioPreview)
+        audioPreview.appendChild(audiosDiv)
+        audioPreview.appendChild(btn)
+        // audioPreview.appendChild(inputAudio)
+
+        this.addCustomWidget(widget)
+
+        // document.addEventListener('wheel', handleMouseWheel)
+
+        const onRemoved = this.onRemoved
+        this.onRemoved = () => {
+          widget.div.remove()
+          try {
+            // document.removeEventListener('wheel', handleMouseWheel)
+          } catch (error) {
+            console.log(error)
+          }
+
+          return onRemoved?.()
+        }
+
+        this.serialize_widgets = true //需要保存参数
+      }
+    }
+  },
+  async loadedGraphNode (node, app) {
+    if (node.type === 'LoadAndCombinedAudio_') {
+      // await sleep(0)
+      let audiosWidget = node.widgets.filter(w => w.name === 'audios')[0]
+      let audioPreview = node.widgets.filter(w => w.name == 'audio_base64')[0]
+
+      let pre = audioPreview.div.querySelector('.audios_preview')
+      for (const d of audiosWidget.value?.base64 || []) {
+        let im = createInputAudioForBatch(d, audiosWidget)
+        pre.appendChild(im)
+      }
+    }
+  }
+})

web/javascript/checkVersion_mixlab.js

@@ -3,7 +3,7 @@ import { app } from '../../../scripts/app.js'
 const repoOwner = 'shadowcz007' // 替换为仓库的所有者
 const repoName = 'comfyui-mixlab-nodes' // 替换为仓库的名称
 
-const version = 'v0.28.3'
+const version = 'v0.35.1'
 
 fetch(`https://api.github.com/repos/${repoOwner}/${repoName}/releases/latest`)
   .then(response => response.json())

web/javascript/command.js

@@ -0,0 +1,689 @@
+function get_url () {
+  // 如果有缓存记录
+  let hostUrl = localStorage.getItem('_hostUrl') || ''
+  if (hostUrl) {
+    return hostUrl
+  }
+  let api_host = `${window.location.hostname}:${window.location.port}`
+  let api_base = ''
+  let url = `${window.location.protocol}//${api_host}${api_base}`
+  return url
+}
+
+function getFilenameAndCategoryFromUrl (url) {
+  const queryString = url.split('?')[1]
+  if (!queryString) {
+    return {}
+  }
+
+  const params = new URLSearchParams(queryString)
+
+  const filename = params.get('filename')
+    ? decodeURIComponent(params.get('filename'))
+    : null
+  const category = params.get('category')
+    ? decodeURIComponent(params.get('category') || '')
+    : ''
+
+  return { category, filename }
+}
+
+async function get_my_app (category = '', filename = null) {
+  let url = get_url()
+  const res = await fetch(`${url}/mixlab/workflow`, {
+    method: 'POST',
+    mode: 'cors', // 允许跨域请求
+    headers: {
+      'Content-Type': 'application/json'
+    },
+    body: JSON.stringify({
+      task: 'my_app',
+      filename,
+      category
+    })
+  })
+  let result = await res.json()
+  let data = []
+  try {
+    for (const res of result.data) {
+      let { output, app } = res.data
+      if (app.filename)
+        data.push({
+          ...app,
+          data: output,
+          date: res.date
+        })
+    }
+  } catch (error) {}
+
+  return data
+}
+
+async function getAppInit () {
+  const { category, filename } = getFilenameAndCategoryFromUrl(
+    window.location.href
+  )
+  return await get_my_app(category, filename)
+}
+
+function success (isSuccess, btn, text) {
+  isSuccess ? (btn.innerText = 'success') : text
+  setTimeout(() => {
+    btn.innerText = text
+  }, 5000)
+}
+
+async function interrupt () {
+  try {
+    await fetch(`${get_url()}/interrupt`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json'
+      },
+      body: undefined
+    })
+  } catch (error) {
+    console.error(error)
+  }
+  return true
+}
+
+async function getQueue (clientId) {
+  try {
+    const res = await fetch(`${get_url()}/queue`)
+    const data = await res.json()
+    return {
+      // Running action uses a different endpoint for cancelling
+      Running: Array.from(data.queue_running, prompt => {
+        if (prompt[3].client_id === clientId) {
+          let prompt_id = prompt[1]
+          return {
+            prompt_id,
+            remove: () => interrupt()
+          }
+        }
+      }),
+      Pending: data.queue_pending.map(prompt => ({ prompt }))
+    }
+  } catch (error) {
+    console.error(error)
+    return { Running: [], Pending: [] }
+  }
+}
+
+// 请求历史数据
+async function getPromptResult (category) {
+  let url = get_url()
+  try {
+    const response = await fetch(`${url}/mixlab/prompt_result`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json'
+      },
+      body: JSON.stringify({
+        action: 'all'
+      })
+    })
+
+    if (response.ok) {
+      const data = await response.json()
+      console.log('#getPromptResult:', category, data)
+
+      return data.result.filter(r => r.appInfo.category == category)
+      // 处理返回的数据
+    } else {
+      console.log('Error:', response.status)
+      // 处理错误情况
+    }
+  } catch (error) {
+    console.log('Error:', error)
+    // 处理异常情况
+  }
+}
+
+// 新的运行工作流的接口
+function queuePromptNew (filename, category, seed, input, client_id,apps=null) {
+  let url = get_url()
+  // var filename = "Text-to-Image_1.json", category = "";
+
+  // 随机seed
+  //  promptWorkflow = randomSeed(seed, promptWorkflow);
+  let d = { filename, category, seed, input, client_id }
+  if (apps) {
+    d.apps = apps
+  }
+
+  const data = JSON.stringify(d)
+  return new Promise((res, rej) => {
+    fetch(`${url}/mixlab/prompt`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json'
+      },
+      body: data
+    })
+      .then(response => {
+        if (!response.ok) {
+          // Handle HTTP error responses
+          if (response.status === 400) {
+            return response.json().then(errorData => {
+              // Process the error data
+              console.error('Error 400:', errorData)
+              alert(JSON.stringify(errorData, null, 2))
+              res(null)
+            })
+          }
+          throw new Error('Network response was not ok')
+        }
+        return response.json() // Process the response data
+      })
+      .then(data => {
+        // Handle the response data
+        console.log('Success:', data)
+        res(true)
+      })
+      .catch(error => {
+        // Handle fetch errors
+        console.error('Fetch error:', error)
+        res(null)
+      })
+  })
+}
+
+// 保存历史数据
+async function savePromptResult (data) {
+  let url = get_url()
+  try {
+    const response = await fetch(`${url}/mixlab/prompt_result`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json'
+      },
+      body: JSON.stringify({
+        action: 'save',
+        data
+      })
+    })
+
+    if (response.ok) {
+      const res = await response.json()
+      console.log('Response:', res)
+      return res
+      // 处理返回的数据
+    } else {
+      console.log('Error:', response.status)
+      // 处理错误情况
+    }
+  } catch (error) {
+    console.log('Error:', error)
+    // 处理异常情况
+  }
+}
+
+async function uploadImage (blob, fileType = '.png', filename) {
+  const body = new FormData()
+  body.append(
+    'image',
+    new File([blob], (filename || new Date().getTime()) + fileType)
+  )
+
+  const url = get_url()
+
+  const resp = await fetch(`${url}/upload/image`, {
+    method: 'POST',
+    body
+  })
+
+  let data = await resp.json()
+  // console.log(data)
+  let { name, subfolder } = data
+  let src = `${url}/view?filename=${encodeURIComponent(
+    name
+  )}&type=input&subfolder=${subfolder}&rand=${Math.random()}`
+
+  return { url: src, name }
+}
+
+async function uploadMask (arrayBuffer, imgurl) {
+  const body = new FormData()
+  const filename = 'clipspace-mask-' + performance.now() + '.png'
+
+  let original_url = new URL(imgurl)
+
+  const original_ref = { filename: original_url.searchParams.get('filename') }
+
+  let original_subfolder = original_url.searchParams.get('subfolder')
+  if (original_subfolder) original_ref.subfolder = original_subfolder
+
+  let original_type = original_url.searchParams.get('type')
+  if (original_type) original_ref.type = original_type
+
+  body.append('image', arrayBuffer, filename)
+  body.append('original_ref', JSON.stringify(original_ref))
+  body.append('type', 'input')
+  body.append('subfolder', 'clipspace')
+
+  const url = get_url()
+
+  const resp = await fetch(`${url}/upload/mask`, {
+    method: 'POST',
+    body
+  })
+
+  // console.log(resp)
+  let data = await resp.json()
+  let { name, subfolder, type } = data
+  let src = `${url}/view?filename=${encodeURIComponent(
+    name
+  )}&type=${type}&subfolder=${subfolder}&rand=${Math.random()}`
+
+  return { url: src, name: 'clipspace/' + name }
+}
+
+const parseImageToBase64 = url => {
+  return new Promise((res, rej) => {
+    fetch(url)
+      .then(response => response.blob())
+      .then(blob => {
+        const reader = new FileReader()
+        reader.onloadend = () => {
+          const base64data = reader.result
+          res(base64data)
+          // 在这里可以将base64数据用于进一步处理或显示图片
+        }
+        reader.readAsDataURL(blob)
+      })
+      .catch(error => {
+        console.log('发生错误:', error)
+      })
+  })
+}
+
+function createImage (url) {
+  let im = new Image()
+  return new Promise((res, rej) => {
+    im.onload = () => res(im)
+    im.src = url
+  })
+}
+
+function convertImageToBlackBasedOnAlpha (image) {
+  const canvas = document.createElement('canvas')
+  const ctx = canvas.getContext('2d')
+
+  // Draw the image onto the canvas
+  canvas.width = image.width
+  canvas.height = image.height
+  ctx.drawImage(image, 0, 0)
+
+  // Get the image data from the canvas
+  const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height)
+  const pixels = imageData.data
+
+  // Modify the RGB values based on the alpha channel
+  for (let i = 0; i < pixels.length; i += 4) {
+    const alpha = pixels[i + 3]
+    if (alpha !== 0) {
+      // Set non-transparent pixels to black
+      pixels[i] = 0 // Red
+      pixels[i + 1] = 0 // Green
+      pixels[i + 2] = 0 // Blue
+    }
+  }
+
+  // Put the modified image data back onto the canvas
+  ctx.putImageData(imageData, 0, 0)
+
+  // Convert the modified canvas to base64 data URL
+  const base64ImageData = canvas.toDataURL('image/png') // Replace 'png' with your desired image format
+
+  return base64ImageData
+}
+
+const blobToBase64 = blob => {
+  return new Promise((res, rej) => {
+    const reader = new FileReader()
+    reader.onloadend = () => {
+      const base64data = reader.result
+      res(base64data)
+      // 在这里可以将base64数据用于进一步处理或显示图片
+    }
+    reader.readAsDataURL(blob)
+  })
+}
+
+function base64ToBlob (base64) {
+  // 去除base64编码中的前缀
+  const base64WithoutPrefix = base64.replace(/^data:image\/\w+;base64,/, '')
+
+  // 将base64编码转换为字节数组
+  const byteCharacters = atob(base64WithoutPrefix)
+
+  // 创建一个存储字节数组的数组
+  const byteArrays = []
+
+  // 将字节数组放入数组中
+  for (let offset = 0; offset < byteCharacters.length; offset += 1024) {
+    const slice = byteCharacters.slice(offset, offset + 1024)
+
+    const byteNumbers = new Array(slice.length)
+    for (let i = 0; i < slice.length; i++) {
+      byteNumbers[i] = slice.charCodeAt(i)
+    }
+
+    const byteArray = new Uint8Array(byteNumbers)
+    byteArrays.push(byteArray)
+  }
+
+  // 创建blob对象
+  const blob = new Blob(byteArrays, { type: 'image/png' }) // 根据实际情况设置MIME类型
+
+  return blob
+}
+
+async function calculateImageHash (blob) {
+  const buffer = await blob.arrayBuffer()
+  const hashBuffer = await crypto.subtle.digest('SHA-256', buffer)
+  const hashArray = Array.from(new Uint8Array(hashBuffer))
+  const hashHex = hashArray
+    .map(byte => byte.toString(16).padStart(2, '0'))
+    .join('')
+  return hashHex
+}
+
+// 获取 rembg 模型
+async function get_rembg_models () {
+  try {
+    const response = await fetch(`${get_url()}/mixlab/folder_paths`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json'
+      },
+      body: JSON.stringify({
+        type: 'rembg'
+      })
+    })
+
+    const data = await response.json()
+    // console.log(data)
+    return data.names
+  } catch (error) {
+    console.error(error)
+  }
+}
+
+//自动抠图
+async function run_rembg (model, base64) {
+  try {
+    const response = await fetch(`${get_url()}/mixlab/rembg`, {
+      method: 'POST',
+      headers: {
+        'Content-Type': 'application/json'
+      },
+      body: JSON.stringify({
+        model,
+        base64
+      })
+    })
+
+    const data = await response.json()
+    // console.log(data)
+    return data.data
+  } catch (error) {
+    console.error(error)
+  }
+}
+
+function copyHtmlWithImagesToClipboard (data, cb) {
+  // 创建一个临时div元素
+  const tempDiv = document.createElement('div')
+
+  // 将HTML字符串赋值给div的innerHTML属性
+  tempDiv.innerHTML = data
+
+  // 获取div中的所有图像元素
+  const images = tempDiv.getElementsByTagName('img')
+
+  // 遍历图像元素，并将图像数据转换为Base64编码
+  for (let i = 0; i < images.length; i++) {
+    const image = images[i]
+    const canvas = document.createElement('canvas')
+    const context = canvas.getContext('2d')
+
+    // 设置canvas尺寸与图像尺寸相同
+    canvas.width = image.width
+    canvas.height = image.height
+
+    // 在canvas上绘制图像
+    context.drawImage(image, 0, 0)
+
+    // 将canvas转换为Base64编码
+    const imageData = canvas.toDataURL()
+
+    // 将Base64编码替换图像元素的src属性
+    image.src = imageData
+  }
+
+  let richText = tempDiv.innerHTML
+
+  // 创建一个新的Blob对象，并将富文本字符串作为数据传递进去
+  const blob = new Blob([richText], { type: 'text/html' })
+
+  // 创建一个ClipboardItem对象，并将Blob对象添加到其中
+  const clipboardItem = new ClipboardItem({ 'text/html': blob })
+
+  // 使用Clipboard API将内容复制到剪贴板
+  navigator.clipboard
+    .write([clipboardItem])
+    .then(() => {
+      console.log('富文本已成功复制到剪贴板')
+      tempDiv.remove()
+      if (cb) cb(true)
+    })
+    .catch(error => {
+      console.error('复制到剪贴板失败:', error)
+      tempDiv.remove()
+      if (cb) cb(false)
+    })
+}
+
+function copyImagesToClipboard (html, cb) {
+  const tempDiv = document.createElement('div')
+  tempDiv.innerHTML = html
+  const images = tempDiv.querySelectorAll('img')
+  const promises = Array.from(images).map(image => {
+    return new Promise(resolve => {
+      const img = new Image()
+      img.src = image.src
+      img.onload = () => {
+        const canvas = document.createElement('canvas')
+        const context = canvas.getContext('2d')
+        canvas.width = img.width
+        canvas.height = img.height
+        context.drawImage(img, 0, 0)
+        canvas.toBlob(blob => {
+          const clipboardItem = new ClipboardItem({ 'image/png': blob })
+          navigator.clipboard
+            .write([clipboardItem])
+            .then(() => {
+              resolve()
+              tempDiv.remove()
+              if (cb) cb(true)
+            })
+            .catch(error => {
+              reject(error)
+              tempDiv.remove()
+              if (cb) cb(false)
+            })
+        })
+      }
+    })
+  })
+  Promise.all([...promises])
+    .then(() => {
+      console.log('所有图片已成功复制到剪贴板')
+      if (cb) cb(true)
+      tempDiv.remove()
+    })
+    .catch(error => {
+      console.error('复制到剪贴板失败:', error)
+      if (cb) cb(false)
+      tempDiv.remove()
+    })
+}
+
+function copyTextToClipboard (html, cb) {
+  const tempDiv = document.createElement('div')
+  tempDiv.innerHTML = html
+
+  const text = tempDiv.innerText
+  const textData = new ClipboardItem({
+    'text/plain': new Blob([text], { type: 'text/plain' })
+  })
+
+  navigator.clipboard
+    .write([textData])
+    .then(() => {
+      console.log('所有文本已成功复制到剪贴板', text)
+      if (cb) cb(true)
+      tempDiv.remove()
+    })
+    .catch(error => {
+      console.error('复制到剪贴板失败:', error)
+      if (cb) cb(false)
+      tempDiv.remove()
+    })
+}
+
+// ComfyUI\web\extensions\core\dynamicPrompts.js
+// 官方实现修改
+// Allows for simple dynamic prompt replacement
+// Inputs in the format {a|b} will have a random value of a or b chosen when the prompt is queued.
+
+/*
+ * Strips C-style line and block comments from a string
+ */
+function dynamicPrompts (prompt) {
+  prompt = prompt.replace(/\/\*[\s\S]*?\*\/|\/\/.*/g, '')
+  while (
+    prompt.replace('\\{', '').includes('{') &&
+    prompt.replace('\\}', '').includes('}')
+  ) {
+    const startIndex = prompt.replace('\\{', '00').indexOf('{')
+    const endIndex = prompt.replace('\\}', '00').indexOf('}')
+
+    const optionsString = prompt.substring(startIndex + 1, endIndex)
+    const options = optionsString.split('|')
+
+    const randomIndex = Math.floor(Math.random() * options.length)
+    const randomOption = options[randomIndex]
+
+    prompt =
+      prompt.substring(0, startIndex) +
+      randomOption +
+      prompt.substring(endIndex + 1)
+  }
+  return prompt
+}
+
+// 遍历所有组合,语法同 动态提示
+function generateAllCombinations (prompt) {
+  prompt = prompt.replace(/\/\*[\s\S]*?\*\/|\/\/.*/g, '')
+
+  // Helper function to get all combinations
+  function getAllCombinations (parts) {
+    if (parts.length === 0) return ['']
+    const [firstPart, ...restParts] = parts
+    const restCombinations = getAllCombinations(restParts)
+    const allCombinations = []
+
+    firstPart.forEach(option => {
+      restCombinations.forEach(combination => {
+        allCombinations.push(option + combination)
+      })
+    })
+
+    return allCombinations
+  }
+
+  // Split prompt into static parts and dynamic parts
+  let parts = []
+  let startIndex = 0
+
+  while (
+    prompt.replace('\\{', '').includes('{') &&
+    prompt.replace('\\}', '').includes('}')
+  ) {
+    startIndex = prompt.replace('\\{', '00').indexOf('{')
+    const endIndex = prompt.replace('\\}', '00').indexOf('}')
+    const staticPart = prompt.substring(0, startIndex)
+    const optionsString = prompt.substring(startIndex + 1, endIndex)
+    const options = optionsString.split('|')
+
+    parts.push([staticPart])
+    parts.push(options)
+
+    prompt = prompt.substring(endIndex + 1)
+  }
+
+  // Add the remaining static part
+  parts.push([prompt])
+
+  // Get all combinations
+  const combinations = getAllCombinations(parts)
+
+  return combinations
+}
+
+const _textNodes = [
+    'TextInput_',
+    'CLIPTextEncode',
+    'PromptSimplification',
+    'ChinesePrompt_Mix'
+  ],
+  _loraNodes = ['CheckpointLoaderSimple', 'LoraLoader'],
+  _numberNodes = ['FloatSlider', 'IntNumber'],
+  _slideNodes = ['PromptSlide'],
+  _imageNodes = [
+    'LoadImage',
+    'VHS_LoadVideo',
+    'ImagesPrompt_',
+    'LoadImagesToBatch'
+  ],
+  _colorNodes = ['Color'],
+  _audioNodes = ['LoadAndCombinedAudio_']
+
+export default {
+  get_url,
+  get_my_app,
+  getAppInit,
+  getFilenameAndCategoryFromUrl,
+  success,
+  interrupt,
+  getQueue,
+  queuePromptNew,
+  savePromptResult,
+  uploadImage,
+  uploadMask,
+  run_rembg,
+  get_rembg_models,
+  parseImageToBase64,
+  createImage,
+  convertImageToBlackBasedOnAlpha,
+  blobToBase64,
+  base64ToBlob,
+  calculateImageHash,
+  copyHtmlWithImagesToClipboard,
+  copyImagesToClipboard,
+  copyTextToClipboard,
+  dynamicPrompts,
+  generateAllCombinations,
+
+  _textNodes,
+  _loraNodes,
+  _numberNodes,
+  _slideNodes,
+  _imageNodes,
+  _colorNodes,
+  _audioNodes
+}

web/javascript/gpt_mixlab.js

@@ -1,205 +1,5 @@
 import { app } from '../../../scripts/app.js'
-// import { api } from '../../../scripts/api.js'
 import { ComfyWidgets } from '../../../scripts/widgets.js'
-import { $el } from '../../../scripts/ui.js'
-
-async function getConfig () {
-  let api_host = `${window.location.hostname}:${window.location.port}`
-  let api_base = ''
-  let url = `${window.location.protocol}//${api_host}${api_base}`
-
-  const res = await fetch(`${url}/mixlab`, {
-    method: 'POST'
-  })
-  return await res.json()
-}
-
-function get_position_style (ctx, widget_width, y, node_height) {
-  const MARGIN = 4 // the margin around the html element
-
-  /* Create a transform that deals with all the scrolling and zooming */
-  const elRect = ctx.canvas.getBoundingClientRect()
-  const transform = new DOMMatrix()
-    .scaleSelf(
-      elRect.width / ctx.canvas.width,
-      elRect.height / ctx.canvas.height
-    )
-    .multiplySelf(ctx.getTransform())
-    .translateSelf(MARGIN, MARGIN + y)
-
-  return {
-    transformOrigin: '0 0',
-    transform: transform,
-    left: `0`,
-    top: `0`,
-    cursor: 'pointer',
-    position: 'absolute',
-    maxWidth: `${widget_width - MARGIN * 2}px`,
-    // maxHeight: `${node_height - MARGIN * 2}px`, // we're assuming we have the whole height of the node
-    width: `${widget_width - MARGIN * 2}px`,
-    // height: `${node_height * 0.3 - MARGIN * 2}px`,
-    // background: '#EEEEEE',
-    display: 'flex',
-    flexDirection: 'column',
-    // alignItems: 'center',
-    justifyContent: 'space-around'
-  }
-}
-
-const getLocalData = key => {
-  let data = {}
-  try {
-    data = JSON.parse(localStorage.getItem(key)) || {}
-  } catch (error) {
-    return {}
-  }
-  return data
-}
-
-app.registerExtension({
-  name: 'Mixlab.GPT.ChatGPTOpenAI',
-  async getCustomWidgets (app) {
-    return {
-      KEY (node, inputName, inputData, app) {
-        // console.log('##inputData', inputData)
-        const widget = {
-          type: inputData[0], // the type, CHEESE
-          name: inputName, // the name, slice
-          size: [128, 32], // a default size
-          draw (ctx, node, width, y) {},
-          computeSize (...args) {
-            return [128, 32] // a method to compute the current size of the widget
-          },
-          async serializeValue (nodeId, widgetIndex) {
-            let data = getLocalData('_mixlab_api_key')
-            return data[node.id] || 'by Mixlab'
-          }
-        }
-        //  widget.something = something;          // maybe adds stuff to it
-        node.addCustomWidget(widget) // adds it to the node
-        return widget // and returns it.
-      },
-      URL (node, inputName, inputData, app) {
-        // console.log('node', inputName, inputData[0])
-        const widget = {
-          type: inputData[0], // the type, CHEESE
-          name: inputName, // the name, slice
-          size: [128, 32], // a default size
-          draw (ctx, node, width, y) {
-            // a method to draw the widget (ctx is a CanvasRenderingContext2D)
-          },
-          computeSize (...args) {
-            return [128, 32] // a method to compute the current size of the widget
-          },
-          async serializeValue (nodeId, widgetIndex) {
-            let data = getLocalData('_mixlab_api_url')
-            return data[node.id] || 'https://api.openai.com/v1'
-          }
-        }
-        //  widget.something = something;          // maybe adds stuff to it
-        node.addCustomWidget(widget) // adds it to the node
-        return widget // and returns it.
-      }
-    }
-  },
-
-  async beforeRegisterNodeDef (nodeType, nodeData, app) {
-    if (nodeType.comfyClass == 'ChatGPTOpenAI') {
-      const orig_nodeCreated = nodeType.prototype.onNodeCreated
-      nodeType.prototype.onNodeCreated = function () {
-        orig_nodeCreated?.apply(this, arguments)
-
-        const api_key = this.widgets.filter(w => w.name == 'api_key')[0]
-        const api_url = this.widgets.filter(w => w.name == 'api_url')[0]
-
-        console.log('ChatGPTOpenAI nodeData', this.widgets)
-
-        const widget = {
-          type: 'div',
-          name: 'chatgptdiv',
-          draw (ctx, node, widget_width, y, widget_height) {
-            Object.assign(
-              this.div.style,
-              get_position_style(ctx, widget_width, api_key.y, node.size[1])
-            )
-          }
-        }
-
-        widget.div = $el('div', {})
-
-        document.body.appendChild(widget.div)
-
-        const inputDiv = (key, placeholder) => {
-          let div = document.createElement('div')
-          const ip = document.createElement('input')
-          ip.type = placeholder === 'Key' ? 'password' : 'text'
-          ip.className = `${'comfy-multiline-input'} ${placeholder}`
-          div.style = `display: flex;
-          align-items: center; 
-          margin: 6px 8px;
-          margin-top: 0;`
-          ip.placeholder = placeholder
-          ip.value = placeholder
-
-          ip.style = `margin-left: 24px;
-          outline: none;
-          border: none;
-          padding: 4px;width: 100%;`
-          const label = document.createElement('label')
-          label.style = 'font-size: 10px;min-width:32px'
-          label.innerText = placeholder
-          div.appendChild(label)
-          div.appendChild(ip)
-
-          ip.addEventListener('change', () => {
-            let data = getLocalData(key)
-            data[this.id] = ip.value.trim()
-            localStorage.setItem(key, JSON.stringify(data))
-            console.log(this.id, key)
-          })
-          return div
-        }
-
-        let inputKey = inputDiv('_mixlab_api_key', 'Key')
-        let inputUrl = inputDiv('_mixlab_api_url', 'URL')
-
-        widget.div.appendChild(inputKey)
-        widget.div.appendChild(inputUrl)
-
-        this.addCustomWidget(widget)
-
-        const onRemoved = this.onRemoved
-        this.onRemoved = () => {
-          inputUrl.remove()
-          inputKey.remove()
-          widget.div.remove()
-          return onRemoved?.()
-        }
-
-        this.serialize_widgets = true //需要保存参数
-      }
-    }
-  },
-  async loadedGraphNode (node, app) {
-    // Fires every time a node is constructed
-    // You can modify widgets/add handlers/etc here
-
-    if (node.type === 'ChatGPTOpenAI') {
-      let widget = node.widgets.filter(w => w.div)[0]
-
-      let apiKey = getLocalData('_mixlab_api_key'),
-        url = getLocalData('_mixlab_api_url')
-
-      let id = node.id
-
-      // console.log('ChatGPTOpenAI serialize_widgets', this)
-
-      widget.div.querySelector('.Key').value = apiKey[id] || 'by Mixlab'
-      widget.div.querySelector('.URL').value =
-        url[id] || 'https://api.openai.com/v1'
-    }
-  }
-})
 
 app.registerExtension({
   name: 'Mixlab.GPT.ShowTextForGPT',
@@ -209,13 +9,16 @@ app.registerExtension({
         text = text.filter(t => t && t?.trim())
 
         if (this.widgets) {
+          // console.log('#ShowTextForGPT',this.widgets)
           // const pos = this.widgets.findIndex(w => w.name === 'text')
           for (let i = 0; i < this.widgets.length; i++) {
-            if (this.widgets[i].name == 'show_text') this.widgets[i].onRemove?.()
+            if (this.widgets[i].name == 'show_text')
+              this.widgets[i].onRemove?.()
+       
           }
-          this.widgets.length = 1
+          this.widgets.length = 2
         }
-        // console.log('ShowTextForGPT',text)
+
         for (let list of text) {
           if (list) {
             // console.log('#####', list)
@@ -228,6 +31,8 @@ app.registerExtension({
             w.inputEl.readOnly = true
             w.inputEl.style.opacity = 0.6
 
+            // w.inputEl.style.display='none'
+
             try {
               if (typeof list != 'string') {
                 let data = JSON.parse(list)

web/javascript/image_mixlab.js

@@ -675,6 +675,17 @@ const createInputImageForBatch = (base64, widget) => {
   return im
 }
 
+// 添加新图片
+const addBase64ToWidgetForLoadImagesToBatch = (
+  base64,
+  imagesWidget,
+  imagesDiv
+) => {
+  imagesWidget.value.base64.push(base64)
+  let im = createInputImageForBatch(base64, imagesWidget)
+  imagesDiv.appendChild(im)
+}
+
 app.registerExtension({
   name: 'Mixlab.Comfy.LoadImagesToBatch',
   async getCustomWidgets (app) {
@@ -705,7 +716,6 @@ app.registerExtension({
 
   async beforeRegisterNodeDef (nodeType, nodeData, app) {
     if (nodeType.comfyClass == 'LoadImagesToBatch') {
-
       const orig_nodeCreated = nodeType.prototype.onNodeCreated
 
       nodeType.prototype.onNodeCreated = function () {
@@ -751,13 +761,18 @@ app.registerExtension({
             base64 = await loadImageToCanvas(base64)
             // console.log(base64)
             if (!imagesWidget.value) imagesWidget.value = { base64: [] }
-            imagesWidget.value.base64.push(base64)
-            let im = createInputImageForBatch(base64, imagesWidget)
-            imagesDiv.appendChild(im)
+            addBase64ToWidgetForLoadImagesToBatch(
+              base64,
+              imagesWidget,
+              imagesDiv
+            )
           }
           reader.readAsDataURL(file)
         })
 
+        // 如果是复制的，有数据 , 这个不生效，取不到数据， 需要在nodeCreated里获取
+        // console.log('#LoadImagesToBatch', imagesWidget.value?.base64)
+
         const btn = document.createElement('button')
         btn.innerText = 'Upload Image'
 
@@ -829,18 +844,36 @@ app.registerExtension({
       }
     }
   },
+
   async loadedGraphNode (node, app) {
     if (node.type === 'LoadImagesToBatch') {
-      // await sleep(0)
       let imagesWidget = node.widgets.filter(w => w.name === 'images')[0]
       let imagePreview = node.widgets.filter(w => w.name == 'image_base64')[0]
+      // console.log('#LoadImagesToBatch', imagesWidget.value?.base64)
+      let imagesDiv = imagePreview.div.querySelector('.images_preview')
 
-      let pre = imagePreview.div.querySelector('.images_preview')
       for (const d of imagesWidget.value?.base64 || []) {
         let im = createInputImageForBatch(d, imagesWidget)
-        pre.appendChild(im)
+        imagesDiv.appendChild(im)
       }
     }
+  },
+  nodeCreated (node, app) {
+    //数据延迟？？
+    setTimeout(() => {
+      // console.log('#LoadImagesToBatch', node.type)
+      if (node.type === 'LoadImagesToBatch') {
+        let imagesWidget = node.widgets.filter(w => w.name === 'images')[0]
+        let imagePreview = node.widgets.filter(w => w.name == 'image_base64')[0]
+
+        let imagesDiv = imagePreview?.div?.querySelector('.images_preview')
+
+        for (const d of imagesWidget.value?.base64 || []) {
+          let im = createInputImageForBatch(d, imagesWidget)
+          imagesDiv.appendChild(im)
+        }
+      }
+    }, 1000)
   }
 })
 
@@ -868,8 +901,8 @@ app.registerExtension({
       const onNodeCreated = nodeType.prototype.onNodeCreated
       nodeType.prototype.onNodeCreated = function () {
         const r = onNodeCreated
-        ? onNodeCreated.apply(this, arguments)
-        : undefined
+          ? onNodeCreated.apply(this, arguments)
+          : undefined
 
         this.size = [400, this.size[1]]
         console.log('##onNodeCreated', this)
@@ -891,20 +924,15 @@ app.registerExtension({
         this.addCustomWidget(widget)
         this.serialize_widgets = true //需要保存参数
 
-
         const onRemoved = this.onRemoved
         this.onRemoved = () => {
           widget.div.remove()
           return onRemoved?.()
         }
 
-
-       return r
-
+        return r
       }
 
-     
-
       const onExecuted = nodeType.prototype.onExecuted
       nodeType.prototype.onExecuted = function (message) {
         onExecuted?.apply(this, arguments)
@@ -964,7 +992,7 @@ app.registerExtension({
             label: 'After'
           }
         ]
-        this.size=[this.size[0],300]
+        this.size = [this.size[0], 300]
       }
     }
   },
@@ -974,7 +1002,6 @@ app.registerExtension({
       // node.widgets[0].div.id = 'mix_comparingtowframes_' + node.id
       // if (node.widgets_values && node.widgets_values[0]) {
       //   node.widgets[0].div.innerHTML = ''
-
       //   let slider = new juxtapose.JXSlider(
       //     '#mix_comparingtowframes_' + node.id,
       //     node.widgets_values,

web/javascript/main_mixlab.js

@@ -1267,7 +1267,7 @@ app.registerExtension({
         })
 
         widget.PictureInPicture = $el('button', {
-          innerText: 'PictureInPicture',
+          innerText: 'Picture In Picture',
           style: {
             display: 'pictureInPictureEnabled' in document ? 'block' : 'none',
             cursor: 'pointer',

web/javascript/p5.js

@@ -0,0 +1,214 @@
+import { app } from '../../../scripts/app.js'
+import { api } from '../../../scripts/api.js'
+import { $el } from '../../../scripts/ui.js'
+
+function base64ToBlobFromURL (base64URL, contentType) {
+  return fetch(base64URL).then(response => response.blob())
+}
+
+async function uploadImage (blob, fileType = '.svg', filename) {
+  // const blob = await (await fetch(src)).blob();
+  const body = new FormData()
+  body.append(
+    'image',
+    new File([blob], (filename || new Date().getTime()) + fileType)
+  )
+
+  const resp = await api.fetchApi('/upload/image', {
+    method: 'POST',
+    body
+  })
+
+  // console.log(resp)
+  let data = await resp.json()
+  let { name, subfolder } = data
+  // let src = api.apiURL(
+  //   `/view?filename=${encodeURIComponent(
+  //     name
+  //   )}&type=input&subfolder=${subfolder}${app.getPreviewFormatParam()}${app.getRandParam()}`
+  // )
+
+  return data
+}
+// 上传得到url
+async function uploadBase64ToFile (base64) {
+  let bg_blob = await base64ToBlobFromURL(base64)
+  let url = await uploadImage(bg_blob, '.png')
+  return url
+}
+
+function get_position_style (ctx, widget_width, y, node_height) {
+  const MARGIN = 4 // the margin around the html element
+
+  /* Create a transform that deals with all the scrolling and zooming */
+  const elRect = ctx.canvas.getBoundingClientRect()
+  const transform = new DOMMatrix()
+    .scaleSelf(
+      elRect.width / ctx.canvas.width,
+      elRect.height / ctx.canvas.height
+    )
+    .multiplySelf(ctx.getTransform())
+    .translateSelf(MARGIN, MARGIN + y)
+
+  return {
+    transformOrigin: '0 0',
+    transform: transform,
+    left: `0`,
+    top: `0`,
+    cursor: 'pointer',
+    position: 'absolute',
+    maxWidth: `${widget_width - MARGIN * 2}px`,
+    // maxHeight: `${node_height - MARGIN * 2}px`, // we're assuming we have the whole height of the node
+    width: `${widget_width - MARGIN * 2}px`,
+    // height: `${node_height * 0.3 - MARGIN * 2}px`,
+    // background: '#EEEEEE',
+    display: 'flex',
+    flexDirection: 'column',
+    // alignItems: 'center',
+    justifyContent: 'space-around'
+  }
+}
+
+const p5InputNode = {
+  name: 'Mixlab.Comfy.P5Input',
+  async getCustomWidgets (app) {
+    return {
+      IMAGEBASE64 (node, inputName, inputData, app) {
+        const widget = {
+          value: {
+            images: []
+          }, // 不能[x,x,x]
+          type: inputData[0], // the type
+          name: inputName, // the name, slice
+          size: [320, 120], // a default size
+          draw (ctx, node, width, y) {},
+          computeSize (...args) {
+            return [128, 32] // a method to compute the current size of the widget
+          }
+        }
+        node.addCustomWidget(widget)
+        return widget
+      }
+    }
+  },
+
+  async beforeRegisterNodeDef (nodeType, nodeData, app) {
+    if (nodeType.comfyClass == 'P5Input') {
+      console.log('P5Input')
+      const orig_nodeCreated = nodeType.prototype.onNodeCreated
+
+      nodeType.prototype.onNodeCreated = function () {
+        orig_nodeCreated?.apply(this, arguments)
+
+        const widget = {
+          type: 'div',
+          name: 'image_base64',
+          draw (ctx, node, widget_width, y, widget_height) {
+            Object.assign(
+              this.div.style,
+              get_position_style(ctx, widget_width - 24, 44, node.size[1])
+            )
+          },
+          serialize: false
+        }
+
+        widget.div = $el('div', {})
+
+        widget.div.style = `margin:12px;width:400px;height:480px;background:white`
+
+        document.body.appendChild(widget.div)
+
+        this.addCustomWidget(widget)
+
+        // document.addEventListener('wheel', handleMouseWheel)
+
+        const onRemoved = this.onRemoved
+        this.onRemoved = () => {
+          widget.div.remove()
+          // window.removeEventListener('message', ms)
+          return onRemoved?.()
+        }
+
+        // 节点的大小控制
+        this.setSize([480, 560])
+        app.canvas.draw(true, true)
+
+        const onResize = this.onResize
+        this.onResize = () => {
+          // 设置最小尺寸
+          if (
+            Math.max(this.size[0], 480) != this.size[0] &&
+            Math.max(this.size[1], 560) != this.size[1]
+          ) {
+            this.setSize([
+              Math.max(this.size[0], 480),
+              Math.max(this.size[1], 560)
+            ])
+          }
+
+          return onResize?.apply(this, arguments)
+        }
+
+        this.serialize_widgets = true //需要保存参数
+      }
+
+      const onExecuted = nodeType.prototype.onExecuted
+      nodeType.prototype.onExecuted = function (message) {
+        onExecuted?.apply(this, arguments)
+        // console.log('##onExecuted', this, message._info)
+        //   app.graph.getNodeById(8).widgets[1].div.querySelector('iframe').contentWindow.postMessage('Hello from parent', '*');
+      }
+    }
+  },
+
+  async loadedGraphNode (node, app) {
+    if (node.type === 'P5Input') {
+    }
+  },
+  nodeCreated (node, app) {
+    //数据延迟？？
+    setTimeout(() => {
+      let widget = node.widgets?.filter(w => w.name == 'image_base64')[0]
+      let framesWidget = node.widgets?.filter(w => w.name == 'frames')[0]
+      if (node.type === 'P5Input' && widget) {
+        if (framesWidget && !framesWidget.value)
+          framesWidget.value = { images: [] }
+
+        framesWidget.value._seed = Math.random()
+
+        let nodeId = node.id
+        //延迟才能获得this.id
+        widget.div.innerHTML = `<iframe src="extensions/comfyui-mixlab-nodes/p5_export/p5.html?id=${nodeId}" 
+          style="border:0;width:100%;height:100%;"
+         ></iframe>`
+
+        // 监听来自iframe的消息
+        const ms = async event => {
+          const data = event.data
+          if (
+            data.from === 'p5.widget' &&
+            data.status === 'save' &&
+            data.frames &&
+            data.frames.length > 0 &&
+            data.nodeId == nodeId
+          ) {
+            const frames = data.frames
+            console.log(frames.length, nodeId)
+            //workflow会存储到local，会卡死
+            framesWidget.value.images = []
+            for (const f of frames) {
+              let file = await uploadBase64ToFile(f)
+              framesWidget.value.images.push(file)
+            }
+            // framesWidget.value.base64 = frames
+            framesWidget.value._seed = Math.random()
+            node.title = 'P5 Input #' + frames.length
+          }
+        }
+        window.addEventListener('message', ms)
+      }
+    }, 1000)
+  }
+}
+
+app.registerExtension(p5InputNode)

web/javascript/sound-lab.js

@@ -0,0 +1,295 @@
+import { app } from '../../../scripts/app.js'
+import { api } from '../../../scripts/api.js'
+import { ComfyWidgets } from '../../../scripts/widgets.js'
+import { $el } from '../../../scripts/ui.js'
+
+import WaveSurfer from 'https://cdn.jsdelivr.net/npm/wavesurfer.js@7/dist/wavesurfer.esm.js'
+
+function get_position_style (ctx, widget_width, y, node_height) {
+  const MARGIN = 4 // the margin around the html element
+
+  /* Create a transform that deals with all the scrolling and zooming */
+  const elRect = ctx.canvas.getBoundingClientRect()
+  const transform = new DOMMatrix()
+    .scaleSelf(
+      elRect.width / ctx.canvas.width,
+      elRect.height / ctx.canvas.height
+    )
+    .multiplySelf(ctx.getTransform())
+    .translateSelf(MARGIN, MARGIN + y)
+
+  return {
+    transformOrigin: '0 0',
+    transform: transform,
+    left: `0`,
+    top: '0',
+    cursor: 'pointer',
+    position: 'absolute',
+    maxWidth: `${widget_width - MARGIN * 2}px`,
+    // maxHeight: `${node_height - MARGIN * 2}px`, // we're assuming we have the whole height of the node
+    width: `${widget_width - MARGIN * 2}px`,
+    // height: `${node_height * 0.3 - MARGIN * 2}px`,
+    // background: '#EEEEEE',
+    display: 'flex',
+    flexDirection: 'column',
+    // alignItems: 'center',
+    justifyContent: 'space-around'
+  }
+}
+
+//把文件转为url访问
+const parseUrl = data => {
+  let { filename, subfolder, type, prompt } = data
+  return {
+    url: api.apiURL(
+      `/view?filename=${encodeURIComponent(
+        filename
+      )}&type=${type}&subfolder=${subfolder}${app.getPreviewFormatParam()}${app.getRandParam()}`
+    ),
+    prompt
+  }
+}
+
+const createWaveSurfer = (wavesurfer, id,url) => {
+  // Create an instance of WaveSurfer
+  if (wavesurfer) {
+    wavesurfer.destroy()
+  }
+  wavesurfer = WaveSurfer.create({
+    container: '#' + id,
+    waveColor: 'rgb(200, 0, 200)',
+    progressColor: 'rgb(100, 0, 100)',
+    // Set a bar width
+    barWidth: 10,
+    // Optionally, specify the spacing between bars
+    barGap: 2,
+    // And the bar radius
+    barRadius: 6,
+    url
+  })
+
+  wavesurfer._auto = true
+
+  // 监听播放结束事件，重新开始播放以实现循环播放
+  wavesurfer.on('finish', function () {
+    // console.log(wavesurfer)
+    if (wavesurfer._auto) wavesurfer.play()
+  })
+
+  wavesurfer.on('interaction', () => {
+    wavesurfer._auto = false
+    if (!wavesurfer.isPlaying()) wavesurfer.play()
+  })
+
+  // 获取当前播放时间的峰值
+  wavesurfer.on('audioprocess', () => {
+    if (wavesurfer.isPlaying()&&wavesurfer.getDecodedData()) {
+      const channelData = wavesurfer.getDecodedData().getChannelData(0); 
+      const currentTime = wavesurfer.getCurrentTime()
+      // console.log(wavesurfer)
+      const sampleRate = wavesurfer.getDecodedData().sampleRate
+
+      // 定义要分析的时间窗口（例如1秒）
+      const windowSize = 1
+      const startSample = Math.floor(currentTime * sampleRate)
+      const endSample = Math.min(
+        startSample + windowSize * sampleRate,
+        channelData.length
+      )
+
+      let peak = 0
+      for (let i = startSample; i < endSample; i++) {
+        const value = Math.abs(channelData[i])
+        if (value > peak) {
+          peak = value
+        }
+      }
+      // console.log('Current Peak:', peak)
+    }
+  })
+
+  return wavesurfer
+}
+
+//更新gui
+function updateWaveWidgetValue (widgets, id, url, prompt, wavesurfer) {
+  let widget = widgets.filter(w => w.name == 'AudioPlay')[0]
+  // 手动更新widget值
+  widget.value = [url, prompt]
+
+  if (widget.div) {
+    widget.div.querySelector('.wave').id = `AudioPlay_${id}`
+  }
+
+  wavesurfer = createWaveSurfer(wavesurfer, `AudioPlay_${id}`,url)
+
+  wavesurfer.on('ready', duration => {
+    console.log('Audio duration: ' + duration + ' seconds')
+    if (widget.div) {
+      widget.div.setAttribute('data-url', url)
+      widget.div.querySelector('.link').setAttribute('href', url)
+      widget.div.querySelector(
+        '.info'
+      ).innerHTML = `<span style="font-size: 12px;
+      margin: 8px;">${duration.toFixed(
+        2
+      )} seconds</span> <br><span style="font-size: 14px;">${prompt||''}</span> <br>`
+    }
+  })
+  
+  
+  wavesurfer.load(url)
+  // console.log('updateWaveWidgetValue' ,url,wavesurfer)
+  return wavesurfer
+}
+
+app.registerExtension({
+  name: 'SoundLab.AudioPlay',
+  async beforeRegisterNodeDef (nodeType, nodeData, app) {
+    if (nodeType.comfyClass == 'AudioPlay') {
+      let that = this
+      // console.log('that', that)
+
+      const orig_nodeCreated = nodeType.prototype.onNodeCreated
+      nodeType.prototype.onNodeCreated = function () {
+        orig_nodeCreated?.apply(this, arguments)
+
+        const widget = {
+          type: 'div',
+          name: 'AudioPlay',
+          draw (ctx, node, widget_width, y, widget_height) {
+            Object.assign(
+              this.div.style,
+              get_position_style(ctx, widget_width, y, node.size[1])
+            )
+          }
+        }
+
+        // console.log('AudioPlay nodeData', this)
+        widget.div = $el('div', {})
+
+        document.body.appendChild(widget.div)
+
+        // wave
+        const waveDiv = document.createElement('div')
+        waveDiv.className = 'wave'
+        waveDiv.style.minHeight = '172px'
+        widget.div.appendChild(waveDiv)
+
+        //prompt 相关信息展示
+        const infoDiv = document.createElement('div')
+        infoDiv.className = 'info'
+        infoDiv.style.marginBottom = '20px'
+        widget.div.appendChild(infoDiv)
+
+        // 按钮的区域
+        let btns = document.createElement('div')
+        btns.className = 'btns'
+        btns.style = `display: flex;
+        width: 100%;
+        justify-content: space-between;`
+        widget.div.appendChild(btns)
+
+        //play button
+        const playBtn = document.createElement('a')
+        playBtn.innerText = 'Play/Pause'
+
+        playBtn.style = `
+        display: flex;
+        padding: 4px 15px;
+        background-color: var(--comfy-input-bg);
+        border-radius: 8px;
+        border-color: var(--border-color);
+        border-style: solid;
+        color: var(--descrip-text);
+        text-decoration: none;
+        border-radius: 5px;
+        transition: background-color 0.3s ease 0s;
+          `
+
+        playBtn.addEventListener('click', e => {
+          e.preventDefault()
+          if (that[`wavesurfer_${this.id}`]) {
+            that[`wavesurfer_${this.id}`]?.playPause()
+            that[`wavesurfer_${this.id}`]._auto = true
+          }
+        })
+        btns.appendChild(playBtn)
+
+        const urlLink = document.createElement('a')
+        urlLink.className = 'link'
+        urlLink.innerText = 'URL'
+        urlLink.setAttribute('target', '_blank')
+        urlLink.style = `display: flex;
+        padding: 4px 15px;
+        background-color: var(--comfy-input-bg);
+        border-radius: 8px;
+        border-color: var(--border-color);
+        border-style: solid;
+        color: var(--descrip-text);
+        text-decoration: none;
+        border-radius: 5px;
+        transition: background-color 0.3s ease 0s;`
+        // urlLink.style.minHeight = '200px'
+        btns.appendChild(urlLink)
+
+
+        //todo 导出视频 that[`wavesurfer_${this.id}`].renderer.exportImage('image/png',1,'dataURL')
+        // https://github.com/diffusion-studio/ffmpeg-js
+
+
+        this.addCustomWidget(widget)
+
+        const onRemoved = this.onRemoved
+        this.onRemoved = () => {
+          widget.div.remove()
+          return onRemoved?.()
+        }
+
+        this.size = [this.size[0], 280]
+        this.serialize_widgets = true //需保存widget的值
+      }
+
+      const onExecuted = nodeType.prototype.onExecuted
+      nodeType.prototype.onExecuted = function (message) {
+        onExecuted?.apply(this, arguments)
+        const audio = message.audio
+        console.log('#onExecuted', `AudioPlay_${this.id}`, message,audio)
+        try {
+          let { url, prompt } = parseUrl(audio[0])
+
+          that[`wavesurfer_${this.id}`] = updateWaveWidgetValue(
+            this.widgets,
+            this.id,
+            url,
+            prompt,
+            that[`wavesurfer_${this.id}`]
+          )
+
+          that[`wavesurfer_${this.id}`]?.playPause()
+        } catch (error) {
+          console.log(error)
+        }
+      }
+    }
+  },
+  async loadedGraphNode (node, app) {
+    if (node.type === 'AudioPlay') {
+      let widget = node.widgets.filter(w => w.name == 'AudioPlay')[0]
+
+      if (widget.value) {
+        let [url, prompt] = widget.value
+
+        this[`wavesurfer_${node.id}`] = updateWaveWidgetValue(
+          node.widgets,
+          node.id,
+          url,
+          prompt,
+          this[`wavesurfer_${node.id}`]
+        )
+      }
+
+      console.log('#loadedGraphNode', node)
+    }
+  }
+})

web/javascript/td_background.js

@@ -0,0 +1,323 @@
+// touchdesigner的背景效果，把appinfo的输出，选择一张图片作为背景
+
+window._bg_img = null
+
+/**
+ * draws the back canvas (the one containing the background and the connections)
+ * @method drawBackCanvas
+ **/
+LGraphCanvas.prototype.drawBackCanvas = function () {
+  var canvas = this.bgcanvas
+  if (
+    canvas.width != this.canvas.width ||
+    canvas.height != this.canvas.height
+  ) {
+    canvas.width = this.canvas.width
+    canvas.height = this.canvas.height
+  }
+
+  if (!this.bgctx) {
+    this.bgctx = this.bgcanvas.getContext('2d')
+  }
+  var ctx = this.bgctx
+  if (ctx.start) {
+    ctx.start()
+  }
+
+  var viewport = this.viewport || [0, 0, ctx.canvas.width, ctx.canvas.height]
+
+  //clear
+  if (this.clear_background) {
+    ctx.clearRect(viewport[0], viewport[1], viewport[2], viewport[3])
+  }
+
+  //show subgraph stack header
+  if (this._graph_stack && this._graph_stack.length) {
+    ctx.save()
+    var parent_graph = this._graph_stack[this._graph_stack.length - 1]
+    var subgraph_node = this.graph._subgraph_node
+    ctx.strokeStyle = subgraph_node.bgcolor
+    ctx.lineWidth = 10
+    ctx.strokeRect(1, 1, canvas.width - 2, canvas.height - 2)
+    ctx.lineWidth = 1
+    ctx.font = '40px Arial'
+    ctx.textAlign = 'center'
+    ctx.fillStyle = subgraph_node.bgcolor || '#AAA'
+    var title = ''
+    for (var i = 1; i < this._graph_stack.length; ++i) {
+      title += this._graph_stack[i]._subgraph_node.getTitle() + ' >> '
+    }
+    ctx.fillText(title + subgraph_node.getTitle(), canvas.width * 0.5, 40)
+    ctx.restore()
+  }
+
+  var bg_already_painted = false
+  if (this.onRenderBackground) {
+    bg_already_painted = this.onRenderBackground(canvas, ctx)
+  }
+
+  //reset in case of error
+  if (!this.viewport) {
+    ctx.restore()
+    ctx.setTransform(1, 0, 0, 1, 0, 0)
+  }
+  this.visible_links.length = 0
+
+  if (this.graph) {
+    //apply transformations
+    ctx.save()
+    this.ds.toCanvasContext(ctx)
+
+    //render BG
+    if (
+      this.ds.scale < 1 &&
+      !bg_already_painted &&
+      this.clear_background_color
+    ) {
+      ctx.fillStyle = this.clear_background_color
+      ctx.fillRect(
+        this.visible_area[0],
+        this.visible_area[1],
+        this.visible_area[2],
+        this.visible_area[3]
+      )
+    }
+
+    // 主要修改
+    if (this.background_image && this.ds.scale > 0.5 && !bg_already_painted) {
+      if (this.zoom_modify_alpha) {
+        //使得 alpha 越接近0时变化越缓慢。
+        let alpha = (1.0 - 0.5 / this.ds.scale) * this.editor_alpha
+        ctx.globalAlpha = Math.min(Math.max(0, Math.sqrt(alpha)), 1)
+        // console.log((1.0 - 0.5 / this.ds.scale) * this.editor_alpha)
+      } else {
+        ctx.globalAlpha = this.editor_alpha
+      }
+      ctx.imageSmoothingEnabled = ctx.imageSmoothingEnabled = false // ctx.mozImageSmoothingEnabled =
+      if (!this._bg_img || this._bg_img.name != this.background_image) {
+        this._bg_img = new Image()
+        this._bg_img.name = this.background_image
+        this._bg_img.src = this.background_image
+        var that = this
+        this._bg_img.onload = function () {
+          that.draw(true, true)
+        }
+      }
+
+      var pattern = null
+      if (this._pattern == null && this._bg_img.width > 0) {
+        pattern = ctx.createPattern(this._bg_img, 'repeat')
+        this._pattern_img = this._bg_img
+        this._pattern = pattern
+      } else {
+        pattern = this._pattern
+      }
+
+      if (pattern) {
+        ctx.fillStyle = pattern
+        ctx.fillRect(
+          this.visible_area[0],
+          this.visible_area[1],
+          this.visible_area[2],
+          this.visible_area[3]
+        )
+        ctx.fillStyle = 'transparent'
+      }
+
+      ctx.globalAlpha = 1.0
+      ctx.imageSmoothingEnabled = ctx.imageSmoothingEnabled = true //= ctx.mozImageSmoothingEnabled
+    }
+
+    //groups
+    if (this.graph._groups.length && !this.live_mode) {
+      this.drawGroups(canvas, ctx)
+    }
+
+    if (this.onDrawBackground) {
+      this.onDrawBackground(ctx, this.visible_area)
+    }
+    if (this.onBackgroundRender) {
+      //LEGACY
+      console.error(
+        'WARNING! onBackgroundRender deprecated, now is named onDrawBackground '
+      )
+      this.onBackgroundRender = null
+    }
+
+    //DEBUG: show clipping area
+    //ctx.fillStyle = "red";
+    //ctx.fillRect( this.visible_area[0] + 10, this.visible_area[1] + 10, this.visible_area[2] - 20, this.visible_area[3] - 20);
+
+    //bg
+    if (this.render_canvas_border) {
+      ctx.strokeStyle = '#235'
+      ctx.strokeRect(0, 0, canvas.width, canvas.height)
+    }
+
+    if (this.render_connections_shadows) {
+      ctx.shadowColor = '#000'
+      ctx.shadowOffsetX = 0
+      ctx.shadowOffsetY = 0
+      ctx.shadowBlur = 6
+    } else {
+      ctx.shadowColor = 'rgba(0,0,0,0)'
+    }
+
+    //draw connections
+    if (!this.live_mode) {
+      this.drawConnections(ctx)
+    }
+
+    ctx.shadowColor = 'rgba(0,0,0,0)'
+
+    //restore state
+    ctx.restore()
+  }
+
+  if (ctx.finish) {
+    ctx.finish()
+  }
+
+  this.dirty_bgcanvas = false
+  this.dirty_canvas = true //to force to repaint the front canvas with the bgcanvas
+}
+
+function imgToCanvasBase64 (img) {
+  const canvas = document.createElement('canvas')
+  const ctx = canvas.getContext('2d')
+  canvas.width = img.width
+  canvas.height = img.height
+  ctx.drawImage(img, 0, 0)
+  const base64 = canvas.toDataURL('image/png')
+
+  return base64
+}
+
+// 使用示例
+function convertImageToBase64 (img) {
+  //   const img = new Image()
+  //   img.src = 'path/to/your/image.jpg' // 替换为你的图片路径
+  // console.log('convertImageToBase64',img)
+  try {
+    const base64 = imgToCanvasBase64(img)
+    return base64
+  } catch (error) {
+    console.error(error)
+  }
+}
+
+function getInputsAndOutputs () {
+  const outputs =
+    `PreviewImage,SaveImage,TransparentImage,VHS_VideoCombine,VideoCombine_Adv,Image Save,SaveImageAndMetadata_`.split(
+      ','
+    )
+
+  let outputsId = []
+
+  for (let node of app.graph._nodes) {
+    if (outputs.includes(node.type)) {
+      outputsId.push(node.id)
+    }
+  }
+
+  return outputsId
+}
+
+function getRandomElement (arr) {
+  const randomIndex = Math.floor(Math.random() * arr.length)
+  return arr[randomIndex]
+}
+
+async function getBG () {
+  var outputs = []
+
+  for (let id of app.graph
+    .getNodeById(50)
+    .widgets.filter(w => w.name === 'output_ids')[0]
+    .value.split('\n')) {
+    if (getInputsAndOutputs().map(Number).includes(Number(id))) {
+      if (app.graph.getNodeById(id).imgs && app.graph.getNodeById(id).imgs[0]) {
+        let b = convertImageToBase64(app.graph.getNodeById(id).imgs[0])
+        // console.log(b)
+        outputs.push(b)
+      }
+    }
+  }
+
+  var BACKGROUND_IMAGE = getRandomElement(outputs),
+    CLEAR_BACKGROUND_COLOR = 'rgba(0,0,0,0.9)'
+
+  if (!window._bg_img) {
+    window._bg_img = app.canvas._bg_img.src
+  }
+  // let img=new Image();
+  // img.src=BACKGROUND_IMAGE;
+
+  //去掉透明度过度
+  // app.canvas.zoom_modify_alpha=false;
+  //整体透明度
+  app.canvas.editor_alpha = 1.1
+  //   app.canvas._pattern=ctx.createPattern(img, "no-repeat");
+  app.canvas.updateBackground(BACKGROUND_IMAGE, CLEAR_BACKGROUND_COLOR)
+  app.canvas.draw(true, true)
+}
+
+class BgRunner {
+  constructor () {
+    this.intervalId = null
+    this.running = false
+  }
+
+  // 要运行的方法
+  bg () {
+    console.log('方法bg正在运行')
+    getBG()
+  }
+
+  // 启动bg方法每秒运行一次
+  start () {
+    if (!this.running) {
+      this.intervalId = setInterval(() => this.bg(), 1500)
+      this.running = true
+    }
+  }
+
+  // 停止bg方法的运行
+  stop () {
+    if (this.running) {
+      clearInterval(this.intervalId)
+      this.intervalId = null
+      this.running = false
+
+      if (window._bg_img) {
+        var BACKGROUND_IMAGE = window._bg_img,
+          CLEAR_BACKGROUND_COLOR = 'rgba(0,0,0,1)'
+        app.canvas.editor_alpha = 1
+
+        app.canvas.updateBackground(BACKGROUND_IMAGE, CLEAR_BACKGROUND_COLOR)
+        app.canvas.draw(true, true)
+      }
+    }
+  }
+
+  // 切换start和stop
+  toggle () {
+    if (this.running) {
+      this.stop()
+    } else {
+      this.start()
+    }
+  }
+
+  // 获取运行状态
+  isRunning () {
+    return this.running
+  }
+}
+
+// 示例用法
+// const runner = new BgRunner();
+// runner.start();
+// setTimeout(() => runner.stop(), 5000);
+
+export const td_bg = new BgRunner()

web/javascript/ui_mixlab.js

@@ -100,7 +100,7 @@ async function start_llama (model = 'Phi-3-mini-4k-instruct-Q5_K_S.gguf') {
     })
 
     const data = await response.json()
-    if (data.llama_cpp_error) {
+    if (data.llama_cpp_error||!data.port) {
       return
     }
 
@@ -163,17 +163,20 @@ async function createMenu () {
 
     // appsButton.onclick = () =>
     appsButton.onclick = async () => {
-      if (window._mixlab_llamacpp) {
-        //显示运行的模型
-        createModelsModal([
-          window._mixlab_llamacpp.url,
-          window._mixlab_llamacpp.model
-        ])
-      } else {
-        let ms = await get_llamafile_models()
-        ms = ms.filter(m => !m.match('-mmproj-'))
-        if (ms.length > 0) createModelsModal(ms)
-      }
+      // if (window._mixlab_llamacpp&&window._mixlab_llamacpp.model&&window._mixlab_llamacpp.model.length>0) {
+      //   //显示运行的模型
+      //   createModelsModal([
+      //     window._mixlab_llamacpp.url,
+      //     window._mixlab_llamacpp.model
+      //   ])
+      // } else {
+      //   // let ms = await get_llamafile_models()
+      //   // ms = ms.filter(m => !m.match('-mmproj-'))
+      //   // if (ms.length > 0) createModelsModal(ms)
+      // }
+      createModelsModal([
+      
+      ])
     }
     menu.append(appsButton)
   }
@@ -800,11 +803,11 @@ async function fetchReadmeContent (url) {
 
 async function startLLM (model) {
   let res = await start_llama(model)
-  window._mixlab_llamacpp = res
+  window._mixlab_llamacpp = res||{ model:[] }
 
-  localStorage.setItem('_mixlab_llama_select', res.model)
+  localStorage.setItem('_mixlab_llama_select', res?.model||'')
 
-  if (document.body.querySelector('#mixlab_chatbot_by_llamacpp')&&window._mixlab_llamacpp.url) {
+  if (document.body.querySelector('#mixlab_chatbot_by_llamacpp')&&window._mixlab_llamacpp?.url) {
     document.body
       .querySelector('#mixlab_chatbot_by_llamacpp')
       .setAttribute('title', window._mixlab_llamacpp.url)
@@ -932,16 +935,16 @@ function createModelsModal (models) {
   const n_gpu_p = document.createElement('p')
   n_gpu_p.innerText = 'n_gpu_layers'
 
-  const n_gpu_div = document.createElement('div')
-  n_gpu_div.style = `display: flex;
+  const batchPageBtn = document.createElement('div')
+  batchPageBtn.style = `display: flex;
   justify-content: center;
   align-items: center;
   font-size: 12px;`
-  n_gpu_div.appendChild(n_gpu_p)
-  n_gpu_div.appendChild(n_gpu)
+  batchPageBtn.innerHTML=`<a href="${get_url()}/mixlab/app" target="_blank" style="color: var(--input-text);
+  background-color: var(--comfy-input-bg);">App</a>`
 
   const title = document.createElement('p')
-  title.innerText = 'Models'
+  title.innerText = 'Mixlab Nodes'
   title.style = `font-size: 18px;
   margin-right: 8px;
   margin-top: 0;`
@@ -953,9 +956,9 @@ function createModelsModal (models) {
   font-size: 12px;
   flex-direction: column; `
   left_d.appendChild(title)
-  title.appendChild(statusIcon)
-  left_d.appendChild(linkIcon)
-  left_d.appendChild(n_gpu_div)
+  // title.appendChild(statusIcon)
+  // left_d.appendChild(linkIcon)
+  left_d.appendChild(batchPageBtn)
   headTitleElement.appendChild(left_d)
 
   // headTitleElement.appendChild(n_gpu_div)
@@ -1010,26 +1013,26 @@ function createModelsModal (models) {
   var modalContent = document.createElement('div')
   modalContent.classList.add('modal-content')
 
-  var input = document.createElement('textarea')
-  input.className = 'comfy-multiline-input'
-  input.style = `    height: 260px;
+  var inputForSystemPrompt = document.createElement('textarea')
+  inputForSystemPrompt.className = 'comfy-multiline-input'
+  inputForSystemPrompt.style = `    height: 260px;
   width: 480px;
   font-size: 16px;
   padding: 18px;`
-  input.value = localStorage.getItem('_mixlab_system_prompt')
+  inputForSystemPrompt.value = localStorage.getItem('_mixlab_system_prompt')
 
-  input.addEventListener('change', e => {
+  inputForSystemPrompt.addEventListener('change', e => {
     e.stopPropagation()
-    localStorage.setItem('_mixlab_system_prompt', input.value)
+    localStorage.setItem('_mixlab_system_prompt', inputForSystemPrompt.value)
   })
 
-  input.addEventListener('click', e => {
+  inputForSystemPrompt.addEventListener('click', e => {
     e.stopPropagation()
   })
 
-  modalContent.appendChild(input)
+  // modalContent.appendChild(inputForSystemPrompt)
 
-  if (!window._mixlab_llamacpp) {
+  if (!window._mixlab_llamacpp||(window._mixlab_llamacpp?.model?.length==0)) {
     for (const m of models) {
       let d = document.createElement('div')
       d.innerText = `${showTextByLanguage('Run', {
@@ -1040,10 +1043,10 @@ function createModelsModal (models) {
       d.addEventListener('click', async e => {
         e.stopPropagation()
         div.remove()
-        startLLM(m)
+        // startLLM(m)
       })
 
-      modalContent.appendChild(d)
+      // modalContent.appendChild(d)
     }
   }
   modal.appendChild(modalContent)
@@ -1414,7 +1417,7 @@ app.registerExtension({
           .setAttribute('title', res.url)
       })
     }else{
-      startLLM('')
+      // startLLM('')
     }
 
     LGraphCanvas.prototype.helpAboutNode = async function (node) {
@@ -1439,10 +1442,14 @@ app.registerExtension({
 
     LGraphCanvas.prototype.fixTheNode = function (node) {
       let new_node = LiteGraph.createNode(node.comfyClass)
-      new_node.pos = [node.pos[0], node.pos[1]]
-      app.canvas.graph.add(new_node, false)
-      copyNodeValues(node, new_node)
-      app.canvas.graph.remove(node)
+      console.log(node)
+      if(new_node){
+        new_node.pos = [node.pos[0], node.pos[1]]
+        app.canvas.graph.add(new_node, false)
+        copyNodeValues(node, new_node)
+        app.canvas.graph.remove(node)
+      }
+     
     }
 
     smart_init()
@@ -1784,6 +1791,7 @@ app.registerExtension({
         {
           content: 'Help ♾️Mixlab', // with a name
           callback: () => {
+            // console.log('#data',node)
             LGraphCanvas.prototype.helpAboutNode(node)
           } // and the callback
         },

web/javascript/utils_mixlab.js

@@ -1,5 +1,5 @@
 import { app } from '../../../scripts/app.js'
-import { $el } from '../../../scripts/ui.js' 
+import { $el } from '../../../scripts/ui.js'
 
 const getLocalData = key => {
   let data = {}
@@ -122,7 +122,7 @@ app.registerExtension({
       nodeType.prototype.onNodeCreated = function () {
         orig_nodeCreated?.apply(this, arguments)
 
-        // console.log('Color nodeData', this.widgets)
+        console.log('Color nodeData', this.div)
 
         const widget = {
           type: 'div',
@@ -273,19 +273,19 @@ app.registerExtension({
 })
 
 const min_max = node => {
-  if(node.widgets){
+  if (node.widgets) {
     const min_value = node.widgets.filter(w => w.name === 'min_value')[0]
     const max_value = node.widgets.filter(w => w.name === 'max_value')[0]
-  
+
     const number = node.widgets.filter(w => w.name === 'number')[0]
     if (number) {
       number.options.min = min_value.value
       number.options.max = max_value.value
-  
+
       number.value = Math.min(number.options.max, number.value)
       number.value = Math.max(number.options.min, number.value)
     }
-  
+
     if (min_value)
       min_value.callback = e => {
         number.options.min = e
@@ -297,22 +297,18 @@ const min_max = node => {
         number.value = e
       }
   }
-  
 }
 
 app.registerExtension({
   name: 'Mixlab.utils.FloatSlider',
   async beforeRegisterNodeDef (nodeType, nodeData, app) {
-
     if (nodeType.comfyClass == 'FloatSlider') {
-      const orig_nodeCreated = nodeType.prototype.onNodeCreated;
+      const orig_nodeCreated = nodeType.prototype.onNodeCreated
       nodeType.prototype.onNodeCreated = function () {
         orig_nodeCreated?.apply(this, arguments)
         min_max(this)
       }
     }
-
-   
   },
   async loadedGraphNode (node, app) {
     if (node.type === 'FloatSlider') {
@@ -323,7 +319,6 @@ app.registerExtension({
 app.registerExtension({
   name: 'Mixlab.utils.IntNumber',
   async beforeRegisterNodeDef (nodeType, nodeData, app) {
-    
     if (nodeType.comfyClass == 'IntNumber') {
       const orig_nodeCreated = nodeType.prototype.onNodeCreated
       nodeType.prototype.onNodeCreated = function () {
@@ -331,7 +326,6 @@ app.registerExtension({
         min_max(this)
       }
     }
-   
   },
   async loadedGraphNode (node, app) {
     if (node.type === 'IntNumber') {
@@ -340,22 +334,145 @@ app.registerExtension({
   }
 })
 
-
 app.registerExtension({
   name: 'Mixlab.utils.TESTNODE_',
   async beforeRegisterNodeDef (nodeType, nodeData, app) {
-    
     if (nodeType.comfyClass == 'TESTNODE_') {
-    
-      const onExecuted = nodeType.prototype.onExecuted;
-			nodeType.prototype.onExecuted = function (message) {
-				onExecuted?.apply(this, arguments);
-        console.log('##',message)
-				 
-			};
-
-
+      const onExecuted = nodeType.prototype.onExecuted
+      nodeType.prototype.onExecuted = function (message) {
+        onExecuted?.apply(this, arguments)
+        console.log('##', message)
+      }
     }
-   
-  }, 
+  }
+})
+
+app.registerExtension({
+  name: 'Mixlab.utils.KeyInput',
+  init () {},
+  async getCustomWidgets (app) {
+    return {
+      KEY (node, inputName, inputData, app) {
+        // console.log('##node', node)
+        const widget = {
+          type: inputData[0], // the type, CHEESE
+          name: inputName, // the name, slice
+          size: [128, 24], // a default size
+          draw (ctx, node, width, y) {},
+          computeSize (...args) {
+            return [128, 32] // a method to compute the current size of the widget
+          },
+          async serializeValue (nodeId, widgetIndex) {
+            let data = getLocalData('_mixlab_api_key')
+            return data[node.id] || 'by Mixlab'
+          }
+        }
+        //  widget.something = something;          // maybe adds stuff to it
+        node.addCustomWidget(widget) // adds it to the node
+        return widget // and returns it.
+      }
+    }
+  },
+
+  async beforeRegisterNodeDef (nodeType, nodeData, app) {
+    if (nodeType.comfyClass == 'KeyInput') {
+      const orig_nodeCreated = nodeType.prototype.onNodeCreated
+      nodeType.prototype.onNodeCreated = function () {
+        orig_nodeCreated?.apply(this, arguments)
+
+        const widget = {
+          type: 'div',
+          name: 'input_key',
+          draw (ctx, node, widget_width, y, widget_height) {
+            Object.assign(
+              this.div.style,
+              get_position_style(ctx, widget_width, 24, node.size[1])
+            )
+          }
+        }
+
+        widget.div = $el('div', {})
+
+        document.body.appendChild(widget.div)
+
+        const inputDiv = (key, placeholder) => {
+          let div = document.createElement('div')
+          div.style = `
+          display: flex;
+          align-items: center; 
+          margin: 6px 8px;
+          margin-top:0px;
+          height:44px;
+          width:220px; 
+          `
+
+          const ip = document.createElement('input')
+          ip.type = 'password'
+          ip.className = `${'comfy-multiline-input'} ${placeholder}`
+
+          ip.placeholder = placeholder
+          // ip.value = placeholder
+
+          ip.style = `margin-left:8px;
+          outline: none;
+          border: none;
+          padding:12px;
+          width: 100%;
+          `
+
+          div.appendChild(ip)
+
+          ip.addEventListener('change', () => {
+            let data = getLocalData(key)
+            data[this.id] = ip.value.trim()
+            localStorage.setItem(key, JSON.stringify(data))
+          })
+
+          return div
+        }
+
+        let inputKey = inputDiv('_mixlab_api_key', 'Key')
+
+        widget.div.appendChild(inputKey)
+
+        this.addCustomWidget(widget)
+
+        const onRemoved = this.onRemoved
+        this.onRemoved = () => {
+          inputKey.remove()
+          widget.div.remove()
+          return onRemoved?.()
+        }
+
+        this.serialize_widgets = true //需要保存参数
+      }
+    }
+  },
+  async loadedGraphNode (node, app) {
+    if (node.type === 'KeyInput') {
+      let widget = node.widgets.filter(w => w.div)[0]
+
+      let apiKey = getLocalData('_mixlab_api_key')
+
+      let id = node.id
+      if (widget.div.querySelector('.Key'))
+        widget.div.querySelector('.Key').value = apiKey[id] || 'by Mixlab'
+    }
+  },
+  nodeCreated (node, app) {
+    //数据延迟？？
+    setTimeout(() => {
+      // console.log('#LoadImagesToBatch', node.type)
+      if (node.type === 'KeyInput') {
+        let widget = node.widgets.filter(w => w.div)[0]
+
+        let apiKey = getLocalData('_mixlab_api_key')
+
+        let id = node.id
+
+        if (widget.div.querySelector('.Key'))
+          widget.div.querySelector('.Key').value = apiKey[id] || 'by Mixlab'
+      }
+    }, 1000)
+  }
 })

web/javascript/video_mixlab.js

@@ -6,8 +6,6 @@ import { $el } from '../../../scripts/ui.js'
 
 // The code is based on ComfyUI-VideoHelperSuite modification.
 
-
-
 function injectCSS (css) {
   // 检查页面中是否已经存在具有相同内容的style标签
   const existingStyle = document.querySelector('style')
@@ -240,15 +238,7 @@ app.registerExtension({
   }
 })
 
-
-function offsetDOMWidget(
-  widget,
-  ctx,
-  node,
-  widgetWidth,
-  widgetY,
-  height
-) {
+function offsetDOMWidget (widget, ctx, node, widgetWidth, widgetY, height) {
   const margin = 10
   const elRect = ctx.canvas.getBoundingClientRect()
   const transform = new DOMMatrix()
@@ -270,18 +260,18 @@ function offsetDOMWidget(
     position: 'absolute',
     background: !node.color ? '' : node.color,
     color: !node.color ? '' : 'white',
-    zIndex: 5, //app.graph._nodes.indexOf(node),
+    zIndex: 5 //app.graph._nodes.indexOf(node),
   })
 }
 
-export const hasWidgets = (node) => {
+export const hasWidgets = node => {
   if (!node.widgets || !node.widgets?.[Symbol.iterator]) {
     return false
   }
   return true
 }
 
-export const cleanupNode = (node) => {
+export const cleanupNode = node => {
   if (!hasWidgets(node)) {
     return
   }
@@ -298,43 +288,43 @@ export const cleanupNode = (node) => {
   }
 }
 
-const CreatePreviewElement = (name, val, format) => {
-  const [type] = format.split('/')  
+const createPreviewElement = (name, val, format) => {
+  const [type] = format.split('/')
   const w = {
-      name,
-      type,
-      value: val,
-      draw: function (ctx, node, widgetWidth, widgetY, height) {
-        const [cw, ch] = this.computeSize(widgetWidth)
-        offsetDOMWidget(this, ctx, node, widgetWidth, widgetY, ch)
-      },
-      computeSize: function (_) {
-        const ratio = this.inputRatio || 1
-        const width = Math.max(220, this.parent.size[0])
-        return [width, (width / ratio + 10)]
-      },
-      onRemoved: function () {
-        if (this.inputEl) {
-          this.inputEl.remove()
-        }
-      },
+    name,
+    type,
+    value: val,
+    draw: function (ctx, node, widgetWidth, widgetY, height) {
+      const [cw, ch] = this.computeSize(widgetWidth)
+      offsetDOMWidget(this, ctx, node, widgetWidth, widgetY, ch)
+    },
+    computeSize: function (_) {
+      const ratio = this.inputRatio || 1
+      const width = Math.max(220, this.parent.size[0])
+      return [width, width / ratio + 10]
+    },
+    onRemoved: function () {
+      if (this.inputEl) {
+        this.inputEl.remove()
+      }
     }
-
-    w.inputEl = document.createElement(type === 'video' ? 'video' : 'img')
-    w.inputEl.src = w.value
-    if (type === 'video') {
-      w.inputEl.setAttribute('type', 'video/webm');
-      w.inputEl.autoplay = true
-      w.inputEl.loop = true
-      w.inputEl.controls = false;
-    }
-    w.inputEl.onload = function () {
-      w.inputRatio = w.inputEl.naturalWidth / w.inputEl.naturalHeight
-    }
-    document.body.appendChild(w.inputEl)
-    return w
   }
 
+  w.inputEl = document.createElement(type === 'video' ? 'video' : 'img')
+  w.inputEl.src = w.value
+
+  if (type === 'video' || format.match('.mp4')) {
+    w.inputEl.setAttribute('type', 'video/webm')
+    w.inputEl.autoplay = true
+    w.inputEl.loop = true
+    w.inputEl.controls = true
+  }
+  w.inputEl.onload = function () {
+    w.inputRatio = w.inputEl.naturalWidth / w.inputEl.naturalHeight
+  }
+  document.body.appendChild(w.inputEl)
+  return w
+}
 
 app.registerExtension({
   name: 'Mixlab.Video.ImageListReplace',
@@ -469,12 +459,17 @@ app.registerExtension({
       }
     }
 
-    if (nodeData?.name == 'VideoCombine_Adv') {
+    if (
+      nodeData?.name == 'VideoCombine_Adv' ||
+      nodeData?.name == 'CombineAudioVideo'
+    ) {
       const onExecuted = nodeType.prototype.onExecuted
       nodeType.prototype.onExecuted = function (message) {
         const prefix = 'vhs_gif_preview_'
         const r = onExecuted ? onExecuted.apply(this, message) : undefined
 
+        if(!this.widgets) this.widgets=[]
+
         if (this.widgets) {
           const pos = this.widgets.findIndex(w => w.name === `${prefix}_0`)
           if (pos !== -1) {
@@ -489,12 +484,13 @@ app.registerExtension({
                 '/view?' + new URLSearchParams(params).toString()
               )
               const w = this.addCustomWidget(
-                CreatePreviewElement(
+                createPreviewElement(
                   `${prefix}_${i}`,
                   previewUrl,
                   params.format || 'image/gif'
                 )
               )
+              console.log(w)
               w.parent = this
             })
           }

web/live.html

@@ -1,525 +0,0 @@
-<!DOCTYPE html>
-<!-- 
-  VoiceStreamAI Client Interface
-  Real-time audio transcription using self-hosted Whisper and WebSocket
-
-  Contributor:
-  - Alessandro Saccoia - alessandro.saccoia@gmail.com
--->
-<html lang="en">
-
-<head>
-    <meta charset="UTF-8">
-    <title>Audio Stream to WebSocket Server</title>
-    <style>
-        body {
-            font-family: Arial, sans-serif;
-            margin: 0;
-            padding: 0;
-            background: #f4f4f4;
-            text-align: center;
-        }
-
-        h1 {
-            color: #333;
-        }
-
-        .controls {
-            margin: 20px auto;
-            padding: 10px;
-            width: 80%;
-            display: flex;
-            justify-content: space-around;
-            align-items: center;
-        }
-
-        .control-group {
-            display: flex;
-            flex-direction: column;
-            align-items: center;
-        }
-
-        .controls input,
-        .controls button,
-        .controls select {
-            padding: 8px;
-            margin: 5px;
-            border: 1px solid #ddd;
-            border-radius: 5px;
-            font-size: 0.9em;
-        }
-
-        #transcription {
-            margin: 20px auto;
-            border: 1px solid #ddd;
-            padding: 10px;
-            width: 80%;
-            height: 150px;
-            overflow-y: auto;
-            background: white;
-        }
-
-        .label {
-            font-size: 0.9em;
-            color: #555;
-            margin-bottom: 5px;
-        }
-
-        button {
-            cursor: pointer;
-        }
-
-        .buffering-strategy-panel {
-            margin-top: 10px;
-        }
-
-        /* ... existing styles ... */
-        .hidden {
-            display: none;
-        }
-    </style>
-
-    <style>
-        body {
-            margin: 0;
-            padding: 0;
-            /* background-color: #333;
-            color:white */
-        }
-
-        #mic_container {
-            display: flex;
-            width: 100%;
-            align-items: center;
-            justify-content: center;
-        }
-
-        #mic {
-            border: 1px solid #ddd;
-            border-radius: 4px;
-            margin-top: 1rem;
-
-            width: 300px
-        }
-
-        #asistant {
-            width: 300px;
-            padding: 12px;
-            font-size: 12px;
-            overflow-y: scroll;
-            height: 300px;
-        }
-    </style>
-</head>
-
-<body>
-
-    <div id="mic_container">
-        <div id="mic"></div>
-        <div id="asistant"></div>
-    </div>
-
-    <h1>VAD + Whisper + </h1>
-
-    <button id="init_server">Server</button>
-
-    <div class="controls">
-        <div class="control-group">
-            <label class="label" for="websocketAddress">WebSocket Address:</label>
-            <input type="text" id="websocketAddress" value="ws://localhost:8725">
-        </div>
-        <div class="control-group">
-            <label class="label" for="bufferingStrategySelect" onchange="toggleBufferingStrategyPanel()">Buffering
-                Strategy:</label>
-            <select id="bufferingStrategySelect">
-                <option value="silence_at_end_of_chunk" selected>Silence at End of Chunk</option>
-            </select>
-        </div>
-        <div class="silence_at_end_of_chunk_options_panel">
-            <div class="control-group">
-                <label class="label" for="chunk_length_seconds">Chunk Length (s):</label>
-                <input type="number" id="chunk_length_seconds" value="3" min="1">
-            </div>
-            <div class="control-group">
-                <label class="label" for="chunk_offset_seconds">Silence at the End of Chunk (s):</label>
-                <input type="number" id="chunk_offset_seconds" value="0.1" min="0">
-            </div>
-        </div>
-        <div class="control-group">
-            <label class="label" for="languageSelect">Language:</label>
-            <select id="languageSelect">
-                <option value="multilingual">Multilingual</option>
-                <option value="english">English</option>
-                <option value="italian">Italian</option>
-                <option value="spanish">Spanish</option>
-                <option value="french">French</option>
-                <option value="german">German</option>
-                <option value="chinese">Chinese</option>
-                <option value="arabic">Arabic</option>
-                <option value="portuguese">Portuguese</option>
-                <option value="russian">Russian</option>
-                <option value="japanese">Japanese</option>
-                <option value="dutch">Dutch</option>
-                <option value="korean">Korean</option>
-                <option value="hindi">Hindi</option>
-                <option value="turkish">Turkish</option>
-                <option value="swedish">Swedish</option>
-                <option value="norwegian">Norwegian</option>
-                <option value="danish">Danish</option>
-                <option value="polish">Polish</option>
-                <option value="finnish">Finnish</option>
-                <option value="thai">Thai</option>
-                <option value="czech">Czech</option>
-                <option value="hungarian">Hungarian</option>
-                <option value="greek">Greek</option>
-            </select>
-        </div>
-        <button id="connectButton">Connect</button>
-    </div>
-    <button id="startButton" disabled>Start Streaming</button>
-    <button id="stopButton" disabled>Stop Streaming</button>
-    <div id="transcription"></div>
-    <br />
-    <div>WebSocket: <span id="webSocketStatus">Not Connected</span></div>
-    <div>Detected Language: <span id="detected_language">Undefined</span></div>
-    <div>Last Processing Time: <span id="processing_time">Undefined</span></div>
-
-    <script type="module">
-
-        // Record plugin
-
-        import WaveSurfer from 'https://cdn.jsdelivr.net/npm/wavesurfer.js@7/dist/wavesurfer.esm.js'
-        import RecordPlugin from 'https://cdn.jsdelivr.net/npm/wavesurfer.js/dist/plugins/record.esm.js'
-
-        // 可视化
-        let wavesurfer, record;
-
-        // ws服务
-        let websocket;
-        let context;
-        let processor;
-        let globalStream;
-
-        const websocket_uri = 'ws://localhost:8765';
-        const bufferSize = 4096;
-        let isRecording = false;
-        let chunk_length_seconds, chunk_offset_seconds, language;
-
-        // Record button
-        const startButton = document.getElementById('startButton'),
-            stopButton = document.getElementById('stopButton'),
-            connectButton = document.getElementById('connectButton'),
-            initServerButton = document.getElementById('init_server')
-
-        const createWaveSurfer = () => {
-            // Create an instance of WaveSurfer
-            if (wavesurfer) {
-                wavesurfer.destroy()
-            }
-            wavesurfer = WaveSurfer.create({
-                container: '#mic',
-                waveColor: 'rgb(200, 0, 200)',
-                progressColor: 'rgb(100, 0, 100)',
-                renderFunction: (channels, ctx) => {
-                    const { width, height } = ctx.canvas
-                    // console.log(width, height)
-                    const scale = channels[0].length / width
-                    const step = 20
-
-                    ctx.translate(0, height / 2)
-                    ctx.strokeStyle = ctx.fillStyle
-                    ctx.beginPath()
-
-                    for (let i = 0; i < width; i += step * 2) {
-                        const index = Math.floor(i * scale)
-                        const value = Math.abs(channels[0][index])
-                        let x = i
-                        let y = value * height * 1.2
-
-                        ctx.moveTo(x, 0)
-                        ctx.lineTo(x, y)
-                        ctx.arc(x + step / 2, y, step / 2, Math.PI, 0, true)
-                        ctx.lineTo(x + step, 0)
-
-                        x = x + step
-                        y = -y
-                        ctx.moveTo(x, 0)
-                        ctx.lineTo(x, y)
-                        ctx.arc(x + step / 2, y, step / 2, Math.PI, 0, false)
-                        ctx.lineTo(x + step, 0)
-                    }
-
-                    ctx.stroke()
-                    ctx.closePath()
-                },
-                // // Set a bar width
-                // barWidth: 20,
-                // // Optionally, specify the spacing between bars
-                // barGap: 4,
-                // // And the bar radius
-                // barRadius: 2,
-            })
-
-            // Initialize the Record plugin
-            record = wavesurfer.registerPlugin(RecordPlugin.create({ scrollingWaveform: false, renderRecordedAudio: false }))
-            // Render recorded audio
-            // recButton.textContent = 'Record'
-
-        }
-
-        startButton.addEventListener('click', e => {
-            record.startRecording()
-
-            if (isRecording) return;
-            isRecording = true;
-
-            const AudioContext = window.AudioContext || window.webkitAudioContext;
-            context = new AudioContext();
-
-            navigator.mediaDevices.getUserMedia({ audio: true }).then(stream => {
-                globalStream = stream;
-                const input = context.createMediaStreamSource(stream);
-                processor = context.createScriptProcessor(bufferSize, 1, 1);
-                processor.onaudioprocess = e => processAudio(e);
-                input.connect(processor);
-                processor.connect(context.destination);
-
-                sendAudioConfig();
-            }).catch(error => console.error('Error accessing microphone', error));
-
-            // Disable start button and enable stop button
-            startButton.disabled = true;
-            stopButton.disabled = false;
-
-        });
-
-        stopButton.addEventListener('click', e => {
-            if (!isRecording) return;
-            isRecording = false;
-
-            if (globalStream) {
-                globalStream.getTracks().forEach(track => track.stop());
-            }
-            if (processor) {
-                processor.disconnect();
-                processor = null;
-            }
-            if (context) {
-                context.close().then(() => context = null);
-            }
-            startButton.disabled = false;
-            stopButton.disabled = true;
-
-            // 可视化部分
-            if (record.isRecording() || record.isPaused()) {
-                record.stopRecording()
-                return
-            }
-        })
-
-        connectButton.addEventListener('click', e => {
-            initWebSocket()
-        })
-
-        initServerButton.addEventListener('click', async e => {
-            const response = await fetch('/mixlab/start_live', {
-                method: 'POST',
-                headers: {
-                    'Content-Type': 'application/json'
-                },
-                body: JSON.stringify({
-                    port: 8323,
-                    model: ''
-                })
-            })
-            console.log(await response.json())
-
-        })
-
-        // Initialize on page load
-        window.onload = () => {
-            initWebSocket()
-            createWaveSurfer();
-        };
-
-        function initWebSocket() {
-            const websocketAddress = document.getElementById('websocketAddress').value;
-            chunk_length_seconds = document.getElementById('chunk_length_seconds').value;
-            chunk_offset_seconds = document.getElementById('chunk_offset_seconds').value;
-            const selectedLanguage = document.getElementById('languageSelect').value;
-            language = selectedLanguage !== 'multilingual' ? selectedLanguage : null;
-
-            if (!websocketAddress) {
-                console.log("WebSocket address is required.");
-                return;
-            }
-
-            if (websocket) websocket.close()
-
-            websocket = new WebSocket(websocketAddress);
-            websocket.onopen = () => {
-                console.log("WebSocket connection established");
-                document.getElementById("webSocketStatus").textContent = 'Connected';
-                startButton.disabled = false;
-            };
-            websocket.onclose = event => {
-                console.log("WebSocket connection closed", event);
-                document.getElementById("webSocketStatus").textContent = 'Not Connected';
-                stopButton.click()
-                startButton.disabled = true;
-                stopButton.disabled = true;
-
-                // setTimeout(()=>initWebSocket(),1000)
-            };
-            websocket.onmessage = event => {
-                console.log("Message from server:", event.data);
-                const transcript_data = JSON.parse(event.data);
-
-                if (transcript_data.status === 'chat_start') {
-                    updateTranscription(transcript_data);
-                    stopButton.click();
-                } else if (transcript_data.status === 'chat_end') {
-                    let asistant = decodeURIComponent(transcript_data.asistant)
-                    document.getElementById('asistant').innerText = asistant;
-                    startButton.click();
-                }
-
-            };
-            websocket.onerror = () => {
-                // setTimeout(()=>initWebSocket(),1000)
-            }
-        }
-
-        function updateTranscription(transcript_data) {
-            const transcriptionDiv = document.getElementById('transcription');
-            const languageDiv = document.getElementById('detected_language');
-
-            if (transcript_data['words'] && transcript_data['words'].length > 0) {
-                // Append words with color based on their probability
-                transcript_data['words'].forEach(wordData => {
-                    const span = document.createElement('span');
-                    const probability = wordData['probability'];
-                    span.textContent = wordData['word'] + ' ';
-
-                    // Set the color based on the probability
-                    if (probability > 0.9) {
-                        span.style.color = 'green';
-                    } else if (probability > 0.6) {
-                        span.style.color = 'orange';
-                    } else {
-                        span.style.color = 'red';
-                    }
-
-                    transcriptionDiv.appendChild(span);
-                });
-
-                // Add a new line at the end
-                transcriptionDiv.appendChild(document.createElement('br'));
-            } else {
-                // Fallback to plain text
-                transcriptionDiv.textContent += transcript_data['text'] + '\n';
-            }
-
-            // Update the language information
-            if (transcript_data['language'] && transcript_data['language_probability']) {
-                languageDiv.textContent = transcript_data['language'] + ' (' + transcript_data['language_probability'].toFixed(2) + ')';
-            }
-
-            // Update the processing time, if available
-            const processingTimeDiv = document.getElementById('processing_time');
-            if (transcript_data['processing_time']) {
-                processingTimeDiv.textContent = 'Processing time: ' + transcript_data['processing_time'].toFixed(2) + ' seconds';
-            }
-        }
-
-
-
-
-        function sendAudioConfig() {
-            let selectedStrategy = document.getElementById('bufferingStrategySelect').value;
-            let processingArgs = {};
-
-            if (selectedStrategy === 'silence_at_end_of_chunk') {
-                processingArgs = {
-                    chunk_length_seconds: parseFloat(document.getElementById('chunk_length_seconds').value),
-                    chunk_offset_seconds: parseFloat(document.getElementById('chunk_offset_seconds').value)
-                };
-            }
-
-            const audioConfig = {
-                type: 'config',
-                data: {
-                    sampleRate: context.sampleRate,
-                    bufferSize: bufferSize,
-                    channels: 1, // Assuming mono channel
-                    language: language,
-                    processing_strategy: selectedStrategy,
-                    processing_args: processingArgs
-                }
-            };
-
-            websocket.send(JSON.stringify(audioConfig));
-        }
-
-        function downsampleBuffer(buffer, inputSampleRate, outputSampleRate) {
-            if (inputSampleRate === outputSampleRate) {
-                return buffer;
-            }
-            var sampleRateRatio = inputSampleRate / outputSampleRate;
-            var newLength = Math.round(buffer.length / sampleRateRatio);
-            var result = new Float32Array(newLength);
-            var offsetResult = 0;
-            var offsetBuffer = 0;
-            while (offsetResult < result.length) {
-                var nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio);
-                var accum = 0, count = 0;
-                for (var i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++) {
-                    accum += buffer[i];
-                    count++;
-                }
-                result[offsetResult] = accum / count;
-                offsetResult++;
-                offsetBuffer = nextOffsetBuffer;
-            }
-            return result;
-        }
-
-        function processAudio(e) {
-            const inputSampleRate = context.sampleRate;
-            const outputSampleRate = 16000; // Target sample rate
-
-            const left = e.inputBuffer.getChannelData(0);
-            const downsampledBuffer = downsampleBuffer(left, inputSampleRate, outputSampleRate);
-            const audioData = convertFloat32ToInt16(downsampledBuffer);
-
-            if (websocket && websocket.readyState === WebSocket.OPEN) {
-                websocket.send(audioData);
-            }
-        }
-
-        function convertFloat32ToInt16(buffer) {
-            let l = buffer.length;
-            const buf = new Int16Array(l);
-            while (l--) {
-                buf[l] = Math.min(1, buffer[l]) * 0x7FFF;
-            }
-            return buf.buffer;
-        }
-
-
-        function toggleBufferingStrategyPanel() {
-            var selectedStrategy = document.getElementById('bufferingStrategySelect').value;
-            if (selectedStrategy === 'silence_at_end_of_chunk') {
-                var panel = document.getElementById('silence_at_end_of_chunk_options_panel');
-                panel.classList.remove('hidden');
-            } else {
-                var panel = document.getElementById('silence_at_end_of_chunk_options_panel');
-                panel.classList.add('hidden');
-            }
-        }
-
-
-    </script>
-</body>
-
-</html>

web/p5_export/p5-widget/p5-widget.html

@@ -0,0 +1,6 @@
+<!DOCTYPE html>
+<meta charset="utf-8">
+<link href="https://fonts.googleapis.com/css?family=Montserrat" rel="stylesheet">
+<title>p5.js-widget</title>
+<div id="app-holder"></div>
+<script src="./main.bundle.js"></script>

web/p5_export/p5-widget/p5-widget.js

@@ -0,0 +1,240 @@
+/******/ (function(modules) { // webpackBootstrap
+/******/ 	// The module cache
+/******/ 	var installedModules = {};
+/******/
+/******/ 	// The require function
+/******/ 	function __webpack_require__(moduleId) {
+/******/
+/******/ 		// Check if module is in cache
+/******/ 		if(installedModules[moduleId])
+/******/ 			return installedModules[moduleId].exports;
+/******/
+/******/ 		// Create a new module (and put it into the cache)
+/******/ 		var module = installedModules[moduleId] = {
+/******/ 			exports: {},
+/******/ 			id: moduleId,
+/******/ 			loaded: false
+/******/ 		};
+/******/
+/******/ 		// Execute the module function
+/******/ 		modules[moduleId].call(module.exports, module, module.exports, __webpack_require__);
+/******/
+/******/ 		// Flag the module as loaded
+/******/ 		module.loaded = true;
+/******/
+/******/ 		// Return the exports of the module
+/******/ 		return module.exports;
+/******/ 	}
+/******/
+/******/
+/******/ 	// expose the modules object (__webpack_modules__)
+/******/ 	__webpack_require__.m = modules;
+/******/
+/******/ 	// expose the module cache
+/******/ 	__webpack_require__.c = installedModules;
+/******/
+/******/ 	// __webpack_public_path__
+/******/ 	__webpack_require__.p = "";
+/******/
+/******/ 	// Load entry module and return exports
+/******/ 	return __webpack_require__(0);
+/******/ })
+/************************************************************************/
+/******/ ([
+/* 0 */
+/***/ (function(module, exports, __webpack_require__) {
+
+	"use strict";
+	var defaults = __webpack_require__(1);
+	var MY_FILENAME = 'p5-widget.js';
+	var IFRAME_FILENAME = 'p5-widget.html';
+	var IFRAME_STYLE = [
+	    'width: 100%',
+	    'background-color: white',
+	    'border: 1px solid #ec245e',
+	    'box-sizing: border-box'
+	];
+	var AVOID_MIXED_CONTENT_WARNINGS = true;
+	var myScriptEl = getMyScriptEl();
+	var myBaseURL = getMyBaseURL(myScriptEl ? myScriptEl.src : "");
+	var autoload = myScriptEl ? !myScriptEl.hasAttribute('data-manual') : false;
+	var nextId = 1;
+	function getMyBaseURL(url) {
+	    var baseURL = url.slice(0, -MY_FILENAME.length);
+	    if (AVOID_MIXED_CONTENT_WARNINGS) {
+	        if (window.location.protocol === 'http:' && /^https:/.test(baseURL)) {
+	            // Our script was loaded over HTTPS, but the embedding page is
+	            // using HTTP. This is likely to result in mixed content warnings
+	            // if e.g. the widget's sketch wants to load resources relative to
+	            // the embedding page's URL, so let's just embed the widget over
+	            // HTTP instead of HTTPS.
+	            baseURL = baseURL.replace('https:', 'http:');
+	        }
+	    }
+	    return baseURL;
+	}
+	function getMyScriptEl() {
+	    return (document.currentScript ||
+	        document.querySelectorAll("script[src$='" + MY_FILENAME + "']")[0]);
+	}
+	// http://stackoverflow.com/a/7557433/2422398
+	function isElementInViewport(el) {
+	    var rect = el.getBoundingClientRect();
+	    return (rect.bottom >= 0 &&
+	        rect.right >= 0 &&
+	        rect.top <= (window.innerHeight ||
+	            document.documentElement.clientHeight) &&
+	        rect.left <= (window.innerWidth ||
+	            document.documentElement.clientWidth));
+	}
+	function getDataHeight(el) {
+	    var height = parseInt(el.getAttribute('data-height'));
+	    if (isNaN(height))
+	        height = defaults.HEIGHT;
+	    return height;
+	}
+	function absoluteURL(url) {
+	    var a = document.createElement('a');
+	    a.setAttribute('href', url);
+	    return a.href;
+	}
+	function getSketch(url, cb) {
+	    var error = function (msg) {
+	        var lines = ['// p5.js-widget failed to retrieve ' + url + '.'];
+	        if (msg && typeof (msg) == 'string') {
+	            lines.push('// ' + msg);
+	        }
+	        cb(lines.join('\n'));
+	    };
+	    var req = new XMLHttpRequest();
+	    req.open('GET', url);
+	    req.onload = function () {
+	        if (req.status == 200) {
+	            cb(req.responseText);
+	        }
+	        else {
+	            error('Server returned HTTP ' + req.status + '.');
+	        }
+	    };
+	    req.onerror = error;
+	    req.send(null);
+	}
+	function replaceScriptWithWidget(el) {
+	    var iframe = document.createElement('iframe');
+	    var height = getDataHeight(el);
+	    var previewWidth = parseInt(el.getAttribute('data-preview-width'));
+	    var baseSketchURL = absoluteURL(el.getAttribute('data-base-url'));
+	    var p5version = el.getAttribute('data-p5-version');
+	    var maxRunTime = parseInt(el.getAttribute('data-max-run-time'));
+	    var autoplay = el.hasAttribute('data-autoplay');
+	    var url;
+	    var qsArgs = [
+	        'id=' + encodeURIComponent(el.getAttribute('data-id'))
+	    ];
+	    var style = IFRAME_STYLE.slice();
+	    function makeWidget(sketch) {
+	        qsArgs.push('sketch=' + encodeURIComponent(sketch));
+	        style.push('min-height: ' + height + 'px');
+	        url = myBaseURL + IFRAME_FILENAME + '?' + qsArgs.join('&');
+	        iframe.setAttribute('src', url);
+	        iframe.setAttribute('style', style.join('; '));
+	        el.parentNode.replaceChild(iframe, el);
+	    }
+	    if (!isNaN(previewWidth) && previewWidth >= 0) {
+	        qsArgs.push('previewWidth=' + previewWidth);
+	    }
+	    if (!isNaN(maxRunTime) && maxRunTime >= 0) {
+	        qsArgs.push('maxRunTime=' + maxRunTime);
+	    }
+	    if (baseSketchURL) {
+	        qsArgs.push('baseSketchURL=' + encodeURIComponent(baseSketchURL));
+	    }
+	    if (p5version) {
+	        qsArgs.push('p5version=' + encodeURIComponent(p5version));
+	    }
+	    if (autoplay) {
+	        qsArgs.push('autoplay=on');
+	    }
+	    if (el.src && el.textContent && el.textContent.trim()) {
+	        return makeWidget([
+	            '// Your widget includes both a "src" attribute and inline script',
+	            '// content, which makes no sense. Please remove one of them.'
+	        ].join('\n'));
+	    }
+	    if (el.src) {
+	        getSketch(el.src, makeWidget);
+	    }
+	    else {
+	        makeWidget(el.textContent);
+	    }
+	}
+	function whenVisible(el, cb) {
+	    var CHECK_INTERVAL_MS = 1000;
+	    var interval;
+	    function maybeMakeVisible() {
+	        if (!isElementInViewport(el))
+	            return;
+	        clearInterval(interval);
+	        window.removeEventListener('scroll', maybeMakeVisible, false);
+	        window.removeEventListener('resize', maybeMakeVisible, false);
+	        cb(el);
+	    }
+	    // We want to check at a fixed interval as a fallback, to make
+	    // sure that we detect when the element is visible even outside
+	    // of the usual means (e.g., because the user did some
+	    // sort of pinch/zoom gesture).
+	    interval = setInterval(maybeMakeVisible, 1000);
+	    window.addEventListener('scroll', maybeMakeVisible, false);
+	    window.addEventListener('resize', maybeMakeVisible, false);
+	    maybeMakeVisible();
+	}
+	function lazilyReplaceScriptWithWidget(el) {
+	    var height = getDataHeight(el);
+	    el.style.display = 'block';
+	    el.style.fontSize = '0';
+	    el.style.width = '100%';
+	    el.style.minHeight = height + 'px';
+	    el.style.background = '#f0f0f0';
+	    if (!el.hasAttribute('data-id')) {
+	        el.setAttribute('data-id', nextId.toString());
+	        nextId++;
+	    }
+	    whenVisible(el, replaceScriptWithWidget);
+	}
+	function lazilyReplaceAllScriptsWithWidget() {
+	    var scripts = document.querySelectorAll("script[type='text/p5']");
+	    [].slice.call(scripts).forEach(function (el) {
+	        lazilyReplaceScriptWithWidget(el);
+	    });
+	}
+	if (autoload) {
+	    if (document.readyState === 'complete') {
+	        lazilyReplaceAllScriptsWithWidget();
+	    }
+	    else {
+	        window.addEventListener('load', lazilyReplaceAllScriptsWithWidget, false);
+	    }
+	}
+	window['p5Widget'] = {
+	    baseURL: myBaseURL,
+	    url: myBaseURL + MY_FILENAME,
+	    replaceScript: lazilyReplaceScriptWithWidget,
+	    replaceAll: lazilyReplaceAllScriptsWithWidget,
+	    defaults: defaults
+	};
+
+
+/***/ }),
+/* 1 */
+/***/ (function(module, exports) {
+
+	"use strict";
+	exports.P5_VERSION = '0.4.23';
+	exports.PREVIEW_WIDTH = 150;
+	exports.HEIGHT = 300;
+	exports.MAX_RUN_TIME = 1000;
+
+
+/***/ })
+/******/ ]);
+//# sourceMappingURL=p5-widget.js.map

web/p5_export/p5-widget/preview-frame.bundle.js

@@ -0,0 +1,501 @@
+/******/ (function(modules) { // webpackBootstrap
+/******/ 	// The module cache
+/******/ 	var installedModules = {};
+/******/
+/******/ 	// The require function
+/******/ 	function __webpack_require__(moduleId) {
+/******/
+/******/ 		// Check if module is in cache
+/******/ 		if(installedModules[moduleId])
+/******/ 			return installedModules[moduleId].exports;
+/******/
+/******/ 		// Create a new module (and put it into the cache)
+/******/ 		var module = installedModules[moduleId] = {
+/******/ 			exports: {},
+/******/ 			id: moduleId,
+/******/ 			loaded: false
+/******/ 		};
+/******/
+/******/ 		// Execute the module function
+/******/ 		modules[moduleId].call(module.exports, module, module.exports, __webpack_require__);
+/******/
+/******/ 		// Flag the module as loaded
+/******/ 		module.loaded = true;
+/******/
+/******/ 		// Return the exports of the module
+/******/ 		return module.exports;
+/******/ 	}
+/******/
+/******/
+/******/ 	// expose the modules object (__webpack_modules__)
+/******/ 	__webpack_require__.m = modules;
+/******/
+/******/ 	// expose the module cache
+/******/ 	__webpack_require__.c = installedModules;
+/******/
+/******/ 	// __webpack_public_path__
+/******/ 	__webpack_require__.p = "";
+/******/
+/******/ 	// Load entry module and return exports
+/******/ 	return __webpack_require__(0);
+/******/ })
+/************************************************************************/
+/******/ ({
+
+/***/ 0:
+/***/ (function(module, exports, __webpack_require__) {
+
+	"use strict";
+	__webpack_require__(216);
+	// @ts-ignore
+	var global = window;
+	function loadScript(url, cb) {
+	    var script = document.createElement('script');
+	    cb = cb || (function () { });
+	    script.onload = cb;
+	    script.onerror = function () {
+	        console.log("Failed to load script: " + url);
+	    };
+	    script.setAttribute('src', url);
+	    document.body.appendChild(script);
+	}
+	function loadScripts(urls, cb) {
+	    cb = cb || (function () { });
+	    var i = 0;
+	    var loadNextScript = function () {
+	        if (i === urls.length) {
+	            return cb();
+	        }
+	        loadScript(urls[i++], loadNextScript);
+	    };
+	    loadNextScript();
+	}
+	function p5url(version) {
+	    return "//cdnjs.cloudflare.com/ajax/libs/p5.js/" + version + "/p5.js";
+	}
+	function LoopChecker(sketch, funcName, maxRunTime) {
+	    var self = {
+	        wasTriggered: false,
+	        getLineNumber: function () {
+	            var index = loopCheckFailureRange[0];
+	            var line = 1;
+	            for (var i = 0; i < index; i++) {
+	                if (sketch[i] === '\n')
+	                    line++;
+	            }
+	            return line;
+	        }
+	    };
+	    var startTime = Date.now();
+	    var loopCheckFailureRange = null;
+	    global[funcName] = function (range) {
+	        if (Date.now() - startTime > maxRunTime) {
+	            self.wasTriggered = true;
+	            loopCheckFailureRange = range;
+	            throw new Error("Loop took over " + maxRunTime + " ms to run");
+	        }
+	    };
+	    setInterval(function () {
+	        startTime = Date.now();
+	    }, maxRunTime / 2);
+	    return self;
+	}
+	function setBaseURL(url) {
+	    var base = document.createElement('base');
+	    base.setAttribute('href', url);
+	    document.head.appendChild(base);
+	}
+	function startSketch(sketch, p5version, maxRunTime, loopCheckFuncName, baseURL, errorCb) {
+	    var sketchScript = document.createElement('script');
+	    var loopChecker = LoopChecker(sketch, loopCheckFuncName, maxRunTime);
+	    if (baseURL) {
+	        setBaseURL(baseURL);
+	    }
+	    sketchScript.textContent = sketch;
+	    global.addEventListener('error', function (e) {
+	        var message = e.message;
+	        var line = undefined;
+	        if (loopChecker.wasTriggered) {
+	            message = "Your loop is taking too long to run.";
+	            line = loopChecker.getLineNumber();
+	        }
+	        else if (typeof (e.lineno) === 'number' &&
+	            (e.filename === '' || e.filename === window.location.href)) {
+	            line = e.lineno;
+	        }
+	        // p5 sketches don't actually stop looping if they throw an exception,
+	        // so try to stop the sketch.
+	        try {
+	            global.noLoop();
+	        }
+	        catch (e) { }
+	        errorCb(message, line);
+	    });
+	    loadScripts([
+	        p5url(p5version),
+	    ], function () {
+	        document.body.appendChild(sketchScript);
+	        if (document.readyState === 'complete') {
+	            new global.p5();
+	        }
+	    });
+	}
+	global.startSketch = startSketch;
+
+
+/***/ }),
+
+/***/ 210:
+/***/ (function(module, exports) {
+
+	/*
+		MIT License http://www.opensource.org/licenses/mit-license.php
+		Author Tobias Koppers @sokra
+	*/
+	// css base code, injected by the css-loader
+	module.exports = function() {
+		var list = [];
+	
+		// return the list of modules as css string
+		list.toString = function toString() {
+			var result = [];
+			for(var i = 0; i < this.length; i++) {
+				var item = this[i];
+				if(item[2]) {
+					result.push("@media " + item[2] + "{" + item[1] + "}");
+				} else {
+					result.push(item[1]);
+				}
+			}
+			return result.join("");
+		};
+	
+		// import a list of modules into the list
+		list.i = function(modules, mediaQuery) {
+			if(typeof modules === "string")
+				modules = [[null, modules, ""]];
+			var alreadyImportedModules = {};
+			for(var i = 0; i < this.length; i++) {
+				var id = this[i][0];
+				if(typeof id === "number")
+					alreadyImportedModules[id] = true;
+			}
+			for(i = 0; i < modules.length; i++) {
+				var item = modules[i];
+				// skip already imported module
+				// this implementation is not 100% perfect for weird media query combinations
+				//  when a module is imported multiple times with different media queries.
+				//  I hope this will never occur (Hey this way we have smaller bundles)
+				if(typeof item[0] !== "number" || !alreadyImportedModules[item[0]]) {
+					if(mediaQuery && !item[2]) {
+						item[2] = mediaQuery;
+					} else if(mediaQuery) {
+						item[2] = "(" + item[2] + ") and (" + mediaQuery + ")";
+					}
+					list.push(item);
+				}
+			}
+		};
+		return list;
+	};
+
+
+/***/ }),
+
+/***/ 211:
+/***/ (function(module, exports, __webpack_require__) {
+
+	/*
+		MIT License http://www.opensource.org/licenses/mit-license.php
+		Author Tobias Koppers @sokra
+	*/
+	var stylesInDom = {},
+		memoize = function(fn) {
+			var memo;
+			return function () {
+				if (typeof memo === "undefined") memo = fn.apply(this, arguments);
+				return memo;
+			};
+		},
+		isOldIE = memoize(function() {
+			return /msie [6-9]\b/.test(self.navigator.userAgent.toLowerCase());
+		}),
+		getHeadElement = memoize(function () {
+			return document.head || document.getElementsByTagName("head")[0];
+		}),
+		singletonElement = null,
+		singletonCounter = 0,
+		styleElementsInsertedAtTop = [];
+	
+	module.exports = function(list, options) {
+		if(false) {
+			if(typeof document !== "object") throw new Error("The style-loader cannot be used in a non-browser environment");
+		}
+	
+		options = options || {};
+		// Force single-tag solution on IE6-9, which has a hard limit on the # of <style>
+		// tags it will allow on a page
+		if (typeof options.singleton === "undefined") options.singleton = isOldIE();
+	
+		// By default, add <style> tags to the bottom of <head>.
+		if (typeof options.insertAt === "undefined") options.insertAt = "bottom";
+	
+		var styles = listToStyles(list);
+		addStylesToDom(styles, options);
+	
+		return function update(newList) {
+			var mayRemove = [];
+			for(var i = 0; i < styles.length; i++) {
+				var item = styles[i];
+				var domStyle = stylesInDom[item.id];
+				domStyle.refs--;
+				mayRemove.push(domStyle);
+			}
+			if(newList) {
+				var newStyles = listToStyles(newList);
+				addStylesToDom(newStyles, options);
+			}
+			for(var i = 0; i < mayRemove.length; i++) {
+				var domStyle = mayRemove[i];
+				if(domStyle.refs === 0) {
+					for(var j = 0; j < domStyle.parts.length; j++)
+						domStyle.parts[j]();
+					delete stylesInDom[domStyle.id];
+				}
+			}
+		};
+	}
+	
+	function addStylesToDom(styles, options) {
+		for(var i = 0; i < styles.length; i++) {
+			var item = styles[i];
+			var domStyle = stylesInDom[item.id];
+			if(domStyle) {
+				domStyle.refs++;
+				for(var j = 0; j < domStyle.parts.length; j++) {
+					domStyle.parts[j](item.parts[j]);
+				}
+				for(; j < item.parts.length; j++) {
+					domStyle.parts.push(addStyle(item.parts[j], options));
+				}
+			} else {
+				var parts = [];
+				for(var j = 0; j < item.parts.length; j++) {
+					parts.push(addStyle(item.parts[j], options));
+				}
+				stylesInDom[item.id] = {id: item.id, refs: 1, parts: parts};
+			}
+		}
+	}
+	
+	function listToStyles(list) {
+		var styles = [];
+		var newStyles = {};
+		for(var i = 0; i < list.length; i++) {
+			var item = list[i];
+			var id = item[0];
+			var css = item[1];
+			var media = item[2];
+			var sourceMap = item[3];
+			var part = {css: css, media: media, sourceMap: sourceMap};
+			if(!newStyles[id])
+				styles.push(newStyles[id] = {id: id, parts: [part]});
+			else
+				newStyles[id].parts.push(part);
+		}
+		return styles;
+	}
+	
+	function insertStyleElement(options, styleElement) {
+		var head = getHeadElement();
+		var lastStyleElementInsertedAtTop = styleElementsInsertedAtTop[styleElementsInsertedAtTop.length - 1];
+		if (options.insertAt === "top") {
+			if(!lastStyleElementInsertedAtTop) {
+				head.insertBefore(styleElement, head.firstChild);
+			} else if(lastStyleElementInsertedAtTop.nextSibling) {
+				head.insertBefore(styleElement, lastStyleElementInsertedAtTop.nextSibling);
+			} else {
+				head.appendChild(styleElement);
+			}
+			styleElementsInsertedAtTop.push(styleElement);
+		} else if (options.insertAt === "bottom") {
+			head.appendChild(styleElement);
+		} else {
+			throw new Error("Invalid value for parameter 'insertAt'. Must be 'top' or 'bottom'.");
+		}
+	}
+	
+	function removeStyleElement(styleElement) {
+		styleElement.parentNode.removeChild(styleElement);
+		var idx = styleElementsInsertedAtTop.indexOf(styleElement);
+		if(idx >= 0) {
+			styleElementsInsertedAtTop.splice(idx, 1);
+		}
+	}
+	
+	function createStyleElement(options) {
+		var styleElement = document.createElement("style");
+		styleElement.type = "text/css";
+		insertStyleElement(options, styleElement);
+		return styleElement;
+	}
+	
+	function createLinkElement(options) {
+		var linkElement = document.createElement("link");
+		linkElement.rel = "stylesheet";
+		insertStyleElement(options, linkElement);
+		return linkElement;
+	}
+	
+	function addStyle(obj, options) {
+		var styleElement, update, remove;
+	
+		if (options.singleton) {
+			var styleIndex = singletonCounter++;
+			styleElement = singletonElement || (singletonElement = createStyleElement(options));
+			update = applyToSingletonTag.bind(null, styleElement, styleIndex, false);
+			remove = applyToSingletonTag.bind(null, styleElement, styleIndex, true);
+		} else if(obj.sourceMap &&
+			typeof URL === "function" &&
+			typeof URL.createObjectURL === "function" &&
+			typeof URL.revokeObjectURL === "function" &&
+			typeof Blob === "function" &&
+			typeof btoa === "function") {
+			styleElement = createLinkElement(options);
+			update = updateLink.bind(null, styleElement);
+			remove = function() {
+				removeStyleElement(styleElement);
+				if(styleElement.href)
+					URL.revokeObjectURL(styleElement.href);
+			};
+		} else {
+			styleElement = createStyleElement(options);
+			update = applyToTag.bind(null, styleElement);
+			remove = function() {
+				removeStyleElement(styleElement);
+			};
+		}
+	
+		update(obj);
+	
+		return function updateStyle(newObj) {
+			if(newObj) {
+				if(newObj.css === obj.css && newObj.media === obj.media && newObj.sourceMap === obj.sourceMap)
+					return;
+				update(obj = newObj);
+			} else {
+				remove();
+			}
+		};
+	}
+	
+	var replaceText = (function () {
+		var textStore = [];
+	
+		return function (index, replacement) {
+			textStore[index] = replacement;
+			return textStore.filter(Boolean).join('\n');
+		};
+	})();
+	
+	function applyToSingletonTag(styleElement, index, remove, obj) {
+		var css = remove ? "" : obj.css;
+	
+		if (styleElement.styleSheet) {
+			styleElement.styleSheet.cssText = replaceText(index, css);
+		} else {
+			var cssNode = document.createTextNode(css);
+			var childNodes = styleElement.childNodes;
+			if (childNodes[index]) styleElement.removeChild(childNodes[index]);
+			if (childNodes.length) {
+				styleElement.insertBefore(cssNode, childNodes[index]);
+			} else {
+				styleElement.appendChild(cssNode);
+			}
+		}
+	}
+	
+	function applyToTag(styleElement, obj) {
+		var css = obj.css;
+		var media = obj.media;
+	
+		if(media) {
+			styleElement.setAttribute("media", media)
+		}
+	
+		if(styleElement.styleSheet) {
+			styleElement.styleSheet.cssText = css;
+		} else {
+			while(styleElement.firstChild) {
+				styleElement.removeChild(styleElement.firstChild);
+			}
+			styleElement.appendChild(document.createTextNode(css));
+		}
+	}
+	
+	function updateLink(linkElement, obj) {
+		var css = obj.css;
+		var sourceMap = obj.sourceMap;
+	
+		if(sourceMap) {
+			// http://stackoverflow.com/a/26603875
+			css += "\n/*# sourceMappingURL=data:application/json;base64," + btoa(unescape(encodeURIComponent(JSON.stringify(sourceMap)))) + " */";
+		}
+	
+		var blob = new Blob([css], { type: "text/css" });
+	
+		var oldSrc = linkElement.href;
+	
+		linkElement.href = URL.createObjectURL(blob);
+	
+		if(oldSrc)
+			URL.revokeObjectURL(oldSrc);
+	}
+
+
+/***/ }),
+
+/***/ 216:
+/***/ (function(module, exports, __webpack_require__) {
+
+	// style-loader: Adds some css to the DOM by adding a <style> tag
+	
+	// load the styles
+	var content = __webpack_require__(217);
+	if(typeof content === 'string') content = [[module.id, content, '']];
+	// add the styles to the DOM
+	var update = __webpack_require__(211)(content, {});
+	if(content.locals) module.exports = content.locals;
+	// Hot Module Replacement
+	if(false) {
+		// When the styles change, update the <style> tags
+		if(!content.locals) {
+			module.hot.accept("!!../node_modules/.store/css-loader@0.23.1/node_modules/css-loader/index.js?sourceMap!../node_modules/.store/postcss-loader@0.8.2/node_modules/postcss-loader/index.js?sourceMap!./preview-frame.css", function() {
+				var newContent = require("!!../node_modules/.store/css-loader@0.23.1/node_modules/css-loader/index.js?sourceMap!../node_modules/.store/postcss-loader@0.8.2/node_modules/postcss-loader/index.js?sourceMap!./preview-frame.css");
+				if(typeof newContent === 'string') newContent = [[module.id, newContent, '']];
+				update(newContent);
+			});
+		}
+		// When the module is disposed, remove the <style> tags
+		module.hot.dispose(function() { update(); });
+	}
+
+/***/ }),
+
+/***/ 217:
+/***/ (function(module, exports, __webpack_require__) {
+
+	exports = module.exports = __webpack_require__(210)();
+	// imports
+	
+	
+	// module
+	exports.push([module.id, "html, body {\r\n  height: 100%;\r\n}\r\n\r\nbody {\r\n  margin: 0;\r\n  display: -ms-flexbox;\r\n  display: flex;\r\n\r\n  /* This centers our sketch horizontally. */\r\n  -ms-flex-pack: center;\r\n      justify-content: center;\r\n\r\n  /* This centers our sketch vertically. */\r\n  -ms-flex-align: center;\r\n      align-items: center;\r\n}\r\n", "", {"version":3,"sources":["/./css/preview-frame.css"],"names":[],"mappings":"AAAA;EACE,aAAa;CACd;;AAED;EACE,UAAU;EACV,qBAAc;EAAd,cAAc;;EAEd,2CAA2C;EAC3C,sBAAwB;MAAxB,wBAAwB;;EAExB,yCAAyC;EACzC,uBAAoB;MAApB,oBAAoB;CACrB","file":"preview-frame.css","sourcesContent":["html, body {\r\n  height: 100%;\r\n}\r\n\r\nbody {\r\n  margin: 0;\r\n  display: flex;\r\n\r\n  /* This centers our sketch horizontally. */\r\n  justify-content: center;\r\n\r\n  /* This centers our sketch vertically. */\r\n  align-items: center;\r\n}\r\n"],"sourceRoot":"webpack://"}]);
+	
+	// exports
+
+
+/***/ })
+
+/******/ });
+//# sourceMappingURL=preview-frame.bundle.js.map

web/p5_export/p5-widget/preview-frame.html

@@ -0,0 +1,43 @@
+<!DOCTYPE html>
+<meta charset="utf-8">
+<title>Preview</title>
+
+<body>
+  <script src="./p5.js"></script>
+
+  <script src="./src/CCapture.js"></script>
+  <!-- <script src="src/gif.js"></script> -->
+  <!-- <script src="src/gif.worker.js"></script> -->
+  <!-- <script src="src/download.js"></script> -->
+  <!-- <script src="src/tar.js"></script> -->
+
+  <script>
+    let capturer = new CCapture({
+      format: 'png',
+      framerate: 60,
+      verbose: true
+    });
+
+
+    var capturer_start = () => {
+      if (frameCount === 1) {
+        capturer.start();
+      }
+    }
+
+
+    var capturer_end = (t = 24) => {
+
+      if (frameCount < t) {
+        capturer.capture(canvas, t);
+      } else if (frameCount === t) {
+        capturer.save(null);
+        capturer.stop();
+        capturer = null;
+      }
+    }
+
+  </script>
+
+  <script src="./preview-frame.bundle.js"></script>
+</body>

web/p5_export/p5-widget/src/Whammy.js

@@ -0,0 +1,585 @@
+/*
+	var vid = new Whammy.Video();
+	vid.add(canvas or data url)
+	vid.compile()
+*/
+
+window.Whammy = (function(){
+	// in this case, frames has a very specific meaning, which will be
+	// detailed once i finish writing the code
+
+	function toWebM(frames, outputAsArray){
+		var info = checkFrames(frames);
+
+		//max duration by cluster in milliseconds
+		var CLUSTER_MAX_DURATION = 30000;
+
+		var EBML = [
+			{
+				"id": 0x1a45dfa3, // EBML
+				"data": [
+					{
+						"data": 1,
+						"id": 0x4286 // EBMLVersion
+					},
+					{
+						"data": 1,
+						"id": 0x42f7 // EBMLReadVersion
+					},
+					{
+						"data": 4,
+						"id": 0x42f2 // EBMLMaxIDLength
+					},
+					{
+						"data": 8,
+						"id": 0x42f3 // EBMLMaxSizeLength
+					},
+					{
+						"data": "webm",
+						"id": 0x4282 // DocType
+					},
+					{
+						"data": 2,
+						"id": 0x4287 // DocTypeVersion
+					},
+					{
+						"data": 2,
+						"id": 0x4285 // DocTypeReadVersion
+					}
+				]
+			},
+			{
+				"id": 0x18538067, // Segment
+				"data": [
+					{
+						"id": 0x1549a966, // Info
+						"data": [
+							{
+								"data": 1e6, //do things in millisecs (num of nanosecs for duration scale)
+								"id": 0x2ad7b1 // TimecodeScale
+							},
+							{
+								"data": "whammy",
+								"id": 0x4d80 // MuxingApp
+							},
+							{
+								"data": "whammy",
+								"id": 0x5741 // WritingApp
+							},
+							{
+								"data": doubleToString(info.duration),
+								"id": 0x4489 // Duration
+							}
+						]
+					},
+					{
+						"id": 0x1654ae6b, // Tracks
+						"data": [
+							{
+								"id": 0xae, // TrackEntry
+								"data": [
+									{
+										"data": 1,
+										"id": 0xd7 // TrackNumber
+									},
+									{
+										"data": 1,
+										"id": 0x73c5 // TrackUID
+									},
+									{
+										"data": 0,
+										"id": 0x9c // FlagLacing
+									},
+									{
+										"data": "und",
+										"id": 0x22b59c // Language
+									},
+									{
+										"data": "V_VP8",
+										"id": 0x86 // CodecID
+									},
+									{
+										"data": "VP8",
+										"id": 0x258688 // CodecName
+									},
+									{
+										"data": 1,
+										"id": 0x83 // TrackType
+									},
+									{
+										"id": 0xe0,  // Video
+										"data": [
+											{
+												"data": info.width,
+												"id": 0xb0 // PixelWidth
+											},
+											{
+												"data": info.height,
+												"id": 0xba // PixelHeight
+											}
+										]
+									}
+								]
+							}
+						]
+					},
+					{
+						"id": 0x1c53bb6b, // Cues
+						"data": [
+							//cue insertion point
+						]
+					}
+
+					//cluster insertion point
+				]
+			}
+		 ];
+
+
+		var segment = EBML[1];
+		var cues = segment.data[2];
+
+		//Generate clusters (max duration)
+		var frameNumber = 0;
+		var clusterTimecode = 0;
+		while(frameNumber < frames.length){
+
+			var cuePoint = {
+					"id": 0xbb, // CuePoint
+					"data": [
+						{
+							"data": Math.round(clusterTimecode),
+							"id": 0xb3 // CueTime
+						},
+						{
+							"id": 0xb7, // CueTrackPositions
+							"data": [
+								{
+									"data": 1,
+									"id": 0xf7 // CueTrack
+								},
+								{
+									"data": 0, // to be filled in when we know it
+									"size": 8,
+									"id": 0xf1 // CueClusterPosition
+								}
+							]
+						}
+					]
+				};
+
+			cues.data.push(cuePoint);
+
+			var clusterFrames = [];
+			var clusterDuration = 0;
+			do {
+				clusterFrames.push(frames[frameNumber]);
+				clusterDuration += frames[frameNumber].duration;
+				frameNumber++;
+			}while(frameNumber < frames.length && clusterDuration < CLUSTER_MAX_DURATION);
+
+			var clusterCounter = 0;
+			var cluster = {
+					"id": 0x1f43b675, // Cluster
+					"data": [
+						{
+							"data": Math.round(clusterTimecode),
+							"id": 0xe7 // Timecode
+						}
+					].concat(clusterFrames.map(function(webp){
+						var block = makeSimpleBlock({
+							discardable: 0,
+							frame: webp.data.slice(4),
+							invisible: 0,
+							keyframe: 1,
+							lacing: 0,
+							trackNum: 1,
+							timecode: Math.round(clusterCounter)
+						});
+						clusterCounter += webp.duration;
+						return {
+							data: block,
+							id: 0xa3
+						};
+					}))
+				}
+
+			//Add cluster to segment
+			segment.data.push(cluster);
+			clusterTimecode += clusterDuration;
+		}
+
+		//First pass to compute cluster positions
+		var position = 0;
+		for(var i = 0; i < segment.data.length; i++){
+			if (i >= 3) {
+				cues.data[i-3].data[1].data[1].data = position;
+			}
+			var data = generateEBML([segment.data[i]], outputAsArray);
+			position += data.size || data.byteLength || data.length;
+			if (i != 2) { // not cues
+				//Save results to avoid having to encode everything twice
+				segment.data[i] = data;
+			}
+		}
+
+		return generateEBML(EBML, outputAsArray)
+	}
+
+	// sums the lengths of all the frames and gets the duration, woo
+
+	function checkFrames(frames){
+		var width = frames[0].width,
+			height = frames[0].height,
+			duration = frames[0].duration;
+		for(var i = 1; i < frames.length; i++){
+			if(frames[i].width != width) throw "Frame " + (i + 1) + " has a different width";
+			if(frames[i].height != height) throw "Frame " + (i + 1) + " has a different height";
+			if(frames[i].duration < 0 || frames[i].duration > 0x7fff) throw "Frame " + (i + 1) + " has a weird duration (must be between 0 and 32767)";
+			duration += frames[i].duration;
+		}
+		return {
+			duration: duration,
+			width: width,
+			height: height
+		};
+	}
+
+
+	function numToBuffer(num){
+		var parts = [];
+		while(num > 0){
+			parts.push(num & 0xff)
+			num = num >> 8
+		}
+		return new Uint8Array(parts.reverse());
+	}
+
+	function numToFixedBuffer(num, size){
+		var parts = new Uint8Array(size);
+		for(var i = size - 1; i >= 0; i--){
+			parts[i] = num & 0xff;
+			num = num >> 8;
+		}
+		return parts;
+	}
+
+	function strToBuffer(str){
+		// return new Blob([str]);
+
+		var arr = new Uint8Array(str.length);
+		for(var i = 0; i < str.length; i++){
+			arr[i] = str.charCodeAt(i)
+		}
+		return arr;
+		// this is slower
+		// return new Uint8Array(str.split('').map(function(e){
+		// 	return e.charCodeAt(0)
+		// }))
+	}
+
+
+	//sorry this is ugly, and sort of hard to understand exactly why this was done
+	// at all really, but the reason is that there's some code below that i dont really
+	// feel like understanding, and this is easier than using my brain.
+
+	function bitsToBuffer(bits){
+		var data = [];
+		var pad = (bits.length % 8) ? (new Array(1 + 8 - (bits.length % 8))).join('0') : '';
+		bits = pad + bits;
+		for(var i = 0; i < bits.length; i+= 8){
+			data.push(parseInt(bits.substr(i,8),2))
+		}
+		return new Uint8Array(data);
+	}
+
+	function generateEBML(json, outputAsArray){
+		var ebml = [];
+		for(var i = 0; i < json.length; i++){
+			if (!('id' in json[i])){
+				//already encoded blob or byteArray
+				ebml.push(json[i]);
+				continue;
+			}
+
+			var data = json[i].data;
+			if(typeof data == 'object') data = generateEBML(data, outputAsArray);
+			if(typeof data == 'number') data = ('size' in json[i]) ? numToFixedBuffer(data, json[i].size) : bitsToBuffer(data.toString(2));
+			if(typeof data == 'string') data = strToBuffer(data);
+
+			if(data.length){
+				var z = z;
+			}
+
+			/*var len = data.size || data.byteLength || data.length;
+			var zeroes = Math.ceil(Math.ceil(Math.log(len)/Math.log(2))/8);
+			var size_str = len.toString(2);
+			var padded = (new Array((zeroes * 7 + 7 + 1) - size_str.length)).join('0') + size_str;
+			var size = (new Array(zeroes)).join('0') + '1' + padded;*/
+
+			var len = data.size || data.byteLength || data.length;
+			var zeroes = 0
+			for( var j = 56; j > 0; j-= 7 ) {
+				if( len > Math.pow( 2, j ) - 2 ) {
+					zeroes = j / 7
+					break
+				}
+			}
+			var size_str = len.toString(2);
+			var base = ( new Array( 8 * ( zeroes + 1 ) + 1 ) ).join( '0' )
+			var pre = ( new Array( zeroes + 1 ) ).join( '0' ) + 1;
+			var padded = base.substr( 0, base.length - size_str.length - pre.length ) + size_str;
+			var size = pre + padded;
+			//i actually dont quite understand what went on up there, so I'm not really
+			//going to fix this, i'm probably just going to write some hacky thing which
+			//converts that string into a buffer-esque thing
+
+			ebml.push(numToBuffer(json[i].id));
+			ebml.push(bitsToBuffer(size));
+			ebml.push(data)
+
+
+		}
+
+		//output as blob or byteArray
+		if(outputAsArray){
+			//convert ebml to an array
+			var buffer = toFlatArray(ebml)
+			return new Uint8Array(buffer);
+		}else{
+			return new Blob(ebml, {type: "video/webm"});
+		}
+	}
+
+	function toFlatArray(arr, outBuffer){
+		if(outBuffer == null){
+			outBuffer = [];
+		}
+		for(var i = 0; i < arr.length; i++){
+			if(typeof arr[i] == 'object'){
+				//an array
+				toFlatArray(arr[i], outBuffer)
+			}else{
+				//a simple element
+				outBuffer.push(arr[i]);
+			}
+		}
+		return outBuffer;
+	}
+
+	//OKAY, so the following two functions are the string-based old stuff, the reason they're
+	//still sort of in here, is that they're actually faster than the new blob stuff because
+	//getAsFile isn't widely implemented, or at least, it doesn't work in chrome, which is the
+	// only browser which supports get as webp
+
+	//Converting between a string of 0010101001's and binary back and forth is probably inefficient
+	//TODO: get rid of this function
+	function toBinStr_old(bits){
+		var data = '';
+		var pad = (bits.length % 8) ? (new Array(1 + 8 - (bits.length % 8))).join('0') : '';
+		bits = pad + bits;
+		for(var i = 0; i < bits.length; i+= 8){
+			data += String.fromCharCode(parseInt(bits.substr(i,8),2))
+		}
+		return data;
+	}
+
+	function generateEBML_old(json){
+		var ebml = '';
+		for(var i = 0; i < json.length; i++){
+			var data = json[i].data;
+			if(typeof data == 'object') data = generateEBML_old(data);
+			if(typeof data == 'number') data = toBinStr_old(data.toString(2));
+
+			var len = data.length;
+			var zeroes = Math.ceil(Math.ceil(Math.log(len)/Math.log(2))/8);
+			var size_str = len.toString(2);
+			var padded = (new Array((zeroes * 7 + 7 + 1) - size_str.length)).join('0') + size_str;
+			var size = (new Array(zeroes)).join('0') + '1' + padded;
+
+			ebml += toBinStr_old(json[i].id.toString(2)) + toBinStr_old(size) + data;
+
+		}
+		return ebml;
+	}
+
+	//woot, a function that's actually written for this project!
+	//this parses some json markup and makes it into that binary magic
+	//which can then get shoved into the matroska comtainer (peaceably)
+
+	function makeSimpleBlock(data){
+		var flags = 0;
+		if (data.keyframe) flags |= 128;
+		if (data.invisible) flags |= 8;
+		if (data.lacing) flags |= (data.lacing << 1);
+		if (data.discardable) flags |= 1;
+		if (data.trackNum > 127) {
+			throw "TrackNumber > 127 not supported";
+		}
+		var out = [data.trackNum | 0x80, data.timecode >> 8, data.timecode & 0xff, flags].map(function(e){
+			return String.fromCharCode(e)
+		}).join('') + data.frame;
+
+		return out;
+	}
+
+	// here's something else taken verbatim from weppy, awesome rite?
+
+	function parseWebP(riff){
+		var VP8 = riff.RIFF[0].WEBP[0];
+
+		var frame_start = VP8.indexOf('\x9d\x01\x2a'); //A VP8 keyframe starts with the 0x9d012a header
+		for(var i = 0, c = []; i < 4; i++) c[i] = VP8.charCodeAt(frame_start + 3 + i);
+
+		var width, horizontal_scale, height, vertical_scale, tmp;
+
+		//the code below is literally copied verbatim from the bitstream spec
+		tmp = (c[1] << 8) | c[0];
+		width = tmp & 0x3FFF;
+		horizontal_scale = tmp >> 14;
+		tmp = (c[3] << 8) | c[2];
+		height = tmp & 0x3FFF;
+		vertical_scale = tmp >> 14;
+		return {
+			width: width,
+			height: height,
+			data: VP8,
+			riff: riff
+		}
+	}
+
+	// i think i'm going off on a riff by pretending this is some known
+	// idiom which i'm making a casual and brilliant pun about, but since
+	// i can't find anything on google which conforms to this idiomatic
+	// usage, I'm assuming this is just a consequence of some psychotic
+	// break which makes me make up puns. well, enough riff-raff (aha a
+	// rescue of sorts), this function was ripped wholesale from weppy
+
+	function parseRIFF(string){
+		var offset = 0;
+		var chunks = {};
+
+		while (offset < string.length) {
+			var id = string.substr(offset, 4);
+			chunks[id] = chunks[id] || [];
+			if (id == 'RIFF' || id == 'LIST') {
+				var len = parseInt(string.substr(offset + 4, 4).split('').map(function(i){
+					var unpadded = i.charCodeAt(0).toString(2);
+					return (new Array(8 - unpadded.length + 1)).join('0') + unpadded
+				}).join(''),2);
+				var data = string.substr(offset + 4 + 4, len);
+				offset += 4 + 4 + len;
+				chunks[id].push(parseRIFF(data));
+			} else if (id == 'WEBP') {
+				// Use (offset + 8) to skip past "VP8 "/"VP8L"/"VP8X" field after "WEBP"
+				chunks[id].push(string.substr(offset + 8));
+				offset = string.length;
+			} else {
+				// Unknown chunk type; push entire payload
+				chunks[id].push(string.substr(offset + 4));
+				offset = string.length;
+			}
+		}
+		return chunks;
+	}
+
+	// here's a little utility function that acts as a utility for other functions
+	// basically, the only purpose is for encoding "Duration", which is encoded as
+	// a double (considerably more difficult to encode than an integer)
+	function doubleToString(num){
+		return [].slice.call(
+			new Uint8Array(
+				(
+					new Float64Array([num]) //create a float64 array
+				).buffer) //extract the array buffer
+			, 0) // convert the Uint8Array into a regular array
+			.map(function(e){ //since it's a regular array, we can now use map
+				return String.fromCharCode(e) // encode all the bytes individually
+			})
+			.reverse() //correct the byte endianness (assume it's little endian for now)
+			.join('') // join the bytes in holy matrimony as a string
+	}
+
+	function WhammyVideo(speed, quality){ // a more abstract-ish API
+		this.frames = [];
+		this.duration = 1000 / speed;
+		this.quality = quality || 0.8;
+	}
+
+	WhammyVideo.prototype.add = function(frame, duration){
+		if(typeof duration != 'undefined' && this.duration) throw "you can't pass a duration if the fps is set";
+		if(typeof duration == 'undefined' && !this.duration) throw "if you don't have the fps set, you need to have durations here.";
+		if(frame.canvas){ //CanvasRenderingContext2D
+			frame = frame.canvas;
+		}
+		if(frame.toDataURL){
+			// frame = frame.toDataURL('image/webp', this.quality);
+			// quickly store image data so we don't block cpu. encode in compile method.
+			frame = frame.getContext('2d').getImageData(0, 0, frame.width, frame.height);
+		}else if(typeof frame != "string"){
+			throw "frame must be a a HTMLCanvasElement, a CanvasRenderingContext2D or a DataURI formatted string"
+		}
+		if (typeof frame === "string" && !(/^data:image\/webp;base64,/ig).test(frame)) {
+			throw "Input must be formatted properly as a base64 encoded DataURI of type image/webp";
+		}
+		this.frames.push({
+			image: frame,
+			duration: duration || this.duration
+		});
+	};
+
+	// deferred webp encoding. Draws image data to canvas, then encodes as dataUrl
+	WhammyVideo.prototype.encodeFrames = function(callback){
+
+		if(this.frames[0].image instanceof ImageData){
+
+			var frames = this.frames;
+			var tmpCanvas = document.createElement('canvas');
+			var tmpContext = tmpCanvas.getContext('2d');
+			tmpCanvas.width = this.frames[0].image.width;
+			tmpCanvas.height = this.frames[0].image.height;
+
+			var encodeFrame = function(index){
+				var frame = frames[index];
+				tmpContext.putImageData(frame.image, 0, 0);
+				frame.image = tmpCanvas.toDataURL('image/webp', this.quality);
+				if(index < frames.length-1){
+					setTimeout(function(){ encodeFrame(index + 1); }, 1);
+				}else{
+					callback();
+				}
+			}.bind(this);
+
+			encodeFrame(0);
+		}else{
+			callback();
+		}
+	};
+
+	WhammyVideo.prototype.compile = function(outputAsArray, callback){
+
+		this.encodeFrames(function(){
+
+			var webm = new toWebM(this.frames.map(function(frame){
+				var webp = parseWebP(parseRIFF(atob(frame.image.slice(23))));
+				webp.duration = frame.duration;
+				return webp;
+			}), outputAsArray);
+			callback(webm);
+			
+		}.bind(this));
+	};
+
+	return {
+		Video: WhammyVideo,
+		fromImageArray: function(images, fps, outputAsArray){
+			return toWebM(images.map(function(image){
+				var webp = parseWebP(parseRIFF(atob(image.slice(23))))
+				webp.duration = 1000 / fps;
+				return webp;
+			}), outputAsArray)
+		},
+		toWebM: toWebM
+		// expose methods of madness
+	}
+})()

web/p5_export/p5-widget/src/tar.js

@@ -0,0 +1,334 @@
+(function () {
+	"use strict";
+
+	var lookup = [
+			'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H',
+			'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P',
+			'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X',
+			'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f',
+			'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n',
+			'o', 'p', 'q', 'r', 's', 't', 'u', 'v',
+			'w', 'x', 'y', 'z', '0', '1', '2', '3',
+			'4', '5', '6', '7', '8', '9', '+', '/'
+		];
+	function clean(length) {
+		var i, buffer = new Uint8Array(length);
+		for (i = 0; i < length; i += 1) {
+			buffer[i] = 0;
+		}
+		return buffer;
+	}
+
+	function extend(orig, length, addLength, multipleOf) {
+		var newSize = length + addLength,
+			buffer = clean((parseInt(newSize / multipleOf) + 1) * multipleOf);
+
+		buffer.set(orig);
+
+		return buffer;
+	}
+
+	function pad(num, bytes, base) {
+		num = num.toString(base || 8);
+		return "000000000000".substr(num.length + 12 - bytes) + num;
+	}	
+	
+	function stringToUint8 (input, out, offset) {
+		var i, length;
+
+		out = out || clean(input.length);
+
+		offset = offset || 0;
+		for (i = 0, length = input.length; i < length; i += 1) {
+			out[offset] = input.charCodeAt(i);
+			offset += 1;
+		}
+
+		return out;
+	}
+
+	function uint8ToBase64(uint8) {
+		var i,
+			extraBytes = uint8.length % 3, // if we have 1 byte left, pad 2 bytes
+			output = "",
+			temp, length;
+
+		function tripletToBase64 (num) {
+			return lookup[num >> 18 & 0x3F] + lookup[num >> 12 & 0x3F] + lookup[num >> 6 & 0x3F] + lookup[num & 0x3F];
+		};
+
+		// go through the array every three bytes, we'll deal with trailing stuff later
+		for (i = 0, length = uint8.length - extraBytes; i < length; i += 3) {
+			temp = (uint8[i] << 16) + (uint8[i + 1] << 8) + (uint8[i + 2]);
+			output += tripletToBase64(temp);
+		}
+
+		// this prevents an ERR_INVALID_URL in Chrome (Firefox okay)
+		switch (output.length % 4) {
+			case 1:
+				output += '=';
+				break;
+			case 2:
+				output += '==';
+				break;
+			default:
+				break;
+		}
+
+		return output;
+	}
+
+	window.utils = {}
+	window.utils.clean = clean;
+	window.utils.pad = pad;
+	window.utils.extend = extend;
+	window.utils.stringToUint8 = stringToUint8;
+	window.utils.uint8ToBase64 = uint8ToBase64;
+}());
+
+(function () {
+	"use strict";
+	
+/*
+struct posix_header {             // byte offset
+	char name[100];               //   0
+	char mode[8];                 // 100
+	char uid[8];                  // 108
+	char gid[8];                  // 116
+	char size[12];                // 124
+	char mtime[12];               // 136
+	char chksum[8];               // 148
+	char typeflag;                // 156
+	char linkname[100];           // 157
+	char magic[6];                // 257
+	char version[2];              // 263
+	char uname[32];               // 265
+	char gname[32];               // 297
+	char devmajor[8];             // 329
+	char devminor[8];             // 337
+	char prefix[155];             // 345
+                                  // 500
+};
+*/
+
+	var utils = window.utils,
+		headerFormat;
+	
+	headerFormat = [
+		{
+			'field': 'fileName',
+			'length': 100
+		},
+		{
+			'field': 'fileMode',
+			'length': 8
+		},
+		{
+			'field': 'uid',
+			'length': 8
+		},
+		{
+			'field': 'gid',
+			'length': 8
+		},
+		{
+			'field': 'fileSize',
+			'length': 12
+		},
+		{
+			'field': 'mtime',
+			'length': 12
+		},
+		{
+			'field': 'checksum',
+			'length': 8
+		},
+		{
+			'field': 'type',
+			'length': 1
+		},
+		{
+			'field': 'linkName',
+			'length': 100
+		},
+		{
+			'field': 'ustar',
+			'length': 8
+		},
+		{
+			'field': 'owner',
+			'length': 32
+		},
+		{
+			'field': 'group',
+			'length': 32
+		},
+		{
+			'field': 'majorNumber',
+			'length': 8
+		},
+		{
+			'field': 'minorNumber',
+			'length': 8
+		},
+		{
+			'field': 'filenamePrefix',
+			'length': 155
+		},
+		{
+			'field': 'padding',
+			'length': 12
+		}
+	];
+
+	function formatHeader(data, cb) {
+		var buffer = utils.clean(512),
+			offset = 0;
+
+		headerFormat.forEach(function (value) {
+			var str = data[value.field] || "",
+				i, length;
+
+			for (i = 0, length = str.length; i < length; i += 1) {
+				buffer[offset] = str.charCodeAt(i);
+				offset += 1;
+			}
+
+			offset += value.length - i; // space it out with nulls
+		});
+
+		if (typeof cb === 'function') {
+			return cb(buffer, offset);
+		}
+		return buffer;
+	}
+	
+	window.header = {}
+	window.header.structure = headerFormat;
+	window.header.format = formatHeader;
+}());
+
+(function () {
+	"use strict";
+
+	var header = window.header,
+		utils = window.utils,
+		recordSize = 512,
+		blockSize;
+	
+	function Tar(recordsPerBlock) {
+		this.written = 0;
+		blockSize = (recordsPerBlock || 20) * recordSize;
+		this.out = utils.clean(blockSize);
+		this.blocks = [];
+		this.length = 0;
+	}
+
+	Tar.prototype.append = function (filepath, input, opts, callback) {
+		var data,
+			checksum,
+			mode,
+			mtime,
+			uid,
+			gid,
+			headerArr;
+
+		if (typeof input === 'string') {
+			input = utils.stringToUint8(input);
+		} else if (input.constructor !== Uint8Array.prototype.constructor) {
+			throw 'Invalid input type. You gave me: ' + input.constructor.toString().match(/function\s*([$A-Za-z_][0-9A-Za-z_]*)\s*\(/)[1];
+		}
+
+		if (typeof opts === 'function') {
+			callback = opts;
+			opts = {};
+		}
+
+		opts = opts || {};
+
+		mode = opts.mode || parseInt('777', 8) & 0xfff;
+		mtime = opts.mtime || Math.floor(+new Date() / 1000);
+		uid = opts.uid || 0;
+		gid = opts.gid || 0;
+
+		data = {
+			fileName: filepath,
+			fileMode: utils.pad(mode, 7),
+			uid: utils.pad(uid, 7),
+			gid: utils.pad(gid, 7),
+			fileSize: utils.pad(input.length, 11),
+			mtime: utils.pad(mtime, 11),
+			checksum: '        ',
+			type: '0', // just a file
+			ustar: 'ustar  ',
+			owner: opts.owner || '',
+			group: opts.group || ''
+		};
+
+		// calculate the checksum
+		checksum = 0;
+		Object.keys(data).forEach(function (key) {
+			var i, value = data[key], length;
+
+			for (i = 0, length = value.length; i < length; i += 1) {
+				checksum += value.charCodeAt(i);
+			}
+		});
+
+		data.checksum = utils.pad(checksum, 6) + "\u0000 ";
+
+		headerArr = header.format(data);
+
+		var headerLength = Math.ceil( headerArr.length / recordSize ) * recordSize; 
+		var inputLength = Math.ceil( input.length / recordSize ) * recordSize;
+
+		this.blocks.push( { header: headerArr, input: input, headerLength: headerLength, inputLength: inputLength } );
+
+	};
+
+	Tar.prototype.save = function() {
+
+		var buffers = [];
+		var chunks = [];
+		var length = 0;
+		var max = Math.pow( 2, 20 );
+
+		var chunk = [];
+		this.blocks.forEach( function( b ) {
+			if( length + b.headerLength + b.inputLength > max ) {
+				chunks.push( { blocks: chunk, length: length } );
+				chunk = [];
+				length = 0;
+			}
+			chunk.push( b );
+			length += b.headerLength + b.inputLength;
+		} );
+		chunks.push( { blocks: chunk, length: length } );
+
+		chunks.forEach( function( c ) {
+
+			var buffer = new Uint8Array( c.length );
+			var written = 0;
+			c.blocks.forEach( function( b ) {
+				buffer.set( b.header, written );
+				written += b.headerLength;
+				buffer.set( b.input, written );
+				written += b.inputLength;
+			} );
+			buffers.push( buffer );
+
+		} );
+
+		buffers.push( new Uint8Array( 2 * recordSize ) );
+
+		return new Blob( buffers, { type: 'octet/stream' } );
+		
+	};
+
+	Tar.prototype.clear = function () {
+		this.written = 0;
+		this.out = utils.clean(blockSize);
+	};
+
+	window.Tar = Tar;
+}());

web/p5_export/p5-widget/static/img/p5js-beta.svg

@@ -0,0 +1,54 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 16.0.0, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+   width="250px" height="114px" viewBox="0 0 250 114" enable-background="new 0 0 250 114" xml:space="preserve">
+<path fill="#EC245E" d="M16.254,27.631v7.998h0.359c0.715-1.113,1.65-2.248,2.805-3.402c1.155-1.154,2.568-2.188,4.24-3.105
+  c1.67-0.912,3.561-1.67,5.67-2.268c2.107-0.596,4.477-0.896,7.104-0.896c4.059,0,7.799,0.777,11.223,2.328
+  c3.422,1.555,6.367,3.684,8.836,6.389c2.465,2.707,4.375,5.891,5.73,9.551c1.352,3.662,2.029,7.602,2.029,11.82
+  s-0.656,8.179-1.971,11.879c-1.312,3.701-3.184,6.925-5.611,9.67c-2.427,2.746-5.371,4.938-8.834,6.566
+  c-3.463,1.631-7.385,2.446-11.76,2.446c-4.061,0-7.781-0.836-11.164-2.506c-3.385-1.672-5.99-3.938-7.82-6.807h-0.238v36.295H2.525
+  V27.631H16.254z M49.684,56.045c0-2.229-0.338-4.438-1.014-6.627c-0.678-2.188-1.693-4.158-3.045-5.91
+  c-1.354-1.748-3.064-3.162-5.135-4.238c-2.07-1.074-4.496-1.611-7.281-1.611c-2.627,0-4.977,0.557-7.045,1.672
+  c-2.07,1.115-3.842,2.549-5.312,4.297c-1.475,1.752-2.588,3.742-3.344,5.971c-0.758,2.229-1.133,4.459-1.133,6.686
+  c0,2.229,0.375,4.438,1.133,6.625c0.756,2.191,1.869,4.16,3.344,5.912c1.471,1.75,3.242,3.164,5.312,4.236
+  c2.068,1.075,4.418,1.61,7.045,1.61c2.785,0,5.211-0.555,7.281-1.67c2.07-1.115,3.781-2.547,5.135-4.299
+  c1.352-1.75,2.367-3.74,3.045-5.97C49.346,60.502,49.684,58.273,49.684,56.045z M189.332,24.893v63.505
+  c0,3.422-0.279,6.666-0.836,9.73c-0.559,3.064-1.611,5.73-3.164,8c-1.551,2.27-3.662,4.078-6.328,5.432
+  c-2.668,1.354-6.146,2.029-10.445,2.029c-1.193,0-2.389-0.08-3.582-0.238c-1.193-0.16-2.148-0.319-2.865-0.479l1.195-12.178
+  c0.637,0.16,1.311,0.279,2.027,0.359c0.717,0.077,1.354,0.118,1.91,0.118c1.67,0,3.023-0.317,4.059-0.955
+  c1.033-0.639,1.83-1.514,2.391-2.627c0.555-1.114,0.914-2.407,1.074-3.881c0.156-1.474,0.236-3.043,0.236-4.715V24.893H189.332z
+   M238.162,42.912c-1.275-1.672-3.025-3.123-5.254-4.357c-2.229-1.234-4.656-1.852-7.283-1.852c-2.309,0-4.416,0.479-6.326,1.434
+  c-1.912,0.953-2.863,2.547-2.863,4.775s1.053,3.803,3.16,4.715c2.109,0.916,5.195,1.852,9.256,2.807
+  c2.146,0.479,4.314,1.115,6.506,1.91c2.189,0.795,4.18,1.85,5.971,3.164c1.789,1.312,3.242,2.945,4.357,4.895
+  c1.111,1.951,1.672,4.318,1.672,7.104c0,3.504-0.658,6.47-1.973,8.896c-1.311,2.428-3.062,4.397-5.254,5.91
+  c-2.189,1.512-4.734,2.606-7.641,3.283c-2.906,0.676-5.908,1.014-9.014,1.014c-4.459,0-8.795-0.816-13.014-2.447
+  c-4.219-1.629-7.721-3.959-10.506-6.982l9.432-8.836c1.592,2.07,3.66,3.781,6.209,5.133c2.547,1.354,5.371,2.029,8.477,2.029
+  c1.033,0,2.088-0.117,3.164-0.357c1.074-0.237,2.068-0.614,2.984-1.133c0.914-0.518,1.65-1.213,2.209-2.09
+  c0.555-0.877,0.834-1.949,0.834-3.225c0-2.389-1.094-4.098-3.281-5.133c-2.191-1.035-5.475-2.07-9.85-3.104
+  c-2.15-0.479-4.24-1.094-6.27-1.853c-2.029-0.756-3.84-1.75-5.432-2.983c-1.596-1.234-2.865-2.764-3.82-4.598
+  c-0.955-1.83-1.436-4.098-1.436-6.805c0-3.184,0.656-5.928,1.973-8.236c1.311-2.312,3.045-4.197,5.191-5.674
+  c2.148-1.471,4.576-2.566,7.283-3.281c2.705-0.717,5.492-1.076,8.357-1.076c4.137,0,8.178,0.717,12.117,2.148
+  c3.939,1.434,7.062,3.625,9.373,6.568L238.162,42.912z M153.559,72.816l8.533-2.576l1.676,5.156l-8.498,2.897l5.275,7.479
+  l-4.447,3.226l-5.553-7.349l-5.408,7.154l-4.318-3.289l5.275-7.223l-8.564-3.09l1.678-5.16l8.6,2.771v-8.896h5.754v8.897H153.559z
+   M124.086,45.836c-1.473-3.301-3.52-6.088-6.148-8.357c-2.625-2.268-5.711-4-9.252-5.193c-3.543-1.193-7.383-1.791-11.521-1.791
+  c-1.512,0-3.203,0.082-5.074,0.238c-1.871,0.162-3.482,0.439-4.834,0.838l0.834-18.268h34.503V0.41H74.481l-1.432,46.201
+  c1.271-0.635,2.725-1.232,4.357-1.791c1.631-0.555,3.301-1.053,5.014-1.49c1.711-0.438,3.463-0.775,5.254-1.016
+  c1.791-0.238,3.48-0.357,5.074-0.357c2.307,0,4.576,0.258,6.805,0.775c2.227,0.518,4.238,1.434,6.029,2.746s3.242,3.045,4.357,5.193
+  c1.113,2.148,1.672,4.855,1.672,8.119c0,2.547-0.418,4.836-1.254,6.865c-0.836,2.026-1.971,3.721-3.402,5.071
+  c-1.434,1.355-3.104,2.39-5.016,3.104c-1.91,0.719-3.939,1.076-6.088,1.076c-3.82,0-7.125-1.017-9.91-3.046
+  c-2.787-2.028-4.775-4.715-5.969-8.059l-0.16,0.059l-10.367,9.716c2.096,3.42,4.799,6.28,8.139,8.553
+  c4.854,3.302,10.824,4.955,17.91,4.955c4.217,0,8.197-0.678,11.938-2.028c3.741-1.352,7.004-3.304,9.791-5.853
+  c2.786-2.545,4.994-5.67,6.627-9.371c1.629-3.701,2.445-7.897,2.445-12.597C126.295,52.939,125.559,49.141,124.086,45.836z
+   M131.07,6.842h2.521c0.244,0,0.484,0.029,0.723,0.086c0.236,0.059,0.447,0.152,0.635,0.283c0.186,0.131,0.336,0.301,0.453,0.508
+  c0.115,0.207,0.172,0.457,0.172,0.749c0,0.365-0.104,0.667-0.311,0.904c-0.207,0.237-0.479,0.407-0.812,0.511v0.02
+  c0.408,0.055,0.742,0.213,1.006,0.475c0.262,0.262,0.393,0.611,0.393,1.051c0,0.354-0.07,0.65-0.209,0.891
+  c-0.143,0.24-0.324,0.434-0.555,0.58c-0.229,0.146-0.488,0.251-0.785,0.314c-0.295,0.064-0.596,0.096-0.898,0.096h-2.33V6.842
+  H131.07z M132.221,9.473h1.023c0.383,0,0.676-0.076,0.877-0.229c0.201-0.153,0.301-0.369,0.301-0.648c0-0.293-0.104-0.5-0.311-0.621
+  c-0.207-0.122-0.529-0.184-0.969-0.184h-0.924v1.682H132.221z M132.221,12.341h1.031c0.146,0,0.307-0.011,0.477-0.032
+  s0.328-0.064,0.471-0.133c0.143-0.066,0.262-0.164,0.355-0.292c0.096-0.128,0.143-0.298,0.143-0.511
+  c0-0.342-0.115-0.579-0.348-0.713c-0.23-0.135-0.582-0.201-1.051-0.201h-1.078V12.341z M136.936,6.842h4.283v1.004h-3.135v1.645
+  h2.969v0.969h-2.969v1.827h3.299v1.022h-4.447V6.842z M144.088,7.846h-1.982V6.842h5.117v1.004h-1.982v5.463h-1.152V7.846
+  L144.088,7.846z M149.449,6.842h0.996l2.787,6.467h-1.316l-0.602-1.479h-2.807l-0.584,1.479h-1.289L149.449,6.842z M150.912,10.843
+  l-0.996-2.631l-1.014,2.631H150.912z"/>
+</svg>

web/p5_export/p5-widget/static/index.css

@@ -0,0 +1,17 @@
+html {
+  min-width: 768px;
+}
+
+body {
+  font-family: Georgia, serif;
+  max-width: 740px;
+  margin: 0 auto;
+}
+
+h1, h2, h3 {
+  font-weight: normal;
+}
+
+a {
+  color: inherit;
+}

web/p5_export/p5-widget/static/vendor/codemirror.css

@@ -0,0 +1,338 @@
+/* BASICS */
+
+.CodeMirror {
+  /* Set height, width, borders, and global font properties here */
+  font-family: monospace;
+  height: 300px;
+  color: black;
+}
+
+/* PADDING */
+
+.CodeMirror-lines {
+  padding: 4px 0; /* Vertical padding around content */
+}
+.CodeMirror pre {
+  padding: 0 4px; /* Horizontal padding of content */
+}
+
+.CodeMirror-scrollbar-filler, .CodeMirror-gutter-filler {
+  background-color: white; /* The little square between H and V scrollbars */
+}
+
+/* GUTTER */
+
+.CodeMirror-gutters {
+  border-right: 1px solid #ddd;
+  background-color: #f7f7f7;
+  white-space: nowrap;
+}
+.CodeMirror-linenumbers {}
+.CodeMirror-linenumber {
+  padding: 0 3px 0 5px;
+  min-width: 20px;
+  text-align: right;
+  color: #999;
+  white-space: nowrap;
+}
+
+.CodeMirror-guttermarker { color: black; }
+.CodeMirror-guttermarker-subtle { color: #999; }
+
+/* CURSOR */
+
+.CodeMirror-cursor {
+  border-left: 1px solid black;
+  border-right: none;
+  width: 0;
+}
+/* Shown when moving in bi-directional text */
+.CodeMirror div.CodeMirror-secondarycursor {
+  border-left: 1px solid silver;
+}
+.cm-fat-cursor .CodeMirror-cursor {
+  width: auto;
+  border: 0;
+  background: #7e7;
+}
+.cm-fat-cursor div.CodeMirror-cursors {
+  z-index: 1;
+}
+
+.cm-animate-fat-cursor {
+  width: auto;
+  border: 0;
+  -webkit-animation: blink 1.06s steps(1) infinite;
+  -moz-animation: blink 1.06s steps(1) infinite;
+  animation: blink 1.06s steps(1) infinite;
+  background-color: #7e7;
+}
+@-moz-keyframes blink {
+  0% {}
+  50% { background-color: transparent; }
+  100% {}
+}
+@-webkit-keyframes blink {
+  0% {}
+  50% { background-color: transparent; }
+  100% {}
+}
+@keyframes blink {
+  0% {}
+  50% { background-color: transparent; }
+  100% {}
+}
+
+/* Can style cursor different in overwrite (non-insert) mode */
+.CodeMirror-overwrite .CodeMirror-cursor {}
+
+.cm-tab { display: inline-block; text-decoration: inherit; }
+
+.CodeMirror-ruler {
+  border-left: 1px solid #ccc;
+  position: absolute;
+}
+
+/* DEFAULT THEME */
+
+.cm-s-default .cm-header {color: blue;}
+.cm-s-default .cm-quote {color: #090;}
+.cm-negative {color: #d44;}
+.cm-positive {color: #292;}
+.cm-header, .cm-strong {font-weight: bold;}
+.cm-em {font-style: italic;}
+.cm-link {text-decoration: underline;}
+.cm-strikethrough {text-decoration: line-through;}
+
+.cm-s-default .cm-keyword {color: #708;}
+.cm-s-default .cm-atom {color: #219;}
+.cm-s-default .cm-number {color: #164;}
+.cm-s-default .cm-def {color: #00f;}
+.cm-s-default .cm-variable,
+.cm-s-default .cm-punctuation,
+.cm-s-default .cm-property,
+.cm-s-default .cm-operator {}
+.cm-s-default .cm-variable-2 {color: #05a;}
+.cm-s-default .cm-variable-3 {color: #085;}
+.cm-s-default .cm-comment {color: #a50;}
+.cm-s-default .cm-string {color: #a11;}
+.cm-s-default .cm-string-2 {color: #f50;}
+.cm-s-default .cm-meta {color: #555;}
+.cm-s-default .cm-qualifier {color: #555;}
+.cm-s-default .cm-builtin {color: #30a;}
+.cm-s-default .cm-bracket {color: #997;}
+.cm-s-default .cm-tag {color: #170;}
+.cm-s-default .cm-attribute {color: #00c;}
+.cm-s-default .cm-hr {color: #999;}
+.cm-s-default .cm-link {color: #00c;}
+
+.cm-s-default .cm-error {color: #f00;}
+.cm-invalidchar {color: #f00;}
+
+.CodeMirror-composing { border-bottom: 2px solid; }
+
+/* Default styles for common addons */
+
+div.CodeMirror span.CodeMirror-matchingbracket {color: #0f0;}
+div.CodeMirror span.CodeMirror-nonmatchingbracket {color: #f22;}
+.CodeMirror-matchingtag { background: rgba(255, 150, 0, .3); }
+.CodeMirror-activeline-background {background: #e8f2ff;}
+
+/* STOP */
+
+/* The rest of this file contains styles related to the mechanics of
+   the editor. You probably shouldn't touch them. */
+
+.CodeMirror {
+  position: relative;
+  overflow: hidden;
+  background: white;
+}
+
+.CodeMirror-scroll {
+  overflow: scroll !important; /* Things will break if this is overridden */
+  /* 30px is the magic margin used to hide the element's real scrollbars */
+  /* See overflow: hidden in .CodeMirror */
+  margin-bottom: -30px; margin-right: -30px;
+  padding-bottom: 30px;
+  height: 100%;
+  outline: none; /* Prevent dragging from highlighting the element */
+  position: relative;
+}
+.CodeMirror-sizer {
+  position: relative;
+  border-right: 30px solid transparent;
+}
+
+/* The fake, visible scrollbars. Used to force redraw during scrolling
+   before actual scrolling happens, thus preventing shaking and
+   flickering artifacts. */
+.CodeMirror-vscrollbar, .CodeMirror-hscrollbar, .CodeMirror-scrollbar-filler, .CodeMirror-gutter-filler {
+  position: absolute;
+  z-index: 6;
+  display: none;
+}
+.CodeMirror-vscrollbar {
+  right: 0; top: 0;
+  overflow-x: hidden;
+  overflow-y: scroll;
+}
+.CodeMirror-hscrollbar {
+  bottom: 0; left: 0;
+  overflow-y: hidden;
+  overflow-x: scroll;
+}
+.CodeMirror-scrollbar-filler {
+  right: 0; bottom: 0;
+}
+.CodeMirror-gutter-filler {
+  left: 0; bottom: 0;
+}
+
+.CodeMirror-gutters {
+  position: absolute; left: 0; top: 0;
+  min-height: 100%;
+  z-index: 3;
+}
+.CodeMirror-gutter {
+  white-space: normal;
+  height: 100%;
+  display: inline-block;
+  vertical-align: top;
+  margin-bottom: -30px;
+  /* Hack to make IE7 behave */
+  *zoom:1;
+  *display:inline;
+}
+.CodeMirror-gutter-wrapper {
+  position: absolute;
+  z-index: 4;
+  background: none !important;
+  border: none !important;
+}
+.CodeMirror-gutter-background {
+  position: absolute;
+  top: 0; bottom: 0;
+  z-index: 4;
+}
+.CodeMirror-gutter-elt {
+  position: absolute;
+  cursor: default;
+  z-index: 4;
+}
+.CodeMirror-gutter-wrapper {
+  -webkit-user-select: none;
+  -moz-user-select: none;
+  user-select: none;
+}
+
+.CodeMirror-lines {
+  cursor: text;
+  min-height: 1px; /* prevents collapsing before first draw */
+}
+.CodeMirror pre {
+  /* Reset some styles that the rest of the page might have set */
+  -moz-border-radius: 0; -webkit-border-radius: 0; border-radius: 0;
+  border-width: 0;
+  background: transparent;
+  font-family: inherit;
+  font-size: inherit;
+  margin: 0;
+  white-space: pre;
+  word-wrap: normal;
+  line-height: inherit;
+  color: inherit;
+  z-index: 2;
+  position: relative;
+  overflow: visible;
+  -webkit-tap-highlight-color: transparent;
+  -webkit-font-variant-ligatures: none;
+  font-variant-ligatures: none;
+}
+.CodeMirror-wrap pre {
+  word-wrap: break-word;
+  white-space: pre-wrap;
+  word-break: normal;
+}
+
+.CodeMirror-linebackground {
+  position: absolute;
+  left: 0; right: 0; top: 0; bottom: 0;
+  z-index: 0;
+}
+
+.CodeMirror-linewidget {
+  position: relative;
+  z-index: 2;
+  overflow: auto;
+}
+
+.CodeMirror-widget {}
+
+.CodeMirror-code {
+  outline: none;
+}
+
+/* Force content-box sizing for the elements where we expect it */
+.CodeMirror-scroll,
+.CodeMirror-sizer,
+.CodeMirror-gutter,
+.CodeMirror-gutters,
+.CodeMirror-linenumber {
+  -moz-box-sizing: content-box;
+  box-sizing: content-box;
+}
+
+.CodeMirror-measure {
+  position: absolute;
+  width: 100%;
+  height: 0;
+  overflow: hidden;
+  visibility: hidden;
+}
+
+.CodeMirror-cursor { position: absolute; }
+.CodeMirror-measure pre { position: static; }
+
+div.CodeMirror-cursors {
+  visibility: hidden;
+  position: relative;
+  z-index: 3;
+}
+div.CodeMirror-dragcursors {
+  visibility: visible;
+}
+
+.CodeMirror-focused div.CodeMirror-cursors {
+  visibility: visible;
+}
+
+.CodeMirror-selected { background: #d9d9d9; }
+.CodeMirror-focused .CodeMirror-selected { background: #d7d4f0; }
+.CodeMirror-crosshair { cursor: crosshair; }
+.CodeMirror-line::selection, .CodeMirror-line > span::selection, .CodeMirror-line > span > span::selection { background: #d7d4f0; }
+.CodeMirror-line::-moz-selection, .CodeMirror-line > span::-moz-selection, .CodeMirror-line > span > span::-moz-selection { background: #d7d4f0; }
+
+.cm-searching {
+  background: #ffa;
+  background: rgba(255, 255, 0, .4);
+}
+
+/* IE7 hack to prevent it from returning funny offsetTops on the spans */
+.CodeMirror span { *vertical-align: text-bottom; }
+
+/* Used to force a border model for a node */
+.cm-force-border { padding-right: .1px; }
+
+@media print {
+  /* Hide the cursor when printing */
+  .CodeMirror div.CodeMirror-cursors {
+    visibility: hidden;
+  }
+}
+
+/* See issue #2901 */
+.cm-tab-wrap-hack:after { content: ''; }
+
+/* Help users use markselection to safely style text background */
+span.CodeMirror-selectedtext { background: none; }

web/p5_export/p5-widget/static/vendor/prism.css

@@ -0,0 +1,123 @@
+/* http://prismjs.com/download.html?themes=prism-okaidia&languages=markup+css+clike+javascript */
+/**
+ * okaidia theme for JavaScript, CSS and HTML
+ * Loosely based on Monokai textmate theme by http://www.monokai.nl/
+ * @author ocodia
+ */
+
+code[class*="language-"],
+pre[class*="language-"] {
+	color: #f8f8f2;
+	background: none;
+	text-shadow: 0 1px rgba(0, 0, 0, 0.3);
+	font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace;
+	text-align: left;
+	white-space: pre;
+	word-spacing: normal;
+	word-break: normal;
+	word-wrap: normal;
+	line-height: 1.5;
+
+	-moz-tab-size: 4;
+	-o-tab-size: 4;
+	tab-size: 4;
+
+	-webkit-hyphens: none;
+	-moz-hyphens: none;
+	-ms-hyphens: none;
+	hyphens: none;
+}
+
+/* Code blocks */
+pre[class*="language-"] {
+	padding: 1em;
+	margin: .5em 0;
+	overflow: auto;
+	border-radius: 0.3em;
+}
+
+:not(pre) > code[class*="language-"],
+pre[class*="language-"] {
+	background: #272822;
+}
+
+/* Inline code */
+:not(pre) > code[class*="language-"] {
+	padding: .1em;
+	border-radius: .3em;
+	white-space: normal;
+}
+
+.token.comment,
+.token.prolog,
+.token.doctype,
+.token.cdata {
+	color: slategray;
+}
+
+.token.punctuation {
+	color: #f8f8f2;
+}
+
+.namespace {
+	opacity: .7;
+}
+
+.token.property,
+.token.tag,
+.token.constant,
+.token.symbol,
+.token.deleted {
+	color: #f92672;
+}
+
+.token.boolean,
+.token.number {
+	color: #ae81ff;
+}
+
+.token.selector,
+.token.attr-name,
+.token.string,
+.token.char,
+.token.builtin,
+.token.inserted {
+	color: #a6e22e;
+}
+
+.token.operator,
+.token.entity,
+.token.url,
+.language-css .token.string,
+.style .token.string,
+.token.variable {
+	color: #f8f8f2;
+}
+
+.token.atrule,
+.token.attr-value,
+.token.function {
+	color: #e6db74;
+}
+
+.token.keyword {
+	color: #66d9ef;
+}
+
+.token.regex,
+.token.important {
+	color: #fd971f;
+}
+
+.token.important,
+.token.bold {
+	font-weight: bold;
+}
+.token.italic {
+	font-style: italic;
+}
+
+.token.entity {
+	cursor: help;
+}
+

web/p5_export/p5-widget/static/vendor/prism.js

@@ -0,0 +1,668 @@
+/* http://prismjs.com/download.html?themes=prism&languages=markup+css+clike+javascript */
+var _self = (typeof window !== 'undefined')
+	? window   // if in browser
+	: (
+		(typeof WorkerGlobalScope !== 'undefined' && self instanceof WorkerGlobalScope)
+		? self // if in worker
+		: {}   // if in node js
+	);
+
+/**
+ * Prism: Lightweight, robust, elegant syntax highlighting
+ * MIT license http://www.opensource.org/licenses/mit-license.php/
+ * @author Lea Verou http://lea.verou.me
+ */
+
+var Prism = (function(){
+
+// Private helper vars
+var lang = /\blang(?:uage)?-(\w+)\b/i;
+var uniqueId = 0;
+
+var _ = _self.Prism = {
+	util: {
+		encode: function (tokens) {
+			if (tokens instanceof Token) {
+				return new Token(tokens.type, _.util.encode(tokens.content), tokens.alias);
+			} else if (_.util.type(tokens) === 'Array') {
+				return tokens.map(_.util.encode);
+			} else {
+				return tokens.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/\u00a0/g, ' ');
+			}
+		},
+
+		type: function (o) {
+			return Object.prototype.toString.call(o).match(/\[object (\w+)\]/)[1];
+		},
+
+		objId: function (obj) {
+			if (!obj['__id']) {
+				Object.defineProperty(obj, '__id', { value: ++uniqueId });
+			}
+			return obj['__id'];
+		},
+
+		// Deep clone a language definition (e.g. to extend it)
+		clone: function (o) {
+			var type = _.util.type(o);
+
+			switch (type) {
+				case 'Object':
+					var clone = {};
+
+					for (var key in o) {
+						if (o.hasOwnProperty(key)) {
+							clone[key] = _.util.clone(o[key]);
+						}
+					}
+
+					return clone;
+
+				case 'Array':
+					// Check for existence for IE8
+					return o.map && o.map(function(v) { return _.util.clone(v); });
+			}
+
+			return o;
+		}
+	},
+
+	languages: {
+		extend: function (id, redef) {
+			var lang = _.util.clone(_.languages[id]);
+
+			for (var key in redef) {
+				lang[key] = redef[key];
+			}
+
+			return lang;
+		},
+
+		/**
+		 * Insert a token before another token in a language literal
+		 * As this needs to recreate the object (we cannot actually insert before keys in object literals),
+		 * we cannot just provide an object, we need anobject and a key.
+		 * @param inside The key (or language id) of the parent
+		 * @param before The key to insert before. If not provided, the function appends instead.
+		 * @param insert Object with the key/value pairs to insert
+		 * @param root The object that contains `inside`. If equal to Prism.languages, it can be omitted.
+		 */
+		insertBefore: function (inside, before, insert, root) {
+			root = root || _.languages;
+			var grammar = root[inside];
+
+			if (arguments.length == 2) {
+				insert = arguments[1];
+
+				for (var newToken in insert) {
+					if (insert.hasOwnProperty(newToken)) {
+						grammar[newToken] = insert[newToken];
+					}
+				}
+
+				return grammar;
+			}
+
+			var ret = {};
+
+			for (var token in grammar) {
+
+				if (grammar.hasOwnProperty(token)) {
+
+					if (token == before) {
+
+						for (var newToken in insert) {
+
+							if (insert.hasOwnProperty(newToken)) {
+								ret[newToken] = insert[newToken];
+							}
+						}
+					}
+
+					ret[token] = grammar[token];
+				}
+			}
+
+			// Update references in other language definitions
+			_.languages.DFS(_.languages, function(key, value) {
+				if (value === root[inside] && key != inside) {
+					this[key] = ret;
+				}
+			});
+
+			return root[inside] = ret;
+		},
+
+		// Traverse a language definition with Depth First Search
+		DFS: function(o, callback, type, visited) {
+			visited = visited || {};
+			for (var i in o) {
+				if (o.hasOwnProperty(i)) {
+					callback.call(o, i, o[i], type || i);
+
+					if (_.util.type(o[i]) === 'Object' && !visited[_.util.objId(o[i])]) {
+						visited[_.util.objId(o[i])] = true;
+						_.languages.DFS(o[i], callback, null, visited);
+					}
+					else if (_.util.type(o[i]) === 'Array' && !visited[_.util.objId(o[i])]) {
+						visited[_.util.objId(o[i])] = true;
+						_.languages.DFS(o[i], callback, i, visited);
+					}
+				}
+			}
+		}
+	},
+	plugins: {},
+
+	highlightAll: function(async, callback) {
+		var env = {
+			callback: callback,
+			selector: 'code[class*="language-"], [class*="language-"] code, code[class*="lang-"], [class*="lang-"] code'
+		};
+
+		_.hooks.run("before-highlightall", env);
+
+		var elements = env.elements || document.querySelectorAll(env.selector);
+
+		for (var i=0, element; element = elements[i++];) {
+			_.highlightElement(element, async === true, env.callback);
+		}
+	},
+
+	highlightElement: function(element, async, callback) {
+		// Find language
+		var language, grammar, parent = element;
+
+		while (parent && !lang.test(parent.className)) {
+			parent = parent.parentNode;
+		}
+
+		if (parent) {
+			language = (parent.className.match(lang) || [,''])[1];
+			grammar = _.languages[language];
+		}
+
+		// Set language on the element, if not present
+		element.className = element.className.replace(lang, '').replace(/\s+/g, ' ') + ' language-' + language;
+
+		// Set language on the parent, for styling
+		parent = element.parentNode;
+
+		if (/pre/i.test(parent.nodeName)) {
+			parent.className = parent.className.replace(lang, '').replace(/\s+/g, ' ') + ' language-' + language;
+		}
+
+		var code = element.textContent;
+
+		var env = {
+			element: element,
+			language: language,
+			grammar: grammar,
+			code: code
+		};
+
+		if (!code || !grammar) {
+			_.hooks.run('complete', env);
+			return;
+		}
+
+		_.hooks.run('before-highlight', env);
+
+		if (async && _self.Worker) {
+			var worker = new Worker(_.filename);
+
+			worker.onmessage = function(evt) {
+				env.highlightedCode = evt.data;
+
+				_.hooks.run('before-insert', env);
+
+				env.element.innerHTML = env.highlightedCode;
+
+				callback && callback.call(env.element);
+				_.hooks.run('after-highlight', env);
+				_.hooks.run('complete', env);
+			};
+
+			worker.postMessage(JSON.stringify({
+				language: env.language,
+				code: env.code,
+				immediateClose: true
+			}));
+		}
+		else {
+			env.highlightedCode = _.highlight(env.code, env.grammar, env.language);
+
+			_.hooks.run('before-insert', env);
+
+			env.element.innerHTML = env.highlightedCode;
+
+			callback && callback.call(element);
+
+			_.hooks.run('after-highlight', env);
+			_.hooks.run('complete', env);
+		}
+	},
+
+	highlight: function (text, grammar, language) {
+		var tokens = _.tokenize(text, grammar);
+		return Token.stringify(_.util.encode(tokens), language);
+	},
+
+	tokenize: function(text, grammar, language) {
+		var Token = _.Token;
+
+		var strarr = [text];
+
+		var rest = grammar.rest;
+
+		if (rest) {
+			for (var token in rest) {
+				grammar[token] = rest[token];
+			}
+
+			delete grammar.rest;
+		}
+
+		tokenloop: for (var token in grammar) {
+			if(!grammar.hasOwnProperty(token) || !grammar[token]) {
+				continue;
+			}
+
+			var patterns = grammar[token];
+			patterns = (_.util.type(patterns) === "Array") ? patterns : [patterns];
+
+			for (var j = 0; j < patterns.length; ++j) {
+				var pattern = patterns[j],
+					inside = pattern.inside,
+					lookbehind = !!pattern.lookbehind,
+					greedy = !!pattern.greedy,
+					lookbehindLength = 0,
+					alias = pattern.alias;
+
+				pattern = pattern.pattern || pattern;
+
+				for (var i=0; i<strarr.length; i++) { // Don’t cache length as it changes during the loop
+
+					var str = strarr[i];
+
+					if (strarr.length > text.length) {
+						// Something went terribly wrong, ABORT, ABORT!
+						break tokenloop;
+					}
+
+					if (str instanceof Token) {
+						continue;
+					}
+
+					pattern.lastIndex = 0;
+
+					var match = pattern.exec(str),
+					    delNum = 1;
+
+					// Greedy patterns can override/remove up to two previously matched tokens
+					if (!match && greedy && i != strarr.length - 1) {
+						// Reconstruct the original text using the next two tokens
+						var nextToken = strarr[i + 1].matchedStr || strarr[i + 1],
+						    combStr = str + nextToken;
+
+						if (i < strarr.length - 2) {
+							combStr += strarr[i + 2].matchedStr || strarr[i + 2];
+						}
+
+						// Try the pattern again on the reconstructed text
+						pattern.lastIndex = 0;
+						match = pattern.exec(combStr);
+						if (!match) {
+							continue;
+						}
+
+						var from = match.index + (lookbehind ? match[1].length : 0);
+						// To be a valid candidate, the new match has to start inside of str
+						if (from >= str.length) {
+							continue;
+						}
+						var to = match.index + match[0].length,
+						    len = str.length + nextToken.length;
+
+						// Number of tokens to delete and replace with the new match
+						delNum = 3;
+
+						if (to <= len) {
+							if (strarr[i + 1].greedy) {
+								continue;
+							}
+							delNum = 2;
+							combStr = combStr.slice(0, len);
+						}
+						str = combStr;
+					}
+
+					if (!match) {
+						continue;
+					}
+
+					if(lookbehind) {
+						lookbehindLength = match[1].length;
+					}
+
+					var from = match.index + lookbehindLength,
+					    match = match[0].slice(lookbehindLength),
+					    to = from + match.length,
+					    before = str.slice(0, from),
+					    after = str.slice(to);
+
+					var args = [i, delNum];
+
+					if (before) {
+						args.push(before);
+					}
+
+					var wrapped = new Token(token, inside? _.tokenize(match, inside) : match, alias, match, greedy);
+
+					args.push(wrapped);
+
+					if (after) {
+						args.push(after);
+					}
+
+					Array.prototype.splice.apply(strarr, args);
+				}
+			}
+		}
+
+		return strarr;
+	},
+
+	hooks: {
+		all: {},
+
+		add: function (name, callback) {
+			var hooks = _.hooks.all;
+
+			hooks[name] = hooks[name] || [];
+
+			hooks[name].push(callback);
+		},
+
+		run: function (name, env) {
+			var callbacks = _.hooks.all[name];
+
+			if (!callbacks || !callbacks.length) {
+				return;
+			}
+
+			for (var i=0, callback; callback = callbacks[i++];) {
+				callback(env);
+			}
+		}
+	}
+};
+
+var Token = _.Token = function(type, content, alias, matchedStr, greedy) {
+	this.type = type;
+	this.content = content;
+	this.alias = alias;
+	// Copy of the full string this token was created from
+	this.matchedStr = matchedStr || null;
+	this.greedy = !!greedy;
+};
+
+Token.stringify = function(o, language, parent) {
+	if (typeof o == 'string') {
+		return o;
+	}
+
+	if (_.util.type(o) === 'Array') {
+		return o.map(function(element) {
+			return Token.stringify(element, language, o);
+		}).join('');
+	}
+
+	var env = {
+		type: o.type,
+		content: Token.stringify(o.content, language, parent),
+		tag: 'span',
+		classes: ['token', o.type],
+		attributes: {},
+		language: language,
+		parent: parent
+	};
+
+	if (env.type == 'comment') {
+		env.attributes['spellcheck'] = 'true';
+	}
+
+	if (o.alias) {
+		var aliases = _.util.type(o.alias) === 'Array' ? o.alias : [o.alias];
+		Array.prototype.push.apply(env.classes, aliases);
+	}
+
+	_.hooks.run('wrap', env);
+
+	var attributes = '';
+
+	for (var name in env.attributes) {
+		attributes += (attributes ? ' ' : '') + name + '="' + (env.attributes[name] || '') + '"';
+	}
+
+	return '<' + env.tag + ' class="' + env.classes.join(' ') + '" ' + attributes + '>' + env.content + '</' + env.tag + '>';
+
+};
+
+if (!_self.document) {
+	if (!_self.addEventListener) {
+		// in Node.js
+		return _self.Prism;
+	}
+ 	// In worker
+	_self.addEventListener('message', function(evt) {
+		var message = JSON.parse(evt.data),
+		    lang = message.language,
+		    code = message.code,
+		    immediateClose = message.immediateClose;
+
+		_self.postMessage(_.highlight(code, _.languages[lang], lang));
+		if (immediateClose) {
+			_self.close();
+		}
+	}, false);
+
+	return _self.Prism;
+}
+
+//Get current script and highlight
+var script = document.currentScript || [].slice.call(document.getElementsByTagName("script")).pop();
+
+if (script) {
+	_.filename = script.src;
+
+	if (document.addEventListener && !script.hasAttribute('data-manual')) {
+		document.addEventListener('DOMContentLoaded', _.highlightAll);
+	}
+}
+
+return _self.Prism;
+
+})();
+
+if (typeof module !== 'undefined' && module.exports) {
+	module.exports = Prism;
+}
+
+// hack for components to work correctly in node.js
+if (typeof global !== 'undefined') {
+	global.Prism = Prism;
+}
+;
+Prism.languages.markup = {
+	'comment': /<!--[\w\W]*?-->/,
+	'prolog': /<\?[\w\W]+?\?>/,
+	'doctype': /<!DOCTYPE[\w\W]+?>/,
+	'cdata': /<!\[CDATA\[[\w\W]*?]]>/i,
+	'tag': {
+		pattern: /<\/?(?!\d)[^\s>\/=.$<]+(?:\s+[^\s>\/=]+(?:=(?:("|')(?:\\\1|\\?(?!\1)[\w\W])*\1|[^\s'">=]+))?)*\s*\/?>/i,
+		inside: {
+			'tag': {
+				pattern: /^<\/?[^\s>\/]+/i,
+				inside: {
+					'punctuation': /^<\/?/,
+					'namespace': /^[^\s>\/:]+:/
+				}
+			},
+			'attr-value': {
+				pattern: /=(?:('|")[\w\W]*?(\1)|[^\s>]+)/i,
+				inside: {
+					'punctuation': /[=>"']/
+				}
+			},
+			'punctuation': /\/?>/,
+			'attr-name': {
+				pattern: /[^\s>\/]+/,
+				inside: {
+					'namespace': /^[^\s>\/:]+:/
+				}
+			}
+
+		}
+	},
+	'entity': /&#?[\da-z]{1,8};/i
+};
+
+// Plugin to make entity title show the real entity, idea by Roman Komarov
+Prism.hooks.add('wrap', function(env) {
+
+	if (env.type === 'entity') {
+		env.attributes['title'] = env.content.replace(/&amp;/, '&');
+	}
+});
+
+Prism.languages.xml = Prism.languages.markup;
+Prism.languages.html = Prism.languages.markup;
+Prism.languages.mathml = Prism.languages.markup;
+Prism.languages.svg = Prism.languages.markup;
+
+Prism.languages.css = {
+	'comment': /\/\*[\w\W]*?\*\//,
+	'atrule': {
+		pattern: /@[\w-]+?.*?(;|(?=\s*\{))/i,
+		inside: {
+			'rule': /@[\w-]+/
+			// See rest below
+		}
+	},
+	'url': /url\((?:(["'])(\\(?:\r\n|[\w\W])|(?!\1)[^\\\r\n])*\1|.*?)\)/i,
+	'selector': /[^\{\}\s][^\{\};]*?(?=\s*\{)/,
+	'string': /("|')(\\(?:\r\n|[\w\W])|(?!\1)[^\\\r\n])*\1/,
+	'property': /(\b|\B)[\w-]+(?=\s*:)/i,
+	'important': /\B!important\b/i,
+	'function': /[-a-z0-9]+(?=\()/i,
+	'punctuation': /[(){};:]/
+};
+
+Prism.languages.css['atrule'].inside.rest = Prism.util.clone(Prism.languages.css);
+
+if (Prism.languages.markup) {
+	Prism.languages.insertBefore('markup', 'tag', {
+		'style': {
+			pattern: /(<style[\w\W]*?>)[\w\W]*?(?=<\/style>)/i,
+			lookbehind: true,
+			inside: Prism.languages.css,
+			alias: 'language-css'
+		}
+	});
+	
+	Prism.languages.insertBefore('inside', 'attr-value', {
+		'style-attr': {
+			pattern: /\s*style=("|').*?\1/i,
+			inside: {
+				'attr-name': {
+					pattern: /^\s*style/i,
+					inside: Prism.languages.markup.tag.inside
+				},
+				'punctuation': /^\s*=\s*['"]|['"]\s*$/,
+				'attr-value': {
+					pattern: /.+/i,
+					inside: Prism.languages.css
+				}
+			},
+			alias: 'language-css'
+		}
+	}, Prism.languages.markup.tag);
+};
+Prism.languages.clike = {
+	'comment': [
+		{
+			pattern: /(^|[^\\])\/\*[\w\W]*?\*\//,
+			lookbehind: true
+		},
+		{
+			pattern: /(^|[^\\:])\/\/.*/,
+			lookbehind: true
+		}
+	],
+	'string': {
+		pattern: /(["'])(\\(?:\r\n|[\s\S])|(?!\1)[^\\\r\n])*\1/,
+		greedy: true
+	},
+	'class-name': {
+		pattern: /((?:\b(?:class|interface|extends|implements|trait|instanceof|new)\s+)|(?:catch\s+\())[a-z0-9_\.\\]+/i,
+		lookbehind: true,
+		inside: {
+			punctuation: /(\.|\\)/
+		}
+	},
+	'keyword': /\b(if|else|while|do|for|return|in|instanceof|function|new|try|throw|catch|finally|null|break|continue)\b/,
+	'boolean': /\b(true|false)\b/,
+	'function': /[a-z0-9_]+(?=\()/i,
+	'number': /\b-?(?:0x[\da-f]+|\d*\.?\d+(?:e[+-]?\d+)?)\b/i,
+	'operator': /--?|\+\+?|!=?=?|<=?|>=?|==?=?|&&?|\|\|?|\?|\*|\/|~|\^|%/,
+	'punctuation': /[{}[\];(),.:]/
+};
+
+Prism.languages.javascript = Prism.languages.extend('clike', {
+	'keyword': /\b(as|async|await|break|case|catch|class|const|continue|debugger|default|delete|do|else|enum|export|extends|finally|for|from|function|get|if|implements|import|in|instanceof|interface|let|new|null|of|package|private|protected|public|return|set|static|super|switch|this|throw|try|typeof|var|void|while|with|yield)\b/,
+	'number': /\b-?(0x[\dA-Fa-f]+|0b[01]+|0o[0-7]+|\d*\.?\d+([Ee][+-]?\d+)?|NaN|Infinity)\b/,
+	// Allow for all non-ASCII characters (See http://stackoverflow.com/a/2008444)
+	'function': /[_$a-zA-Z\xA0-\uFFFF][_$a-zA-Z0-9\xA0-\uFFFF]*(?=\()/i
+});
+
+Prism.languages.insertBefore('javascript', 'keyword', {
+	'regex': {
+		pattern: /(^|[^/])\/(?!\/)(\[.+?]|\\.|[^/\\\r\n])+\/[gimyu]{0,5}(?=\s*($|[\r\n,.;})]))/,
+		lookbehind: true,
+		greedy: true
+	}
+});
+
+Prism.languages.insertBefore('javascript', 'class-name', {
+	'template-string': {
+		pattern: /`(?:\\\\|\\?[^\\])*?`/,
+		greedy: true,
+		inside: {
+			'interpolation': {
+				pattern: /\$\{[^}]+\}/,
+				inside: {
+					'interpolation-punctuation': {
+						pattern: /^\$\{|\}$/,
+						alias: 'punctuation'
+					},
+					rest: Prism.languages.javascript
+				}
+			},
+			'string': /[\s\S]+/
+		}
+	}
+});
+
+if (Prism.languages.markup) {
+	Prism.languages.insertBefore('markup', 'tag', {
+		'script': {
+			pattern: /(<script[\w\W]*?>)[\w\W]*?(?=<\/script>)/i,
+			lookbehind: true,
+			inside: Prism.languages.javascript,
+			alias: 'language-javascript'
+		}
+	});
+}
+
+Prism.languages.js = Prism.languages.javascript;

web/p5_export/p5.html

@@ -0,0 +1,105 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>Document</title>
+
+
+  <style>
+    body {
+      height: 600px;
+      padding: 24px;
+    }
+
+    iframe {
+      min-height: 600px !important
+    }
+
+    /* 自定义滚动条样式 */
+    ::-webkit-scrollbar {
+      width: 8px;
+      /* 滚动条宽度 */
+    }
+
+    ::-webkit-scrollbar-track {
+      background: #f1f1f1;
+      /* 滚动条轨道颜色 */
+    }
+
+    ::-webkit-scrollbar-thumb {
+      background: #888;
+      /* 滚动条滑块颜色 */
+    }
+
+    ::-webkit-scrollbar-thumb:hover {
+      background: #555;
+      /* 滚动条滑块悬停颜色 */
+    }
+  </style>
+  <!-- <script src="./p5.js"></script> -->
+  <!-- <script src="./sketch.js"></script> -->
+  <!-- <script src="./src/CCapture.js"></script> -->
+  <!-- <script src="./src/gif.js"></script>
+   <script src="./src/gif.worker.js"></script> -->
+  <!-- <script src="./src/download.js"></script> -->
+  <!-- <script src="./src/webm-writer-0.2.0.js"></script> -->
+
+</head>
+
+<body>
+
+  <script>
+
+    function getIdFromUrl(url) {
+      const urlParams = new URLSearchParams(new URL(url).search);
+      return urlParams.get('id');
+    }
+
+
+
+    // 监听来自iframe的消息
+    window.addEventListener('message', (event) => {
+      const data = event.data;
+      const nodeId = getIdFromUrl(window.location.href);
+
+      if (data.from === 'p5.widget' && data.status === 'save') {
+        const frames = data.frames;
+        // 示例用法
+        // const url = 'https://example.com/page?id=12345';
+        window.parent.postMessage({
+          frames,
+          from: 'p5.widget',
+          status: 'save',
+          nodeId
+        }, '*');
+        window.location.reload()
+      }
+      // console.log(data)
+      // if (data.from === 'p5.widget' && data.status === 'capture') {
+      //   window.parent.postMessage({
+      //     from: 'p5.widget',
+      //     status: 'capture',
+      //     frameCount: data.frameCount,
+      //     maxCount: data.maxCount,
+      //     nodeId
+      //   }, '*');
+      // }
+    });
+
+
+  </script>
+  <script type="text/p5" data-height="500" data-preview-width="300">
+function setup() {
+createCanvas(100, 100);
+}
+    
+function draw() {
+background(255, 0, 200);
+}
+    </script>
+  <script src="./p5-widget/p5-widget.js"></script>
+</body>
+
+</html>

web/p5_export/sketch.js

@@ -0,0 +1,70 @@
+function setup () {
+  createCanvas(400, 400, WEBGL)
+  angleMode(DEGREES)
+}
+
+function draw () {
+  if (frameCount === 1) {
+    capturer.start()
+  }
+
+  background(30)
+  noStroke()
+  translate(0, 0, sin(frameCount) * 400 - 800)
+
+  rotateX(frameCount)
+  rotateY(frameCount)
+  rotateZ(frameCount)
+
+  var w = 20
+
+  randomSeed(1)
+  for (var x = -width / 2; x <= width / 2; x += w) {
+    for (var y = -width / 2; y <= width / 2; y += w) {
+      for (var z = -width / 2; z <= width / 2; z += w) {
+        var r = random(255)
+        var g = random(255)
+        var b = random(255)
+
+        fill(r, g, b)
+
+        push()
+        translate(x, y, z)
+        box(w)
+        pop()
+      }
+    }
+  }
+  // console.log(frameRate());
+
+  if (frameCount < 60) {
+    capturer.capture(canvas)
+  } else if (frameCount === 60) {
+    capturer.save(function (blob) {
+        // console.log(blob)
+
+        // 示例用法
+        // const blob = new Blob([/* 数据 */], { type: 'video/webm' });
+        blobToBase64(blob).then(base64String => {
+            console.log(base64String);
+         
+            const video = document.createElement('video');
+            video.controls = true; // 显示视频控件（播放、暂停等）
+            video.src = base64String; // 设置视频的 src 属性为 Base64 数据 URL
+            video.width = 640; // 设置视频宽度
+            video.height = 360; // 设置视频高度
+
+            // 将 video 元素添加到页面中
+            document.body.appendChild(video)
+
+            // 自动播放视频
+            video.play();
+
+        }).catch(error => {
+            console.error('转换失败:', error);
+        });
+
+    })
+    capturer.stop()
+  }
+}

workflow/LLM_base_workflow.json

@@ -0,0 +1,408 @@
+{
+  "last_node_id": 21,
+  "last_link_id": 16,
+  "nodes": [
+    {
+      "id": 10,
+      "type": "ChatGPTOpenAI",
+      "pos": [
+        489,
+        689
+      ],
+      "size": {
+        "0": 403.2580261230469,
+        "1": 309.2166442871094
+      },
+      "flags": {},
+      "order": 5,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "api_key",
+          "type": "STRING",
+          "link": null,
+          "widget": {
+            "name": "api_key"
+          }
+        },
+        {
+          "name": "custom_model_name",
+          "type": "STRING",
+          "link": null,
+          "widget": {
+            "name": "custom_model_name"
+          }
+        },
+        {
+          "name": "custom_api_url",
+          "type": "STRING",
+          "link": 13,
+          "widget": {
+            "name": "custom_api_url"
+          },
+          "slot_index": 2
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": [
+            12
+          ],
+          "shape": 3,
+          "slot_index": 0
+        },
+        {
+          "name": "messages",
+          "type": "STRING",
+          "links": null,
+          "shape": 3
+        },
+        {
+          "name": "session_history",
+          "type": "STRING",
+          "links": null,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "ChatGPTOpenAI"
+      },
+      "widgets_values": [
+        "hi",
+        "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.",
+        "gpt-3.5-turbo",
+        447210757728856,
+        "randomize",
+        1,
+        "openai",
+        "",
+        "",
+        ""
+      ]
+    },
+    {
+      "id": 3,
+      "type": "ShowTextForGPT",
+      "pos": [
+        982,
+        686
+      ],
+      "size": {
+        "0": 400,
+        "1": 200
+      },
+      "flags": {},
+      "order": 7,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "link": 12,
+          "widget": {
+            "name": "text"
+          }
+        },
+        {
+          "name": "output_dir",
+          "type": "STRING",
+          "link": null,
+          "widget": {
+            "name": "output_dir"
+          }
+        }
+      ],
+      "outputs": [
+        {
+          "name": "STRING",
+          "type": "STRING",
+          "links": null,
+          "shape": 6
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "ShowTextForGPT"
+      },
+      "widgets_values": [
+        "",
+        "",
+        " Hi there! What can I help you with?"
+      ]
+    },
+    {
+      "id": 11,
+      "type": "SiliconflowLLM",
+      "pos": [
+        489,
+        318
+      ],
+      "size": {
+        "0": 395.197998046875,
+        "1": 262
+      },
+      "flags": {},
+      "order": 4,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "api_key",
+          "type": "STRING",
+          "link": 16,
+          "widget": {
+            "name": "api_key"
+          }
+        },
+        {
+          "name": "custom_model_name",
+          "type": "STRING",
+          "link": null,
+          "widget": {
+            "name": "custom_model_name"
+          }
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": [
+            15
+          ],
+          "shape": 3,
+          "slot_index": 0
+        },
+        {
+          "name": "messages",
+          "type": "STRING",
+          "links": null,
+          "shape": 3
+        },
+        {
+          "name": "session_history",
+          "type": "STRING",
+          "links": null,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "SiliconflowLLM"
+      },
+      "widgets_values": [
+        "",
+        "",
+        "You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible.",
+        "Qwen/Qwen2-7B-Instruct",
+        593422808835285,
+        "randomize",
+        1,
+        ""
+      ]
+    },
+    {
+      "id": 17,
+      "type": "ShowTextForGPT",
+      "pos": [
+        975,
+        329
+      ],
+      "size": {
+        "0": 400,
+        "1": 200
+      },
+      "flags": {},
+      "order": 6,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "link": 15,
+          "widget": {
+            "name": "text"
+          }
+        },
+        {
+          "name": "output_dir",
+          "type": "STRING",
+          "link": null,
+          "widget": {
+            "name": "output_dir"
+          }
+        }
+      ],
+      "outputs": [
+        {
+          "name": "STRING",
+          "type": "STRING",
+          "links": null,
+          "shape": 6
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "ShowTextForGPT"
+      },
+      "widgets_values": [
+        "",
+        "",
+        "Hello! How can I assist you today?"
+      ]
+    },
+    {
+      "id": 18,
+      "type": "KeyInput",
+      "pos": [
+        46,
+        319
+      ],
+      "size": {
+        "0": 315,
+        "1": 70
+      },
+      "flags": {},
+      "order": 0,
+      "mode": 0,
+      "outputs": [
+        {
+          "name": "key",
+          "type": "STRING",
+          "links": [
+            16
+          ],
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "KeyInput"
+      },
+      "widgets_values": [
+        null,
+        null
+      ]
+    },
+    {
+      "id": 12,
+      "type": "TextInput_",
+      "pos": [
+        31,
+        871
+      ],
+      "size": [
+        407.9377612789413,
+        76
+      ],
+      "flags": {},
+      "order": 1,
+      "mode": 0,
+      "outputs": [
+        {
+          "name": "STRING",
+          "type": "STRING",
+          "links": [
+            13
+          ],
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "TextInput_"
+      },
+      "widgets_values": [
+        "http://127.0.0.1:8000/v1"
+      ]
+    },
+    {
+      "id": 20,
+      "type": "Note",
+      "pos": [
+        35,
+        664
+      ],
+      "size": [
+        350.04604707424306,
+        116.54209784249178
+      ],
+      "flags": {},
+      "order": 2,
+      "mode": 0,
+      "properties": {
+        "text": ""
+      },
+      "widgets_values": [
+        "api_key 填写对应平台的Key\ncustom model和api 根据需要自行填写\n\n如果不填写custome，则按照model和api_url选择的选项"
+      ],
+      "color": "#432",
+      "bgcolor": "#653"
+    },
+    {
+      "id": 21,
+      "type": "Note",
+      "pos": [
+        42,
+        438
+      ],
+      "size": {
+        "0": 350.0460510253906,
+        "1": 116.54209899902344
+      },
+      "flags": {},
+      "order": 3,
+      "mode": 0,
+      "properties": {
+        "text": ""
+      },
+      "widgets_values": [
+        "API key节点不会保存到workflow的json文件。\n\n：：会保存到appinfo导出的app.json里\n\n\n注册https://cloud.siliconflow.cn/account/ak 领取免费的API"
+      ],
+      "color": "#432",
+      "bgcolor": "#653"
+    }
+  ],
+  "links": [
+    [
+      12,
+      10,
+      0,
+      3,
+      0,
+      "STRING"
+    ],
+    [
+      13,
+      12,
+      0,
+      10,
+      2,
+      "STRING"
+    ],
+    [
+      15,
+      11,
+      0,
+      17,
+      0,
+      "STRING"
+    ],
+    [
+      16,
+      18,
+      0,
+      11,
+      0,
+      "STRING"
+    ]
+  ],
+  "groups": [],
+  "config": {},
+  "extra": {
+    "ds": {
+      "scale": 0.9646149645000006,
+      "offset": [
+        170.81398913081276,
+        -128.0066534315481
+      ]
+    }
+  },
+  "version": 0.4
+}

workflow/text-to-audio-base-workflow.json

@@ -0,0 +1,415 @@
+{
+  "last_node_id": 20,
+  "last_link_id": 28,
+  "nodes": [
+    {
+      "id": 11,
+      "type": "EmptyLatentAudio",
+      "pos": [
+        576,
+        480
+      ],
+      "size": {
+        "0": 240,
+        "1": 58
+      },
+      "flags": {},
+      "order": 0,
+      "mode": 0,
+      "outputs": [
+        {
+          "name": "LATENT",
+          "type": "LATENT",
+          "links": [
+            12
+          ],
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "EmptyLatentAudio"
+      },
+      "widgets_values": [
+        47.6
+      ]
+    },
+    {
+      "id": 7,
+      "type": "CLIPTextEncode",
+      "pos": [
+        384,
+        288
+      ],
+      "size": {
+        "0": 432,
+        "1": 144
+      },
+      "flags": {},
+      "order": 4,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "clip",
+          "type": "CLIP",
+          "link": 26
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            6
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "CLIPTextEncode"
+      },
+      "widgets_values": [
+        ""
+      ],
+      "color": "#322",
+      "bgcolor": "#533"
+    },
+    {
+      "id": 3,
+      "type": "KSampler",
+      "pos": [
+        864,
+        96
+      ],
+      "size": {
+        "0": 315,
+        "1": 262
+      },
+      "flags": {},
+      "order": 5,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "model",
+          "type": "MODEL",
+          "link": 18
+        },
+        {
+          "name": "positive",
+          "type": "CONDITIONING",
+          "link": 4
+        },
+        {
+          "name": "negative",
+          "type": "CONDITIONING",
+          "link": 6
+        },
+        {
+          "name": "latent_image",
+          "type": "LATENT",
+          "link": 12,
+          "slot_index": 3
+        }
+      ],
+      "outputs": [
+        {
+          "name": "LATENT",
+          "type": "LATENT",
+          "links": [
+            13
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "KSampler"
+      },
+      "widgets_values": [
+        173246216057483,
+        "randomize",
+        50,
+        4.98,
+        "dpmpp_3m_sde_gpu",
+        "exponential",
+        1
+      ]
+    },
+    {
+      "id": 6,
+      "type": "CLIPTextEncode",
+      "pos": [
+        384,
+        96
+      ],
+      "size": {
+        "0": 432,
+        "1": 144
+      },
+      "flags": {},
+      "order": 3,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "clip",
+          "type": "CLIP",
+          "link": 25
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            4
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "CLIPTextEncode"
+      },
+      "widgets_values": [
+        "heaven church electronic dance music"
+      ],
+      "color": "#232",
+      "bgcolor": "#353"
+    },
+    {
+      "id": 12,
+      "type": "VAEDecodeAudio",
+      "pos": [
+        1200,
+        96
+      ],
+      "size": {
+        "0": 210,
+        "1": 46
+      },
+      "flags": {},
+      "order": 6,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "samples",
+          "type": "LATENT",
+          "link": 13
+        },
+        {
+          "name": "vae",
+          "type": "VAE",
+          "link": 14,
+          "slot_index": 1
+        }
+      ],
+      "outputs": [
+        {
+          "name": "AUDIO",
+          "type": "AUDIO",
+          "links": [
+            28
+          ],
+          "shape": 3,
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "VAEDecodeAudio"
+      }
+    },
+    {
+      "id": 19,
+      "type": "AudioPlay",
+      "pos": [
+        1482,
+        96
+      ],
+      "size": [
+        210,
+        280
+      ],
+      "flags": {},
+      "order": 7,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "audio",
+          "type": "AUDIO",
+          "link": 28
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "AudioPlay"
+      },
+      "widgets_values": [
+        [
+          "/view?filename=._00001_.wav&type=temp&subfolder=&rand=0.5323617364976663",
+          null
+        ]
+      ]
+    },
+    {
+      "id": 4,
+      "type": "CheckpointLoaderSimple",
+      "pos": [
+        -56,
+        304
+      ],
+      "size": [
+        390.12250902572447,
+        98
+      ],
+      "flags": {},
+      "order": 1,
+      "mode": 0,
+      "outputs": [
+        {
+          "name": "MODEL",
+          "type": "MODEL",
+          "links": [
+            18
+          ],
+          "slot_index": 0
+        },
+        {
+          "name": "CLIP",
+          "type": "CLIP",
+          "links": [],
+          "slot_index": 1
+        },
+        {
+          "name": "VAE",
+          "type": "VAE",
+          "links": [
+            14
+          ],
+          "slot_index": 2
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "CheckpointLoaderSimple"
+      },
+      "widgets_values": [
+        "stable_audio_open_1.0.safetensors"
+      ]
+    },
+    {
+      "id": 10,
+      "type": "CLIPLoader",
+      "pos": [
+        -18,
+        93
+      ],
+      "size": {
+        "0": 335.6534118652344,
+        "1": 82
+      },
+      "flags": {},
+      "order": 2,
+      "mode": 0,
+      "outputs": [
+        {
+          "name": "CLIP",
+          "type": "CLIP",
+          "links": [
+            25,
+            26
+          ],
+          "shape": 3,
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "CLIPLoader"
+      },
+      "widgets_values": [
+        "t5_base.safetensors",
+        "stable_audio"
+      ]
+    }
+  ],
+  "links": [
+    [
+      4,
+      6,
+      0,
+      3,
+      1,
+      "CONDITIONING"
+    ],
+    [
+      6,
+      7,
+      0,
+      3,
+      2,
+      "CONDITIONING"
+    ],
+    [
+      12,
+      11,
+      0,
+      3,
+      3,
+      "LATENT"
+    ],
+    [
+      13,
+      3,
+      0,
+      12,
+      0,
+      "LATENT"
+    ],
+    [
+      14,
+      4,
+      2,
+      12,
+      1,
+      "VAE"
+    ],
+    [
+      18,
+      4,
+      0,
+      3,
+      0,
+      "MODEL"
+    ],
+    [
+      25,
+      10,
+      0,
+      6,
+      0,
+      "CLIP"
+    ],
+    [
+      26,
+      10,
+      0,
+      7,
+      0,
+      "CLIP"
+    ],
+    [
+      28,
+      12,
+      0,
+      19,
+      0,
+      "AUDIO"
+    ]
+  ],
+  "groups": [],
+  "config": {},
+  "extra": {
+    "ds": {
+      "scale": 0.7972024500000005,
+      "offset": {
+        "0": 206.23990767458133,
+        "1": 402.8380652485712
+      }
+    }
+  },
+  "version": 0.4
+}