update fast api

README.md

@@ -24,7 +24,7 @@ EasyAnimate is a pipeline based on the transformer architecture that can be used
 We will support quick pull-ups from different platforms, refer to [Quick Start](#quick-start).
 
 What's New:
-- Updated to v2 version, supports a maximum of 144 frames (6s, 24fps) for generation. [ 2024.05.26 ]
+- Updated to v2 version, supports a maximum of 144 frames (768x768, 6s, 24fps) for generation. [ 2024.05.26 ]
 - Create Code! Support for Windows and Linux Now. [ 2024.04.12 ]
 
 These are our generated results:
@@ -59,7 +59,7 @@ EasyAnimateV2:
 |--|--|--|--|--| 
 | PixArt-XL-2-512x512.tar | Pixart | 11.4GB | [download](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/Diffusion_Transformer/PixArt-XL-2-512x512.tar)| Pixart-Alpha official weights |
 | easyanimate_portrait.safetensors | Checkpoint of Pixart | 2.3GB | [download](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/Personalized_Model/easyanimate_portrait.safetensors) | Training with internal portrait datasets |
-| easyanimate_portrait_lora.safetensors | Lora of Pixart | 654.0MB | [download](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/Personalized_Model/easyanimate_portrait_lora.safetensors)| Training with internal portrait datasets |
+| easyanimate_portrait_lora.safetensors | Lora of Pixart | 485.11MB | [download](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/Personalized_Model/easyanimate_portrait_lora.safetensors)| Training with internal portrait datasets |
 </details>
 
 # Result Gallery
@@ -186,6 +186,8 @@ EasyAnimateV2:
 - Step 3: Select the generated model based on the page, fill in prompt, neg_prompt, guidance_scale, and seed, click on generate, wait for the generated result, and save the result in the samples folder.
 
 ### 2. Model Training
+We have provided a simple demo of training the Lora model through image data, which can be found in the [wiki](https://github.com/aigc-apps/EasyAnimate/wiki/Training-Lora) for details.
+
 If you want to train a text to image and video generation model. You need to arrange the dataset in this format.
 
 ```
@@ -251,15 +253,35 @@ sh scripts/train_t2iv.sh
 ```
 
 # Algorithm Detailed
-We build EasyAnimate by introducing additional motion module upon [PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha),so that can extend the DiT model from 2D image generation to 3D video generation. The pipeline is shwon as follows.
+### 1. Data Preprocessing
+**Video Cut**
 
-<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/pipeline.png" alt="ui" style="zoom:50%;" />
+For long video cut, EasyAnimate utilizes PySceneDetect to identify scene changes within the video and performs scene cutting based on certain threshold values to ensure consistency in the themes of the video segments. After cutting, we only keep segments with lengths ranging from 3 to 10 seconds for model training.
 
-The motion module is used to capture the temporal information among frames. The structure is shown as follows.
+**Video Cleaning and Description**
 
-<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/motion_module.png" alt="motion" style="zoom:50%;" />
+Following SVD's data preparation process, EasyAnimate provides a simple yet effective data processing pipeline for high-quality data filtering and labeling. It also supports distributed processing to accelerate the speed of data preprocessing. The overall process is as follows:
 
-We introduce attention mechanisms in the temporal dimension to enable the model to learn temporal information for generating continuous video frames. At the same time, we utilize an additional Grid Reshape calculation to expand the number of input tokens for the attention mechanism, thus making greater use of the spatial information in images to achieve better generative results.
+- Duration filtering: Analyze the basic information of the video to filter out low-quality videos that are short in duration or low in resolution.
+- Aesthetic filtering: Filter out videos with poor content (blurry, dim, etc.) by calculating the average aesthetic score of uniformly distributed 4 frames.
+- Text filtering: Use easyocr to calculate the text proportion of middle frames to filter out videos with a large proportion of text.
+- Motion filtering: Calculate interframe optical flow differences to filter out videos that move too slowly or too quickly.
+- Text description: Recaption video frames using videochat2 and vila. PAI is also developing a higher quality video recaption model, which will be released for use as soon as possible.
+
+### 2. Model Architecture
+We have adopted [PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha) as the base model and modified the VAE and DiT model structures on this basis to better support video generation. The overall structure of EasyAnimate is as follows:
+
+The diagram below outlines the pipeline of EasyAnimate. It includes the Text Encoder, Video VAE (video encoder and decoder), and Diffusion Transformer (DiT). The T5 Encoder is used as the text encoder. Other components are detailed in the sections below.
+
+<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/pipeline_v2.jpg" alt="ui" style="zoom:50%;" />
+
+To introduce feature information along the temporal axis, EasyAnimate incorporates the Motion Module to achieve the expansion from 2D images to 3D videos. For better generation effects, it jointly finetunes the Backbone together with the Motion Module, thereby achieving image generation and video generation within a single Pipeline.
+
+Additionally, referencing U-ViT, it introduces a skip connection structure into EasyAnimate to further optimize deeper features by incorporating shallow features. A fully connected layer is also zero-initialized for each skip connection structure, allowing it to be applied as a plug-in module to previously trained and well-performing DiTs.
+
+Moreover, it proposes Slice VAE, which addresses the memory difficulties encountered by MagViT when dealing with long and large videos, while also achieving greater compression in the temporal dimension during video encoding and decoding stages compared to MagViT.
+
+For more details, please refer to [arxiv]().
 
 # Reference
 - magvit: https://github.com/google-research/magvit

README_zh-CN.md

@@ -25,7 +25,7 @@ EasyAnimate是一个基于transformer结构的pipeline，可用于生成AI图片
 我们会逐渐支持从不同平台快速启动，请参阅 [快速启动](#快速启动)。
 
 新特性：
-- 更新到v2版本，最大支持144帧(6s, 24fps)生成。[ 2024.05.26 ]
+- 更新到v2版本，最大支持144帧(768x768, 6s, 24fps)生成。[ 2024.05.26 ]
 - 创建代码！现在支持 Windows 和 Linux。[ 2024.04.12 ]
 
 这些是我们的生成结果:
@@ -186,6 +186,8 @@ EasyAnimateV2:
 - 步骤3：根据页面选择生成模型，填入prompt、neg_prompt、guidance_scale和seed等，点击生成，等待生成结果，结果保存在sample文件夹中。
 
 ### 2. 模型训练
+我们给出了一个简单的demo通过图片数据训练lora模型，详情可以查看[wiki](https://github.com/aigc-apps/EasyAnimate/wiki/Training-Lora)。
+
 如果期望训练一个文生图视频的生成模型，您需要以这种格式排列数据集。
 ```
 📦 project/
@@ -254,15 +256,35 @@ sh scripts/train_t2iv.sh
 ```
 
 # 算法细节
-我们使用了[PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha)作为基础模型，并在此基础上引入额外的运动模块（motion module）来将DiT模型从2D图像生成扩展到3D视频生成上来。其框架图如下：
+### 1. 数据预处理
+**视频分割**
 
-<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/pipeline.png" alt="ui" style="zoom:50%;" />
+对于较长的视频分割，EasyAnimate使用PySceneDetect以识别视频内的场景变化并基于这些转换，根据一定的门限值来执行场景剪切，以确保视频片段的主题一致性。切割后，我们只保留长度在3到10秒之间的片段用于模型训练。
 
-其中，Motion Module 用于捕捉时序维度的帧间关系，其结构如下：
+**视频清洗与描述**
 
-<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/motion_module.png" alt="motion" style="zoom:50%;" />
+参考SVD的数据准备流程，EasyAnimate提供了一条简单但有效的数据处理链路来进行高质量的数据筛选与打标。并且支持了分布式处理来提升数据预处理的速度，其整体流程如下：
 
-我们在时序维度上引入注意力机制来让模型学习时序信息，以进行连续视频帧的生成。同时，我们利用额外的网格计算（Grid Reshape），来扩大注意力机制的input token数目，从而更多地利用图像的空间信息以达到更好的生成效果。Motion Module 作为一个单独的模块，在推理时可以用在不同的DiT基线模型上。此外，EasyAnimate不仅支持了motion-module模块的训练，也支持了DiT基模型/LoRA模型的训练，以方便用户根据自身需要来完成自定义风格的模型训练，进而生成任意风格的视频。
+- 时长过滤： 统计视频基本信息，来过滤时间短/分辨率低的低质量视频
+- 美学过滤： 通过计算视频均匀4帧的美学得分均值，来过滤内容较差的视频（模糊、昏暗等）
+- 文本过滤： 通过easyocr计算中间帧的文本占比，来过滤文本占比过大的视频
+- 运动过滤： 计算帧间光流差异来过滤运动过慢或过快的视频。
+- 文本描述： 通过videochat2和vila对视频帧进行recaption。PAI也在自研质量更高的视频recaption模型，将在第一时间放出供大家使用。
+
+### 2. 模型结构
+我们使用了[PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha)作为基础模型，并在此基础上修改了VAE和DiT的模型结构来更好地支持视频的生成。EasyAnimate的整体结构如下：
+
+下图概述了EasyAnimate的管道。它包括Text Encoder、Video VAE（视频编码器和视频解码器）和Diffusion Transformer（DiT）。T5 Encoder用作文本编码器。其他组件将在以下部分中详细说明。
+
+<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/pipeline_v2.jpg" alt="ui" style="zoom:50%;" />
+
+为了引入特征点在时间轴上的特征信息，EasyAnimate引入了运动模块（Motion Module），以实现从2D图像到3D视频的扩展。为了更好的生成效果，其联合图片和视频将Backbone连同Motion Module一起Finetune。在一个Pipeline中即实现了图片的生成，也实现了视频的生成。
+
+另外，参考U-ViT，其将跳连接结构引入到EasyAnimate当中，通过引入浅层特征进一步优化深层特征，并且0初始化了一个全连接层给每一个跳连接结构，使其可以作为一个可插入模块应用到之前已经训练的还不错的DIT中。
+
+同时，其提出了Slice VAE，用于解决MagViT在面对长、大视频时编解码上的显存困难，同时相比于MagViT在视频编解码阶段进行了时间维度更大的压缩。
+
+更多细节可以看查看[arxiv]()。
 
 # 参考文献
 - magvit: https://github.com/google-research/magvit

app.py

@@ -1,3 +1,6 @@
+import time 
+
+from easyanimate.api.api import infer_forward_api, update_diffusion_transformer_api, update_edition_api
 from easyanimate.ui.ui import ui_modelscope, ui
 
 if __name__ == "__main__":
@@ -5,6 +8,7 @@ if __name__ == "__main__":
     ui_mode = "normal"
     # Server ip
     server_name = "0.0.0.0"
+    server_port = 7860
 
     # Params below is used when ui_mode = "modelscope"
     edition = "v2"
@@ -13,7 +17,22 @@ if __name__ == "__main__":
     savedir_sample = "samples"
 
     if ui_mode != "modelscope":
-        demo = ui()
+        demo, controller = ui()
     else:
-        demo = ui_modelscope(edition, config_path, model_name, savedir_sample)
-    demo.launch(server_name=server_name)
+        demo, controller = ui_modelscope(edition, config_path, model_name, savedir_sample)
+
+    # launch gradio
+    app, _, _ = demo.queue(status_update_rate=1).launch(
+        server_name=server_name,
+        server_port=server_port,
+        prevent_thread_lock=True
+    )
+    
+    # launch api
+    infer_forward_api(None, app, controller)
+    update_diffusion_transformer_api(None, app, controller)
+    update_edition_api(None, app, controller)
+    
+    # not close the python
+    while True:
+        time.sleep(5)

easyanimate/api/api.py

@@ -0,0 +1,96 @@
+import io
+import base64
+import torch
+import gradio as gr
+
+from fastapi import FastAPI
+from io import BytesIO
+
+# Function to encode a file to Base64
+def encode_file_to_base64(file_path):
+    with open(file_path, "rb") as file:
+        # Encode the data to Base64
+        file_base64 = base64.b64encode(file.read())
+        return file_base64
+
+def update_edition_api(_: gr.Blocks, app: FastAPI, controller):
+    @app.post("/easyanimate/update_edition")
+    def _update_edition_api(
+        datas: dict,
+    ):
+        edition = datas.get('edition', 'v2')
+
+        try:
+            controller.update_edition(
+                edition
+            )
+            comment = "Success"
+        except Exception as e:
+            torch.cuda.empty_cache()
+            comment = f"Error. error information is {str(e)}"
+
+        return {"message": comment}
+
+def update_diffusion_transformer_api(_: gr.Blocks, app: FastAPI, controller):
+    @app.post("/easyanimate/update_diffusion_transformer")
+    def _update_diffusion_transformer_api(
+        datas: dict,
+    ):
+        diffusion_transformer_path = datas.get('diffusion_transformer_path', 'none')
+
+        try:
+            controller.update_diffusion_transformer(
+                diffusion_transformer_path
+            )
+            comment = "Success"
+        except Exception as e:
+            torch.cuda.empty_cache()
+            comment = f"Error. error information is {str(e)}"
+
+        return {"message": comment}
+
+def infer_forward_api(_: gr.Blocks, app: FastAPI, controller):
+    @app.post("/easyanimate/infer_forward")
+    def _infer_forward_api(
+        datas: dict,
+    ):
+        base_model_path = datas.get('base_model_path', 'none')
+        motion_module_path = datas.get('motion_module_path', 'none')
+        lora_model_path = datas.get('lora_model_path', 'none')
+        lora_alpha_slider = datas.get('lora_alpha_slider', 0.55)
+        prompt_textbox = datas.get('prompt_textbox', None)
+        negative_prompt_textbox = datas.get('negative_prompt_textbox', '')
+        sampler_dropdown = datas.get('sampler_dropdown', 'Euler')
+        sample_step_slider = datas.get('sample_step_slider', 30)
+        width_slider = datas.get('width_slider', 672)
+        height_slider = datas.get('height_slider', 384)
+        is_image = datas.get('is_image', False)
+        length_slider = datas.get('length_slider', 144)
+        cfg_scale_slider = datas.get('cfg_scale_slider', 6)
+        seed_textbox = datas.get("seed_textbox", 43)
+
+        try:
+            save_sample_path, comment = controller.generate(
+                "",
+                base_model_path,
+                motion_module_path,
+                lora_model_path, 
+                lora_alpha_slider,
+                prompt_textbox, 
+                negative_prompt_textbox, 
+                sampler_dropdown, 
+                sample_step_slider, 
+                width_slider, 
+                height_slider, 
+                is_image,
+                length_slider, 
+                cfg_scale_slider, 
+                seed_textbox,
+                is_api = True,
+            )
+        except Exception as e:
+            torch.cuda.empty_cache()
+            save_sample_path = ""
+            comment = f"Error. error information is {str(e)}"
+
+        return {"message": comment, "save_sample_path": save_sample_path, "base64_encoding": encode_file_to_base64(save_sample_path)}

easyanimate/api/post_infer.py

@@ -0,0 +1,94 @@
+import base64
+import json
+import sys
+import time
+from datetime import datetime
+from io import BytesIO
+
+import cv2
+import requests
+import base64
+
+
+def post_diffusion_transformer(diffusion_transformer_path, url='http://127.0.0.1:7860'):
+    datas = json.dumps({
+        "diffusion_transformer_path": diffusion_transformer_path
+    })
+    r = requests.post(f'{url}/easyanimate/update_diffusion_transformer', data=datas, timeout=1500)
+    data = r.content.decode('utf-8')
+    return data
+
+def post_update_edition(edition, url='http://0.0.0.0:7860'):
+    datas = json.dumps({
+        "edition": edition
+    })
+    r = requests.post(f'{url}/easyanimate/update_edition', data=datas, timeout=1500)
+    data = r.content.decode('utf-8')
+    return data
+
+def post_infer(is_image, length_slider, url='http://127.0.0.1:7860'):
+    datas = json.dumps({
+        "base_model_path": "none",
+        "motion_module_path": "none",
+        "lora_model_path": "none", 
+        "lora_alpha_slider": 0.55, 
+        "prompt_textbox": "This video shows Mount saint helens, washington - the stunning scenery of a rocky mountains during golden hours - wide shot. A soaring drone footage captures the majestic beauty of a coastal cliff, its red and yellow stratified rock faces rich in color and against the vibrant turquoise of the sea.", 
+        "negative_prompt_textbox": "Strange motion trajectory, a poor composition and deformed video, worst quality, normal quality, low quality, low resolution, duplicate and ugly, strange body structure, long and strange neck, bad teeth, bad eyes, bad limbs, bad hands, rotating camera, blurry camera, shaking camera", 
+        "sampler_dropdown": "Euler", 
+        "sample_step_slider": 30, 
+        "width_slider": 672, 
+        "height_slider": 384, 
+        "is_image": is_image,
+        "length_slider": length_slider,
+        "cfg_scale_slider": 6,
+        "seed_textbox": 43,
+    })
+    r = requests.post(f'{url}/easyanimate/infer_forward', data=datas, timeout=1500)
+    data = r.content.decode('utf-8')
+    return data
+
+if __name__ == '__main__':
+    # initiate time
+    now_date    = datetime.now()
+    time_start  = time.time()  
+    
+    # -------------------------- #
+    #  Step 1: update edition
+    # -------------------------- #
+    edition = "v2"
+    outputs = post_update_edition(edition)
+    print('Output update edition: ', outputs)
+
+    # -------------------------- #
+    #  Step 2: update edition
+    # -------------------------- #
+    diffusion_transformer_path = "/your-path/EasyAnimate/models/Diffusion_Transformer/EasyAnimateV2-XL-2-512x512"
+    outputs = post_diffusion_transformer(diffusion_transformer_path)
+    print('Output update edition: ', outputs)
+
+    # -------------------------- #
+    #  Step 3: infer
+    # -------------------------- #
+    is_image = False
+    length_slider = 27
+    outputs = post_infer(is_image, length_slider)
+    
+    # Get decoded data
+    outputs = json.loads(outputs)
+    base64_encoding = outputs["base64_encoding"]
+    decoded_data = base64.b64decode(base64_encoding)
+
+    if is_image or length_slider == 1:
+        file_path = "1.png"
+    else:
+        file_path = "1.mp4"
+    with open(file_path, "wb") as file:
+        file.write(decoded_data)
+        
+    # End of record time
+    # The calculated time difference is the execution time of the program, expressed in seconds / s
+    time_end = time.time()  
+    time_sum = (time_end - time_start) % 60 
+    print('# --------------------------------------------------------- #')
+    print(f'#   Total expenditure: {time_sum}s')
+    print('# --------------------------------------------------------- #')

easyanimate/ui/ui.py

@@ -72,6 +72,8 @@ class EasyAnimateController:
         self.vae                   = None
         self.transformer           = None
         self.pipeline              = None
+        self.motion_module_path    = "none"
+        self.base_model_path       = "none"
         self.lora_model_path       = "none"
         
         self.weight_dtype = torch.bfloat16
@@ -135,6 +137,7 @@ class EasyAnimateController:
         return gr.Dropdown.update()
 
     def update_motion_module(self, motion_module_dropdown):
+        self.motion_module_path = motion_module_dropdown
         print("Update motion module")
         if motion_module_dropdown == "none":
             return gr.Dropdown.update()
@@ -155,6 +158,7 @@ class EasyAnimateController:
             return gr.Dropdown.update()
 
     def update_base_model(self, base_model_dropdown):
+        self.base_model_path = base_model_dropdown
         print("Update base model")
         if base_model_dropdown == "none":
             return gr.Dropdown.update()
@@ -172,6 +176,10 @@ class EasyAnimateController:
             return gr.Dropdown.update()
 
     def update_lora_model(self, lora_model_dropdown):
+        print("Update lora model")
+        if lora_model_dropdown == "none":
+            self.lora_model_path = "none"
+            return gr.Dropdown.update()
         lora_model_dropdown = os.path.join(self.personalized_model_dir, lora_model_dropdown)
         self.lora_model_path = lora_model_dropdown
         return gr.Dropdown.update()
@@ -181,6 +189,7 @@ class EasyAnimateController:
         diffusion_transformer_dropdown,
         motion_module_dropdown,
         base_model_dropdown,
+        lora_model_dropdown, 
         lora_alpha_slider,
         prompt_textbox, 
         negative_prompt_textbox, 
@@ -191,13 +200,22 @@ class EasyAnimateController:
         is_image,
         length_slider, 
         cfg_scale_slider, 
-        seed_textbox
-    ):    
+        seed_textbox,
+        is_api = False,
+    ):
         global sample_idx
         if self.transformer is None:
             raise gr.Error(f"Please select a pretrained model path.")
-        if motion_module_dropdown == "": 
-            raise gr.Error(f"Please select a motion module.")
+
+        if self.base_model_path != base_model_dropdown:
+            self.update_base_model(base_model_dropdown)
+
+        if self.motion_module_path != motion_module_dropdown:
+            self.update_motion_module(motion_module_dropdown)
+
+        if self.lora_model_path != lora_model_dropdown:
+            print("Update lora model")
+            self.update_lora_model(lora_model_dropdown)
 
         if is_xformers_available(): self.transformer.enable_xformers_memory_efficient_attention()
 
@@ -207,7 +225,7 @@ class EasyAnimateController:
             self.pipeline = merge_lora(self.pipeline, self.lora_model_path, multiplier=lora_alpha_slider)
         self.pipeline.to("cuda")
 
-        if seed_textbox != -1 and seed_textbox != "": torch.manual_seed(int(seed_textbox))
+        if int(seed_textbox) != -1 and seed_textbox != "": torch.manual_seed(int(seed_textbox))
         else: seed_textbox = np.random.randint(0, 1e10)
         generator = torch.Generator(device="cuda").manual_seed(int(seed_textbox))
         
@@ -229,7 +247,10 @@ class EasyAnimateController:
             torch.cuda.ipc_collect()
             if self.lora_model_path != "none":
                 self.pipeline = unmerge_lora(self.pipeline, self.lora_model_path, multiplier=lora_alpha_slider)
-            return gr.Image.update(), gr.Video.update()
+            if is_api:
+                return save_sample_path, f"Error. error information is {str(e)}"
+            else:
+                return gr.Image.update(), gr.Video.update()
 
         # lora part
         if self.lora_model_path != "none":
@@ -256,7 +277,7 @@ class EasyAnimateController:
         index = len([path for path in os.listdir(self.savedir_sample)]) + 1
         prefix = str(index).zfill(3)
 
-        if is_image:
+        if is_image or length_slider == 1:
             save_sample_path = os.path.join(self.savedir_sample, prefix + f".png")
 
             image = sample[0, :, 0]
@@ -264,11 +285,19 @@ class EasyAnimateController:
             image = (image * 255).numpy().astype(np.uint8)
             image = Image.fromarray(image)
             image.save(save_sample_path)
-            return gr.Image.update(value=save_sample_path, visible=True), gr.Video.update(value=None, visible=False)
+
+            if is_api:
+                return save_sample_path, "Success"
+            else:
+                return gr.Image.update(value=save_sample_path, visible=True), gr.Video.update(value=None, visible=False)
         else:
             save_sample_path = os.path.join(self.savedir_sample, prefix + f".mp4")
             save_videos_grid(sample, save_sample_path, fps=12 if self.edition == "v1" else 24)
-            return gr.Image.update(visible=False, value=None), gr.Video.update(value=save_sample_path, visible=True)
+
+            if is_api:
+                return save_sample_path, "Success"
+            else:
+                return gr.Image.update(visible=False, value=None), gr.Video.update(value=save_sample_path, visible=True)
 
 def ui():
     controller = EasyAnimateController()
@@ -326,8 +355,7 @@ def ui():
                     interactive=True,
                     visible=False
                 )
-                motion_module_dropdown.change(fn=controller.update_motion_module, inputs=[motion_module_dropdown], outputs=[motion_module_dropdown])
-                
+
                 motion_module_refresh_button = gr.Button(value="\U0001F503", elem_classes="toolbutton", visible=False)
                 def update_motion_module():
                     controller.refresh_motion_module()
@@ -340,7 +368,6 @@ def ui():
                     value="none",
                     interactive=True,
                 )
-                base_model_dropdown.change(fn=controller.update_base_model, inputs=[base_model_dropdown], outputs=[base_model_dropdown])
                 
                 lora_model_dropdown = gr.Dropdown(
                     label="Select LoRA model (optional)",
@@ -348,8 +375,7 @@ def ui():
                     value="none",
                     interactive=True,
                 )
-                lora_model_dropdown.change(fn=controller.update_lora_model, inputs=[lora_model_dropdown], outputs=[lora_model_dropdown])
-                
+
                 lora_alpha_slider = gr.Slider(label="LoRA alpha", value=0.55, minimum=0, maximum=2, interactive=True)
                 
                 personalized_refresh_button = gr.Button(value="\U0001F503", elem_classes="toolbutton")
@@ -368,10 +394,10 @@ def ui():
                 """
             )
             
-            prompt_textbox = gr.Textbox(label="Prompt", lines=2)
-            negative_prompt_textbox = gr.Textbox(label="Negative prompt", lines=2, value="Strange motion trajectory, a poor composition and deformed video, worst quality, normal quality, low quality, low resolution, duplicate and ugly" )
+            prompt_textbox = gr.Textbox(label="Prompt", lines=2, value="This video shows the majestic beauty of a waterfall cascading down a cliff into a serene lake. The waterfall, with its powerful flow, is the central focus of the video. The surrounding landscape is lush and green, with trees and foliage adding to the natural beauty of the scene")
+            negative_prompt_textbox = gr.Textbox(label="Negative prompt", lines=2, value="Strange motion trajectory, a poor composition and deformed video, worst quality, normal quality, low quality, low resolution, duplicate and ugly, strange body structure, long and strange neck, bad teeth, bad eyes, bad limbs, bad hands, rotating camera, blurry camera, shaking camera" )
                 
-            with gr.Row().style(equal_height=False):
+            with gr.Row():
                 with gr.Column():
                     with gr.Row():
                         sampler_dropdown   = gr.Dropdown(label="Sampling method", choices=list(scheduler_dict.keys()), value=list(scheduler_dict.keys())[0])
@@ -385,7 +411,7 @@ def ui():
                     cfg_scale_slider = gr.Slider(label="CFG Scale",        value=6.0, minimum=0,   maximum=20)
                     
                     with gr.Row():
-                        seed_textbox = gr.Textbox(label="Seed", value=-1)
+                        seed_textbox = gr.Textbox(label="Seed", value=43)
                         seed_button  = gr.Button(value="\U0001F3B2", elem_classes="toolbutton")
                         seed_button.click(fn=lambda: gr.Textbox.update(value=random.randint(1, 1e8)), inputs=[], outputs=[seed_textbox])
 
@@ -395,7 +421,7 @@ def ui():
                 result_video = gr.Video(label="Generated Animation", interactive=False)
 
             is_image.change(
-                lambda x: length_slider.update(visible=not x),
+                lambda x: gr.update(visible=not x),
                 inputs=[is_image],
                 outputs=[length_slider],
             )
@@ -419,6 +445,7 @@ def ui():
                     diffusion_transformer_dropdown,
                     motion_module_dropdown,
                     base_model_dropdown,
+                    lora_model_dropdown,
                     lora_alpha_slider,
                     prompt_textbox, 
                     negative_prompt_textbox, 
@@ -433,7 +460,7 @@ def ui():
                 ],
                 outputs=[result_image, result_video]
             )
-    return demo
+    return demo, controller
 
 
 class EasyAnimateController_Modelscope:
@@ -497,7 +524,7 @@ class EasyAnimateController_Modelscope:
         self.pipeline.scheduler = scheduler_dict[sampler_dropdown](**OmegaConf.to_container(self.inference_config.noise_scheduler_kwargs))
         self.pipeline.to("cuda")
 
-        if seed_textbox != -1 and seed_textbox != "": torch.manual_seed(int(seed_textbox))
+        if int(seed_textbox) != -1 and seed_textbox != "": torch.manual_seed(int(seed_textbox))
         else: seed_textbox = np.random.randint(0, 1e10)
         generator = torch.Generator(device="cuda").manual_seed(int(seed_textbox))
         
@@ -523,7 +550,7 @@ class EasyAnimateController_Modelscope:
         index = len([path for path in os.listdir(self.savedir_sample)]) + 1
         prefix = str(index).zfill(3)
         
-        if is_image:
+        if is_image or length_slider == 1:
             save_sample_path = os.path.join(self.savedir_sample, prefix + f".png")
 
             image = sample[0, :, 0]
@@ -549,10 +576,10 @@ def ui_modelscope(edition, config_path, model_name, savedir_sample):
             """
         )
         with gr.Column(variant="panel"):
-            prompt_textbox = gr.Textbox(label="Prompt", lines=2)
-            negative_prompt_textbox = gr.Textbox(label="Negative prompt", lines=2, value="Strange motion trajectory, a poor composition and deformed video, worst quality, normal quality, low quality, low resolution, duplicate and ugly" )
+            prompt_textbox = gr.Textbox(label="Prompt", lines=2, value="This video shows the majestic beauty of a waterfall cascading down a cliff into a serene lake. The waterfall, with its powerful flow, is the central focus of the video. The surrounding landscape is lush and green, with trees and foliage adding to the natural beauty of the scene")
+            negative_prompt_textbox = gr.Textbox(label="Negative prompt", lines=2, value="Strange motion trajectory, a poor composition and deformed video, worst quality, normal quality, low quality, low resolution, duplicate and ugly, strange body structure, long and strange neck, bad teeth, bad eyes, bad limbs, bad hands, rotating camera, blurry camera, shaking camera" )
                 
-            with gr.Row().style(equal_height=False):
+            with gr.Row():
                 with gr.Column():
                     with gr.Row():
                         sampler_dropdown   = gr.Dropdown(label="Sampling method", choices=list(scheduler_dict.keys()), value=list(scheduler_dict.keys())[0])
@@ -566,15 +593,15 @@ def ui_modelscope(edition, config_path, model_name, savedir_sample):
                         length_slider    = gr.Slider(label="Animation length", value=80,  minimum=40,  maximum=96,   step=1)
                         cfg_scale_slider = gr.Slider(label="CFG Scale",        value=6.0, minimum=0,   maximum=20)
                     else:
-                        width_slider     = gr.Slider(label="Width",            value=672, minimum=384, maximum=704, step=16)
-                        height_slider    = gr.Slider(label="Height",           value=384, minimum=384, maximum=704, step=16)
+                        width_slider     = gr.Slider(label="Width",            value=672, minimum=256, maximum=704, step=16)
+                        height_slider    = gr.Slider(label="Height",           value=384, minimum=256, maximum=704, step=16)
                         with gr.Row():
                             is_image      = gr.Checkbox(False, label="Generate Image")
                             length_slider = gr.Slider(label="Animation length", value=144, minimum=9,   maximum=144,  step=9)
                         cfg_scale_slider = gr.Slider(label="CFG Scale",        value=6.0, minimum=0,   maximum=20)
                     
                     with gr.Row():
-                        seed_textbox = gr.Textbox(label="Seed", value=-1)
+                        seed_textbox = gr.Textbox(label="Seed", value=43)
                         seed_button  = gr.Button(value="\U0001F3B2", elem_classes="toolbutton")
                         seed_button.click(fn=lambda: gr.Textbox.update(value=random.randint(1, 1e8)), inputs=[], outputs=[seed_textbox])
 
@@ -584,7 +611,7 @@ def ui_modelscope(edition, config_path, model_name, savedir_sample):
                 result_video = gr.Video(label="Generated Animation", interactive=False)
 
             is_image.change(
-                lambda x: length_slider.update(visible=not x),
+                lambda x: gr.update(visible=not x),
                 inputs=[is_image],
                 outputs=[length_slider],
             )
@@ -605,4 +632,4 @@ def ui_modelscope(edition, config_path, model_name, savedir_sample):
                 ],
                 outputs=[result_image, result_video]
             )
-    return demo
+    return demo, controller