update v0.0.4

2024-03-31 13:08:41 +08:00
parent 35aada8ce8
commit bf53829530
106 changed files with 6927 additions and 889 deletions
@@ -9,8 +9,8 @@
 </p>

 ## 📖 Table of Contents
- [Introduction](#-introduction)
 - [News](#-news)
+- [Introduction](#-introduction)
 - [Installation](#%EF%B8%8F-installation)
 - [Getting Started](#-getting-started)
 - [SCEPTER Studio](#%EF%B8%8F-scepter-studio)
@@ -18,6 +18,15 @@
 - [Features](#-features)
 - [Learn More](#-learn-more)
 - [License](#license)
+- [Acknowledgement](#acknowledgement)
+
+## 🎉 News
+- [2024.03]: We optimize the training UI and checkpoint management. New [LAR-Gen](https://arxiv.org/abs/2403.19534) model has been added on SCEPTER Studio, supporting `zoom-out`, `virtual try on`, `inpainting`.
+- [2024.02]: We release new SCEdit controllable image synthesis models for SD v2.1 and SD XL. Multiple strategies applied to accelerate inference time for SCEPTER Studio.
+- [2024.01]: We release **SCEPTER Studio**, an integrated toolkit for data management, model training and inference based on [Gradio](https://www.gradio.app/).
+- [2024.01]: [SCEdit](https://arxiv.org/abs/2312.11392) support controllable image synthesis for training and inference.
+- [2023.12]: We propose [SCEdit](https://arxiv.org/abs/2312.11392), an efficient and controllable generation framework.
+- [2023.12]: We release [🪄SCEPTER](https://github.com/modelscope/scepter/) library.

 ## 📝 Introduction

@@ -28,7 +37,7 @@ Main Feature:
 - Task:
  - Text-to-image generation
  - Controllable image synthesis
-  - Image editing (TODO)
+  - Image editing
 - Training / Inference:
  - Distribute: DDP / FSDP / FairScale / Xformers
  - File system: Local / Http / OSS / Modelscope
@@ -40,15 +49,9 @@ Main Feature:
 Currently supported approaches (and counting):

 1. SD Series: [Stable Diffusion v1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5) / [Stable Diffusion v2.1](https://huggingface.co/runwayml/stable-diffusion-v1-5) / [Stable Diffusion XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
-2. SCEdit: [SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing](https://arxiv.org/abs/2312.11392)  [![Arxiv link](https://img.shields.io/static/v1?label=arXiv&message=SCEdit&color=red&logo=arxiv)](https://arxiv.org/abs/2312.11392) [![Page link](https://img.shields.io/badge/Page-SCEdit-Gree)](https://scedit.github.io/)
-3. Res-Tuning(TODO): [Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone](https://arxiv.org/abs/2310.19859) [![Arxiv link](https://img.shields.io/static/v1?label=arXiv&message=ResTuning&color=red&logo=arxiv)](https://arxiv.org/abs/2310.19859) [![Page link](https://img.shields.io/badge/Page-ResTuning-Gree)](https://res-tuning.github.io/)
-
-## 🎉 News
- [2024.02]: We release new SCEdit controllable image synthesis models for SD v2.1 and SD XL. Multiple strategies applied to accelerate inference time for SCEPTER Studio.
- [2024.01]: We release **SCEPTER Studio**, an integrated toolkit for data management, model training and inference based on [Gradio](https://www.gradio.app/).
- [2024.01]: [SCEdit](https://arxiv.org/abs/2312.11392) support controllable image synthesis for training and inference.
- [2023.12]: We propose [SCEdit](https://arxiv.org/abs/2312.11392), an efficient and controllable generation framework.
- [2023.12]: We release [🪄SCEPTER](https://github.com/modelscope/scepter/) library.
+2. SCEdit(CVPR2024): [SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing](https://arxiv.org/abs/2312.11392)  [![Arxiv link](https://img.shields.io/static/v1?label=arXiv&message=SCEdit&color=red&logo=arxiv)](https://arxiv.org/abs/2312.11392) [![Page link](https://img.shields.io/badge/Page-SCEdit-Gree)](https://scedit.github.io/)
+3. Res-Tuning(NeurIPS2023 TODO): [Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone](https://arxiv.org/abs/2310.19859) [![Arxiv link](https://img.shields.io/static/v1?label=arXiv&message=ResTuning&color=red&logo=arxiv)](https://arxiv.org/abs/2310.19859) [![Page link](https://img.shields.io/badge/Page-ResTuning-Gree)](https://res-tuning.github.io/)
+4. LAR-Gen: [Locate, Assign, Refine: Taming Customized Image Inpainting with Text-Subject Guidance](https://arxiv.org/abs/2403.19534)  [![Arxiv link](https://img.shields.io/static/v1?label=arXiv&message=LARGen&color=red&logo=arxiv)](https://arxiv.org/abs/2403.19534) [![Page link](https://img.shields.io/badge/Page-LARGen-Gree)](https://ali-vilab.github.io/largen-page/)

 ## 🛠️ Installation

@@ -94,7 +97,7 @@ For the data format used by SCEPTER Studio, please refer to [3D_example_csv.zip]
 To facilitate starting training in command-line mode, you can use a dataset in text format, please refer to [3D_example_txt.zip](https://modelscope.cn/api/v1/models/damo/scepter/repo?Revision=master&FilePath=datasets/3D_example_txt.zip)

 ```shell
-mkdir -p cache/dataset/ && wget 'https://modelscope.cn/api/v1/models/damo/scepter_scedit/repo?Revision=master&FilePath=dataset/3D_example_txt.zip' -O cache/dataset/3D_example_txt.zip && unzip cache/dataset/3D_example_txt.zip -d cache/dataset/ && rm cache/dataset/3D_example_txt.zip
+mkdir -p cache/datasets/ && wget 'https://modelscope.cn/api/v1/models/damo/scepter_scedit/repo?Revision=master&FilePath=dataset/3D_example_txt.zip' -O cache/datasets/3D_example_txt.zip && unzip cache/datasets/3D_example_txt.zip -d cache/datasets/ && rm cache/datasets/3D_example_txt.zip
 ```

 ### Training
@@ -165,6 +168,14 @@ python scepter/tools/run_inference.py --cfg scepter/methods/scedit/ctr/sd21_768_
 python scepter/tools/run_inference.py --cfg scepter/methods/scedit/ctr/sd21_768_sce_ctr_pose.yaml --num_samples 1 --prompt 'super mario' --save_folder 'test_mario_pose' --image_size 768 --task control --image 'asset/images/pose_source.png' --control_mode source --pretrained_model ms://damo/scepter_scedit@controllable_model/SD2.1/pose_control/0_SwiftSCETuning/pytorch_model.bin   # pose
 ```

+### Customize Modules
+Refer to `example`, build your modules in `example/impls`.
+```python
+cd example
+python run.py --cfg example.yaml
+```
+
+
 ## 🖥️ SCEPTER Studio

 ### Launch
@@ -181,16 +192,94 @@ git clone https://github.com/modelscope/scepter.git
 PYTHONPATH=. python scepter/tools/webui.py --cfg scepter/methods/studio/scepter_ui.yaml
 ```

-The startup of **SCEPTER Studio** eliminates the need for manual downloading and organizing of models; it will automatically load the corresponding models and store them in a local directory. 
-Depending on the network and hardware situation, the initial startup usually requires 15-60 minutes, primarily involving the download and processing of SDv1.5, SDv2.1, and SDXL models. 
+The startup of **SCEPTER Studio** eliminates the need for manual downloading and organizing of models; it will automatically load the corresponding models and store them in a local directory.
+Depending on the network and hardware situation, the initial startup usually requires 15-60 minutes, primarily involving the download and processing of SDv1.5, SDv2.1, and SDXL models.
 Therefore, subsequent startups will become much faster (about one minute) as downloading is no longer required.

+* LAR-Gen: we release `zoom-out`, `virtual try on`, `inpainting(text guided)`, `inpainting(text + reference image guided)` image editing capabilities.
+Please note that the **Data Preprocess** button must be clicked before clicking the **Generate** button.
+<p align="center">
+<img src="./asset/images/largen.gif">
+</p>
+
 ### Modelscope Studio

 We deploy a work studio on Modelscope that includes only the inference tab, please refer to [ms_scepter_studio](https://www.modelscope.cn/studios/damo/scepter_studio/summary)

 ## 🖼️ Gallery

+### LAR-Gen: Zoom-Out
+<table>
+  <tr>
+    <td><strong>Origin Image</strong><br>Prompt: a temple on fire</td>
+    <td><strong>Zoom-Out</strong><br>CenterAround:0.75</td>
+    <td><strong>Zoom-Out</strong><br>CenterAround:0.75</td>
+    <td><strong>Zoom-Out</strong><br>CenterAround:0.75</td>
+    <td><strong>Zoom-Out</strong><br>CenterAround:0.75</td>
+  </tr>
+  <tr>
+    <td><img src="asset/images/zoom_out/ex1_scene_im.jpeg" width="240"></td>
+    <td><img src="asset/images/zoom_out/ex1_zoom_out1.jpeg" width="240"></td>
+    <td><img src="./asset/images/zoom_out/ex1_zoom_out2.jpeg" width="240"></td>
+    <td><img src="./asset/images/zoom_out/ex1_zoom_out3.jpeg" width="240"></td>
+    <td><img src="./asset/images/zoom_out/ex1_zoom_out4.jpeg" width="240"></td>
+  </tr>
+</table>
+
+### LAR-Gen: Virtual-Try-on
+<table>
+  <tr>
+    <td><strong>Model Image</strong></td>
+    <td><strong>Model Mask</strong></td>
+    <td><strong>Clothing Image</strong></td>
+    <td><strong>Clothing Mask</strong></td>
+    <td><strong>Try-on Output</strong></td>
+  </tr>
+  <tr>
+    <td><img src="asset/images/virtual_try_on/model.jpg" width="240"></td>
+    <td><img src="asset/images/virtual_try_on/ex2_scene_mask.jpg" width="240"></td>
+    <td><img src="asset/images/virtual_try_on/tshirt.jpg" width="240"></td>
+    <td><img src="asset/images/virtual_try_on/ex2_subject_mask.jpg" width="240"></td>
+    <td><img src="asset/images/virtual_try_on/try_on_out.jpeg" width="240"></td>
+  </tr>
+</table>
+
+### LAR-Gen: Inpainting(Text-guided)
+<table>
+  <tr>
+    <td><strong>Origin Image</strong><br>Prompt: a blue and white porcelain</td>
+    <td><strong>Inpainting Mask1</strong></td>
+    <td><strong>Inpainting Output1</strong></td>
+    <td><strong>Inpainting Mask2</strong><br>Prompt: a clock</td>
+    <td><strong>Inpainting Output2</strong></td>
+  </tr>
+  <tr>
+    <td><img src="asset/images/inpainting_text/ex3_scene_im.jpg" width="240"></td>
+    <td><img src="asset/images/inpainting_text/ex3_scene_mask.jpg" width="240"></td>
+    <td><img src="asset/images/inpainting_text/inpainting_text.jpeg" width="240"></td>
+    <td><img src="asset/images/inpainting_text/ex3_scene_mask2.jpg" width="240"></td>
+    <td><img src="asset/images/inpainting_text/inpainting_text2.jpeg" width="240"></td>
+  </tr>
+</table>
+
+### LAR-Gen: Inpainting(Text + Reference Image Guided)
+<table>
+  <tr>
+    <td><strong>Origin Image</strong><br>Prompt: a dog wearing sunglasses</td>
+    <td><strong>Origin Mask</strong></td>
+    <td><strong>Reference Image</strong></td>
+    <td><strong>Reference Mask</strong></td>
+    <td><strong>Inpainting Output</strong></td>
+  </tr>
+  <tr>
+    <td><img src="asset/images/inpainting_text_ref/ex4_scene_im.jpg" width="240"></td>
+    <td><img src="asset/images/inpainting_text_ref/ex4_scene_mask.jpg" width="240"></td>
+    <td><img src="asset/images/inpainting_text_ref/ex4_subject_im.jpg" width="240"></td>
+    <td><img src="asset/images/inpainting_text_ref/ex4_subject_mask.jpg" width="240"></td>
+    <td><img src="asset/images/inpainting_text_ref/inpainting_text_ref.jpeg" width="240"></td>
+  </tr>
+</table>
+
 ### Dragon Year Special: Dragon Tuner

 <table>
@@ -244,22 +333,31 @@ We deploy a work studio on Modelscope that includes only the inference tab, plea
 |   SD 2.1  |     🪄     |    🪄    |     🪄     |     🪄    |     🪄     |
 |   SD XL   |     🪄     |    🪄    |     🪄     |     🪄    |     🪄     |

+### Image Editing
+- LAR-Gen
+
+| **Model** | **Locate** | **Assign** | **Refine** |
+|:---------:|:----------:|:----------:|:----------:|
+|   SD XL   |     🪄     |    🪄       |     ⏳     |
+
 ### Model URL

 - ✅ indicates support for both training and inference.
 - 🪄 denotes that the model has been published.
+- ⏳ denotes that the module has not been integrated currently.
 - More models will be released in the future.

 | Model  | URL                                                                                                                                            |
 |--------|------------------------------------------------------------------------------------------------------------------------------------------------|
-| SCEdit | [ModelScope](https://modelscope.cn/models/damo/scepter_scedit/summary) [HuggingFace](https://huggingface.co/scepter-studio/scepter_scedit) |
+| SCEdit | [ModelScope](https://modelscope.cn/models/iic/scepter_scedit/summary) [HuggingFace](https://huggingface.co/scepter-studio/scepter_scedit) |
+| LAR-Gen | [ModelScope](https://www.modelscope.cn/models/iic/LARGEN/summary) |

 PS: Scripts running within the SCEPTER framework will automatically fetch and load models based on the required dependency files, eliminating the need for manual downloads.


 ## 🔍 Learn More

- [Alibaba TongYi Vision Intelligence Lab](https://github.com/damo-vilab)
+- [Alibaba TongYi Vision Intelligence Lab](https://github.com/ali-vilab)

  Discover more about open-source projects on image generation, video generation, and editing tasks.

@@ -271,7 +369,20 @@ PS: Scripts running within the SCEPTER framework will automatically fetch and lo

  SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning) is an extensible framwork designed to faciliate lightweight model fine-tuning and inference.

+## BibTeX
+If our work is useful for your research, please consider citing:
+```bibtex
+@misc{scepter,
+    title = {SCEPTER, https://github.com/modelscope/scepter},
+    author = {SCEPTER},
+    year = {2023}
+}
+```
+

 ## License

 This project is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE).
+
+## Acknowledgement
+Thanks to [Stability-AI](https://github.com/Stability-AI), [SWIFT library](https://github.com/modelscope/swift/) and [Fooocus](https://github.com/lllyasviel/Fooocus) for their awesome work.