97 lines
3.4 KiB
Markdown
97 lines
3.4 KiB
Markdown
# ComfyUI_CSGO_Wrapper
|
||
You can using InstantX's CSGO in comfyUI
|
||
|
||
**CSGO From: [link](https://github.com/instantX-research/CSGO)**
|
||
|
||
Update
|
||
---
|
||
|
||
**2024/09/07**
|
||
* fix runway loader error, using single clip_vision /controlnet weight now；
|
||
* 修复runway跑路导致的diffuser加载报错，现在直接使用IP adapter的SDXL 图片解码单体模型（请放在clip_vision目录下）和单体controlnet SDLX模型（请放在controlnet目录下）；
|
||
* vae is not necessary,if you get black img ,please using vae
|
||
* vae并不是必须的，但是如果出现黑图或者解码不正确，才使用；
|
||
|
||
1.Installation
|
||
-----
|
||
In the ./ComfyUI /custom_node directory, run the following:
|
||
```
|
||
git clone https://github.com/smthemex/ComfyUI_CSGO_Wrapper.git
|
||
```
|
||
|
||
2.requirements
|
||
----
|
||
For ComfyUI users, all libraries in the requirements file should be available. If not, please uncomment the # and reinstall
|
||
```
|
||
pip install -r requirements.txt
|
||
```
|
||
3 Need model
|
||
----
|
||
3.1 base SDXl ckpt and vae and clip_vision and controlnet
|
||
h94/IP-Adapter/sdxl_models [link](https://huggingface.co/h94/IP-Adapter/tree/main/sdxl_models/image_encoder)
|
||
TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic [link](https://huggingface.co/TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic)
|
||
no need config
|
||
```
|
||
├── ComfyUI/models/checkpoints/
|
||
| ├──any SDXL weights # for example: Jumpernaut XL_v9-RunDiffusionPhoto_v2.safetensors
|
||
├── ComfyUI/models/vae/
|
||
| ├──any SDXL vae weights # sdxl.vae.safetensors
|
||
├── ComfyUI/models/clip_vision/
|
||
| ├──model.safetensors # h94/IP-Adapter/sdxl_models/model.safetensors
|
||
├── ComfyUI/models/controlnet /
|
||
| ├──TTPLANET_Controlnet_Tile_realistic_v2_fp16.safetensors # TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic
|
||
```
|
||
3.2 main ckpt
|
||
CSGO models [link](https://huggingface.co/InstantX/CSGO/tree/main)
|
||
```
|
||
├── ComfyUI/models/checkpoints/
|
||
| ├──acsgo_4_32.bin #need token 4/32
|
||
| ├──acsgo.bin #need token 4/16
|
||
```
|
||
|
||
3.3 if using LLM (unnecessary)
|
||
Salesforce/blip-image-captioning-large [link](https://huggingface.co/Salesforce/blip-image-captioning-large/tree/main)
|
||
```
|
||
├── any path dir/
|
||
| ├── model.safetensors
|
||
| ├── config.json
|
||
| ├── preprocessor_config.json
|
||
| ├── special_tokens_map.json
|
||
| ├── tokenizer.json
|
||
| ├── tokenizer_config.json
|
||
| ├── vocab.txt
|
||
```
|
||
or change prompt input using any other LLM.
|
||
|
||
4 Example
|
||
----
|
||
conternt+style img
|
||
|
||
style img + prompt
|
||
|
||
conternt+style img +llm new 最新
|
||
|
||
|
||
|
||
5 Function Description of Nodes
|
||
---
|
||
* content_tokens: keep in 4;
|
||
* style_tokens: if using csgo.bin keep num "16",if using csgo_4_32.bin keep num "32" ;
|
||
* text_only: if not using content img ;
|
||
* Blip is not necessary;
|
||
|
||
|
||
6 Citation
|
||
------
|
||
CSGO
|
||
``` python
|
||
@article{xing2024csgo,
|
||
title={CSGO: Content-Style Composition in Text-to-Image Generation},
|
||
author={Peng Xing and Haofan Wang and Yanpeng Sun and Qixun Wang and Xu Bai and Hao Ai and Renyuan Huang and Zechao Li},
|
||
year={2024},
|
||
journal = {arXiv 2408.16766},
|
||
}
|
||
}
|
||
```
|
||
|