ComfyUI-ELLA

ComfyUI implementation for ELLA.

🌟 Changelog

• [2024.4.19] Initial repo

📚 Example workflows

The examples directory has workflow examples. You can directly load these images as workflow into ComfyUI for use.

🎉 It works with controlnet! And EMMA is working in progress.

📗 Install

Download or git clone this repository inside ComfyUI/custom_nodes/ directory. ComfyUI-ELLA requires the latest version of ComfyUI. If something doesn't work be sure to upgrade.

cd ComfyUI/custom_nodes
git clone https://github.com/TencentQQGYLab/ComfyUI-ELLA

Next install dependencies.

cd ComfyUI-ELLA
pip install -r requirements.txt

📙 Models

These models must be placed in the corresponding directories under models.

Remember you can also use any custom location setting an ella & ella_encoder entry in the extra_model_paths.yaml file.

• ComfyUI/models/ella, create it if not present.
  • Place ELLA Models here
• ComfyUI/models/ella_encoder, create it if not present.
  • Place t5 model here, it should be a folder of transfomers structure with config.json

In summary, you should have the following model directory structure:

ComfyUI/models/ella/
└── ella-sd1.5-tsc-t5xl.safetensors

# It may be a little different because I only include the text_encoder part here
ComfyUI/models/ella_encoder/
└── models--google--flan-t5-xl
    ├── config.json
    ├── model.safetensors
    ├── special_tokens_map.json
    ├── spiece.model
    ├── tokenizer_config.json
    └── tokenizer.json

📖 Usage (🚧 Work in progress)

Load ELLA Model

Apply ELLA

Load T5 TextEncoder #ELLA

T5 Text Encode #ELLA

ELLA Combine Embeds

Convert Condition to ELLA Embeds

📝 TODO

• Support prompt weighting

😋 Thanks

• ComfyUI: https://github.com/comfyanonymous/ComfyUI
• Diffusers (borrowed timestep modules): https://github.com/huggingface/diffusers

😉 Citation

@misc{hu2024ella,
      title={ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment}, 
      author={Xiwei Hu and Rui Wang and Yixiao Fang and Bin Fu and Pei Cheng and Gang Yu},
      year={2024},
      eprint={2403.05135},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}