Docs: Update README with implemented features and roadmap; update CN README
This commit is contained in:
ARM64_EC
@@ -14,6 +14,22 @@ ComfyUI nodes wrapping LongCat image generation and editing pipelines (diffusers
|
||||
- **LongCatSampler**: Handles the sampling process (denoising) using the LongCat transformer.
|
||||
- **LongCatImageSizeScale**: Scales images to a target pixel area and rounds dimensions to multiples of 16.
|
||||
|
||||
## Implemented (What works today)
|
||||
- LongCat transformer: `LongCatImageTransformer2DModel` implemented and used by the pipelines.
|
||||
- Pipelines:
|
||||
- `LongCatImagePipeline` (text-to-image): prompt rewrite, tokenizer usage, latent packing, denoising loop, and VAE decode implemented.
|
||||
- `LongCatImageEditPipeline` (image editing): implemented with VL prompt handling, image latents, and edit-aware denoising.
|
||||
- Nodes (ComfyUI):
|
||||
- `LongCatCheckpointLoader` (basic transformer load; CLIP/VAE loading is scaffolded/placeholder and needs improvements to map different checkpoint formats).
|
||||
- `TextEncodeLongCatImage` (T2I text encoding; uses CLIP tokenizer/encoder)
|
||||
- `TextEncodeLongCatImageEdit` (Edit text & image encoding — partially implemented; some behavior remains to be fully wired to CLIP/VL models)
|
||||
- `VAEEncodeLongCat` and `VAEDecodeLongCat` (VAE encode / decode wrappers)
|
||||
- `LongCatSampler` (wraps ComfyUI samplers)
|
||||
- `LongCatImageSizeScale` (image scaling node)
|
||||
- Utilities: `longcat_image/utils/model_utils.py` provides functions like `pack_latents`/`unpack_latents`, `prepare_pos_ids`, `split_quotation`, `retrieve_timesteps`, and optimized scaling helpers.
|
||||
- Training examples: scripts and example configs for LoRA, SFT, Edit, and DPO training under `train_examples/`.
|
||||
- Tests: Minimal unit tests for nodes (`tests/test_nodes.py`) for CI / smoke check.
|
||||
|
||||
## Installation
|
||||
1. **Copy Folder**: Copy the `comfyui_longcat` folder into your ComfyUI `custom_nodes` directory.
|
||||
* Example: `ComfyUI/custom_nodes/comfyui_longcat`
|
||||
@@ -38,34 +54,26 @@ ComfyUI nodes wrapping LongCat image generation and editing pipelines (diffusers
|
||||
- Tests: `pytest tests/test_nodes.py`
|
||||
- Key modules: `nodes.py`, `longcat_image/pipelines/*`, `longcat_image/models/longcat_image_dit.py`.
|
||||
|
||||
## Status
|
||||
## To-Do
|
||||
### Completed / in-progress items
|
||||
- Implemented core transformer and pipelines for text-to-image and edit flows.
|
||||
- Node wrappers for basic operation and VAE/text encoding/decoding.
|
||||
- Training example scripts for LoRA, SFT, DPO, and Edit training flows.
|
||||
|
||||
### Implemented (in this repo)
|
||||
- ComfyUI integration: several nodes and pipelines are implemented and available under `longcat`.
|
||||
- Nodes & pipelines: `LongCatCheckpointLoader`, `TextEncodeLongCatImage`, `TextEncodeLongCatImageEdit`, `VAEEncodeLongCat`, `VAEDecodeLongCat`, `LongCatSampler`, `LongCatImageSizeScale`.
|
||||
- Basic inference scripts: `scripts/inference_t2i.py`, `scripts/inference_edit.py`.
|
||||
- Core model implementation: `longcat_image/models/longcat_image_dit.py` and pipeline code in `longcat_image/pipelines/`.
|
||||
- Utilities: `longcat_image/utils/*` and helpers for distributed/accelerate usage.
|
||||
- Tests: basic pytest in `tests/test_nodes.py`.
|
||||
- Training examples present: `train_examples/` contains example scripts and configs for LoRA, SFT, DPO, and edit training.
|
||||
### Planned / Roadmap
|
||||
- [ ] Finalize `LongCatCheckpointLoader` to robustly load:
|
||||
- Diffusers-style directory checkpoints (separate transformer, vae, tokenizer, scheduler subfolders)
|
||||
- Single-file safetensors checkpoints and mapping to model components
|
||||
- CLIP and VAE loading and correct device placement
|
||||
- [ ] Implement `TextEncodeLongCatImageEdit` fully (support multi-image & VL inputs and correct token/image merging).
|
||||
- [ ] Add documentation: step-by-step ComfyUI usage, example flow screenshots, and a model download/prepare helper script.
|
||||
- [ ] Add automated pipeline tests, including smoke tests and tests across dtype/device combos.
|
||||
- [ ] Add Git LFS support for model weights and an example of safe weight handling.
|
||||
- [ ] Add GitHub Actions CI to run linting and tests for PRs.
|
||||
- [ ] Add `pre-commit` config to avoid accidental commits of binary or cache files.
|
||||
- [ ] Add examples showing how to run training scripts for LoRA/SFT/DPO and how to apply saved checkpoints to the ComfyUI nodes.
|
||||
|
||||
### In-repo examples (training)
|
||||
- `train_examples/lora/`: LoRA example training script and configs.
|
||||
- `train_examples/sft/`: Supervised Fine-Tuning example script and configs.
|
||||
- `train_examples/dpo/`: Direct Preference Optimization example script and configs.
|
||||
- `train_examples/edit/`: Edit-specific training examples.
|
||||
|
||||
### Planned / To-Do
|
||||
- [ ] Add robust fallback when prompt rewriting model lacks `generate`.
|
||||
- [ ] Guard `LongCatImageEditPipeline` for `image=None` misuse.
|
||||
- [ ] Expand docs with example ComfyUI workflows, screenshots, and training tutorials (LoRA, SFT, DPO).
|
||||
- [ ] Provide a model download script and configuration guidance (including example HF hub integration).
|
||||
- [ ] Add comprehensive automated tests (pipeline smoke tests, dtype/device matrix, sample inputs).
|
||||
- [ ] Add example training recipes that are ready-to-run for LoRA/SFT/DPO and thorough documentation on the training steps.
|
||||
- [ ] Add Git LFS support for model weights and add guidance for releasing model checkpoints.
|
||||
- [ ] Add continuous integration (CI) to run tests and checks via GitHub Actions.
|
||||
- [ ] Add pre-commit hooks and consistent formatting (Black / isort / flake8).
|
||||
- [ ] Improve release and packaging automation (setup.py / wheel, CI release process).
|
||||
If you'd like, I can implement the `LongCatCheckpointLoader` improvements and the `TextEncodeLongCatImageEdit` node next, and add GitHub Actions for CI.
|
||||
|
||||
## Credits
|
||||
- LongCat base model & pipelines: original LongCat project (LongCat team).
|
||||
|
||||
+34
-26
@@ -12,6 +12,22 @@
|
||||
- **LongCatSampler**：使用 LongCat Transformer 进行采样（去噪）。
|
||||
- **LongCatImageSizeScale**：按目标像素面积缩放，并将宽高对齐到 16 的倍数。
|
||||
|
||||
## 已实现（当前可用）
|
||||
- Transformer: `LongCatImageTransformer2DModel` 已实现并用于管线。
|
||||
- 管线：
|
||||
- `LongCatImagePipeline`（文生图）：实现了提示词改写、tokenizer、潜变量打包、去噪循环与 VAE 解码等功能。
|
||||
- `LongCatImageEditPipeline`（图像编辑）：实现了视觉-语言提示处理、图像潜变量与编辑专用去噪逻辑。
|
||||
- 节点（ComfyUI）：
|
||||
- `LongCatCheckpointLoader`：实现了基本的 transformer 加载（注意：CLIP/VAE 的完全加载仍为占位/脚手架实现，需要改进以支持多种模型格式）。
|
||||
- `TextEncodeLongCatImage`：T2I 文本编码实现（基于 CLIP tokenizer/encoder）。
|
||||
- `TextEncodeLongCatImageEdit`：编辑模式的文本+图像编码（部分实现，尚需完善对 CLIP/VL 的完整支持）。
|
||||
- `VAEEncodeLongCat` 与 `VAEDecodeLongCat`：VAE 编码/解码封装。
|
||||
- `LongCatSampler`：封装 ComfyUI 的采样器。
|
||||
- `LongCatImageSizeScale`：图像缩放节点。
|
||||
- 工具函数：`longcat_image/utils/model_utils.py` 包含 `pack_latents`/`unpack_latents`、`prepare_pos_ids`、`split_quotation`、`retrieve_timesteps`、`optimized_scale` 等函数。
|
||||
- 训练示例：包含 LoRA、SFT、DPO、编辑训练脚本和示例配置（位于 `train_examples/`）。
|
||||
- 测试：提供简单的节点级单元测试（`tests/test_nodes.py`）。
|
||||
|
||||
## 安装
|
||||
1. **复制文件夹**：将 `comfyui_longcat` 文件夹复制到 ComfyUI 的 `custom_nodes` 目录下。
|
||||
* 例如：`ComfyUI/custom_nodes/comfyui_longcat`
|
||||
@@ -36,34 +52,26 @@
|
||||
- 测试：`pytest tests/test_nodes.py`
|
||||
- 关键模块：`nodes.py`、`longcat_image/pipelines/*`、`longcat_image/models/longcat_image_dit.py`。
|
||||
|
||||
## 状态
|
||||
## 待办
|
||||
### 已完成 / 进行中
|
||||
- 实现了核心 transformer 与用于文生图/编辑的管线。
|
||||
- 实现了基本的节点封装（加载器、编码器、VAE、采样器、大小缩放）。
|
||||
- 提供了针对 LoRA、SFT、DPO 和编辑训练流程的示例脚本。
|
||||
|
||||
### 已实现（仓库中）
|
||||
- ComfyUI 集成：实现了若干节点和管线，命名空间 `longcat` 中可用。
|
||||
- 节点与管线：`LongCatCheckpointLoader`、`TextEncodeLongCatImage`、`TextEncodeLongCatImageEdit`、`VAEEncodeLongCat`、`VAEDecodeLongCat`、`LongCatSampler`、`LongCatImageSizeScale`。
|
||||
- 基础推理脚本：`scripts/inference_t2i.py`、`scripts/inference_edit.py`。
|
||||
- 模型实现：`longcat_image/models/longcat_image_dit.py` 与 `longcat_image/pipelines/` 下的管线代码。
|
||||
- 工具类：`longcat_image/utils/*`，以及用于 accelerate 的分布式/加速辅助函数。
|
||||
- 测试：基本单元测试位于 `tests/test_nodes.py`。
|
||||
- 训练示例：`train_examples/` 提供 LoRA、SFT、DPO、编辑相关训练的示例脚本与配置。
|
||||
### 计划 / 路线图
|
||||
- [ ] 完善 `LongCatCheckpointLoader` 对多种权重格式的兼容：
|
||||
- 支持 Diffusers 风格的 checkpoint（subfolder: transformer, vae, tokenizer, scheduler 等）
|
||||
- 支持单文件 safetensors/checkpoint 并正确映射模型组件
|
||||
- 正确加载 CLIP 与 VAE，并支持自动的设备与 offload 配置
|
||||
- [ ] 完成 `TextEncodeLongCatImageEdit` 的实现（支持多图输入、VL 合并和 placeholder 替换）。
|
||||
- [ ] 补充文档：一步步的 ComfyUI 使用流程、示例截图和模型准备脚本。
|
||||
- [ ] 增加自动化管线测试（冒烟测试、不同 dtype/device 组合）。
|
||||
- [ ] 为大权重文件启用 Git LFS 并给出示例与说明。
|
||||
- [ ] 增加 GitHub Actions CI（PR 检查，lint 和运行测试）。
|
||||
- [ ] 添加 `pre-commit` 配置以避免误提交缓存和二进制文件。
|
||||
- [ ] 补充示例说明如何运行 LoRA / SFT / DPO 训练脚本以及如何将训练产物应用到 ComfyUI 节点。
|
||||
|
||||
### 仓库中的训练示例
|
||||
- `train_examples/lora/`：LoRA 示例训练脚本及配置。
|
||||
- `train_examples/sft/`：SFT（监督微调）示例脚本与配置。
|
||||
- `train_examples/dpo/`：DPO（偏好优化）示例脚本与配置。
|
||||
- `train_examples/edit/`：针对编辑任务的训练示例。
|
||||
|
||||
### 规划 / 待办
|
||||
- [ ] 当提示词改写模型缺少 `generate` 时增加可靠回退。
|
||||
- [ ] 为 `LongCatImageEditPipeline` 增加 `image=None` 误用保护。
|
||||
- [ ] 扩展文档，提供完整的 ComfyUI 流程示例、截图与训练教程（LoRA、SFT、DPO）。
|
||||
- [ ] 提供模型下载脚本和配置指南（包括 Hugging Face Hub 的示例）。
|
||||
- [ ] 增加全面的自动化测试（管线冒烟测试、dtype/device 组合、示例输入）。
|
||||
- [ ] 提供可运行的 LoRA / SFT / DPO 训练配方，并补充详细的训练步骤文档。
|
||||
- [ ] 为模型权重等大文件启用 Git LFS 并说明如何发布权重。
|
||||
- [ ] 添加 CI（GitHub Actions）用于运行测试与检查。
|
||||
- [ ] 添加 pre-commit 钩子和统一格式化（Black / isort / flake8）。
|
||||
- [ ] 提升发布与打包自动化（wheel / setup.py，CI 自动发布）。
|
||||
如果您希望，我可以继续：完善 `LongCatCheckpointLoader` 和 `TextEncodeLongCatImageEdit`，或为仓库添加 CI 配置与 Git LFS 支持。
|
||||
|
||||
## 致谢
|
||||
- LongCat 基础模型与管线：原 LongCat 项目（LongCat 团队）。
|
||||
|
||||
Reference in New Issue
Block a user