222 lines
12 KiB
Markdown
222 lines
12 KiB
Markdown
# ComfyUI-3D-Pack
|
|
An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, Differentiable Rendering, SDS/VSD Optimization, etc.)
|
|
|
|
<span style="font-size:1.5em;">
|
|
<a href=#currently-support>Features</a> —
|
|
<a href=#roadmap>Roadmap</a> —
|
|
<a href=#install>Install</a> —
|
|
<a href=#run>Run</a> —
|
|
<a href=#tips>Tips</a>
|
|
</span>
|
|
|
|
### Note: this project is still a WIP
|
|
|
|
## Currently support:
|
|
- For use case please check [Example Workflows](./_Example_Workflows/). [**Last update: 05/03/2024**]
|
|
- **Note:** you need to put [Example Inputs Files & Folders](_Example_Workflows/_Example_Inputs_Files/) under ComfyUI Root Directory\ComfyUI\input folder before you can run the example workflow
|
|
|
|
- **TripoSR**: [VAST-AI-Research/TripoSR](https://github.com/VAST-AI-Research/TripoSR) | [ComfyUI-Flowty-TripoSR](https://github.com/flowtyone/ComfyUI-Flowty-TripoSR)
|
|
- Generate NeRF representation and using marching cube to turn it into 3D mesh
|
|
|
|
<video controls autoplay loop src="https://github.com/MrForExample/ComfyUI-3D-Pack/assets/62230687/ec4f8df5-5907-4bbf-ba19-c0565fe95a97"></video>
|
|
|
|
- **Wonder3D**: [xxlong0/Wonder3D](https://github.com/xxlong0/Wonder3D)
|
|
- Generate spatial consistent 6 views images & normal maps from a single image
|
|
|
|
|
|
- **Large Multiview Gaussian Model**: [3DTopia/LGM](https://github.com/3DTopia/LGM)
|
|
- Enable single image to 3D Gaussian in less than 30 seconds on a RTX3080 GPU, later you can also convert 3D Gaussian to mesh
|
|
|
|
<video controls autoplay loop src="https://github.com/MrForExample/ComfyUI-3D-Pack/assets/62230687/e221d7f8-49ac-4ed4-809b-d4c790b6270e"></video>
|
|
|
|
- **Triplane Gaussian Transformers**: [VAST-AI-Research/TriplaneGaussian](https://github.com/VAST-AI-Research/TriplaneGaussian)
|
|
- Enable single image to 3D Gaussian in less than 10 seconds on a RTX3080 GPU, later you can also convert 3D Gaussian to mesh
|
|
|
|
<video controls autoplay loop src="https://github.com/MrForExample/ComfyUI-3D-Pack/assets/62230687/90e7f298-bdbd-4c15-9378-1ca46cbb4871"></video>
|
|
|
|
- **Preview 3DGS and 3D Mesh**: 3D Visualization inside ComfyUI:
|
|
- Using [gsplat.js](https://github.com/huggingface/gsplat.js/tree/main) and [three.js](https://github.com/mrdoob/three.js/tree/dev) for 3DGS & 3D Mesh visualization respectively
|
|
|
|
<video controls autoplay loop src="https://github.com/MrForExample/ComfyUI-3D-Pack/assets/62230687/9f3c56b1-afb3-4bf1-8845-ab1025a87463"></video>
|
|
|
|
- **Stack Orbit Camera Poses**: Automatically generate all range of camera pose combinations
|
|
- You can use it to conditioning the [StableZero123 (You need to Download the checkpoint first)](https://comfyanonymous.github.io/ComfyUI_examples/3d/), with full range of camera poses in one prompt pass
|
|
- You can use it to generate the orbit camera poses and directly input to other 3D process node (e.g. GaussianSplatting and BakeTextureToMesh)
|
|
- Example usage:
|
|
|
|
<img src="_Example_Workflows/_Example_Outputs/Cammy_Cam_Rotate_Clockwise_Camposes.png" width="256"/> <img src="_Example_Workflows/_Example_Outputs/Cammy_Cam_Rotate_Counter_Clockwise_Camposes.png" width="256"/>
|
|
<br>
|
|
<img src="_Example_Workflows/_Example_Outputs/Cammy_Cam_Rotate_Clockwise.gif" width="256"/> <img src="_Example_Workflows/_Example_Outputs/Cammy_Cam_Rotate_Counter_Clockwise.gif" width="256"/>
|
|
- Coordinate system:
|
|
- Azimuth: In top view, from angle 0 rotate 360 degree with step -90 you get (0, -90, -180/180, 90, 0), in this case camera rotates clock-wise, vice versa.
|
|
- Elevation: 0 when camera points horizontally forward, pointing down to the ground is negitive angle, vice versa.
|
|
|
|
- **3D Gaussian Splatting**
|
|
- [Improved Differential Gaussian Rasterization](https://github.com/ashawkey/diff-gaussian-rasterization)
|
|
- Better Compactness-based Densification method from [Gsgen](https://gsgen3d.github.io/),
|
|
- Support initialize gaussians from given 3D mesh (Optional)
|
|
- Support mini-batch optimazation
|
|
- Multi-View images as inputs
|
|
- Export to standard 3DGS .ply format supported
|
|
|
|
- **Gaussian Splatting Orbit Renderer**
|
|
- Render 3DGS to images sequences or video, given a 3DGS file and camera poses generated by **Stack Orbit Camera Poses** node
|
|
|
|
- Bake Multi-View images into UVTexture of given 3D mesh using [Nvdiffrast](https://github.com/NVlabs/nvdiffrast), supports:
|
|
- Export to .obj, .ply, .glb
|
|
|
|
- **NeuS**
|
|
- Fit a coarse mesh from sparse multi-view images & normal maps, as little as 4 to 6 views, pretty good at reconstruct the shape from reference images but texture lacking details.
|
|
|
|
- **Deep Marching Tetrahedrons**
|
|
- Allow convert 3DGS .ply file to 3D mesh <br>
|
|
*Note: I didn't spent time to turn the hyperprameters yet, the result will be improved in the future!*
|
|
|
|
- **Save & Load 3D file**
|
|
- .obj, .ply, .glb for 3D Mesh
|
|
- .ply for 3DGS
|
|
|
|
- **Switch Axis for 3DGS & 3D Mesh**
|
|
- Since different algorithms likely use different coordinate system, so the ability to re-mapping the axis of coordinate is crucial for passing generated result between differnt nodes.
|
|
|
|
## Roadmap:
|
|
- [x] Add DMTet algorithm to allow conversion from points cloud(Gaussian/.ply) to mesh (.obj, .ply, .glb)
|
|
|
|
- [x] Integrate [Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers](https://zouzx.github.io/TriplaneGaussian/)
|
|
|
|
- [x] Add interactive 3D UI inside ComfuUI to visulaize training and generated results for 3D representations
|
|
|
|
- [x] Add a new node to generate renderer image sequence given a 3D gaussians and orbit camera poses (So we can later feed it to the differentiable renderer to bake it onto a given mesh)
|
|
|
|
- [x] Integrate [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://me.kiui.moe/lgm/)
|
|
|
|
- [ ] Add a general SDS/ISM Optimization algorithm to allow training 3D representations with diffusion model, *The real fun starts here* ;)
|
|
- Need to do some in-depth research on Interval Score Matching (ISM), since math behind it makes perfect sense and also there are so many ways we could improve upon the result obtained from [LucidDreamer](https://github.com/EnVision-Research/LucidDreamer)
|
|
|
|
- [ ] Improve 3DGS to Mesh conversion algorithms:
|
|
- Support to training DMTet with images(RGB, Alpha, Normal Map)
|
|
- Find better methods to converts 3DGS or Points Cloud to Mesh (Normal maps reconstruction maybe?)
|
|
|
|
- [ ] Add Structure from motion (SfM) initialization for 3DGS (Better first guess -> Faster convergence & Better result)
|
|
|
|
- [ ] Add a few best Nerf algorithms (No idea yet, [instant-ngp](https://github.com/NVlabs/instant-ngp) maybe?)
|
|
|
|
## Install:
|
|
|
|
**[IMPORTANT!!!]** <br> Currently this package is only been tested in following setups:
|
|
- Windows 10/11 (Tested on my laptop)
|
|
- Ubuntu 23.10 [(Tested by @watsieboi)](https://github.com/MrForExample/ComfyUI-3D-Pack/issues/16)
|
|
- ComfyUI python_embed/Miniconda/Conda Python 3.11.x
|
|
- Torch version >= 2.1.2+cu121
|
|
|
|
<br>
|
|
|
|
Assume you have already downloaded [ComfyUI](https://github.com/comfyanonymous/ComfyUI) & Configed your [CUDA](https://developer.nvidia.com/cuda-12-1-0-download-archive) environment.
|
|
|
|
### Install Method 0: Directly inside ComfyUI Python Embeded Environment
|
|
***Node: (I've only built the packages with windows_python3.11_cuda12.1, whitch is ComfyUI Windows Portable's default setup)***
|
|
|
|
First install [Visual Studio Build Tools 2022/2019](https://visualstudio.microsoft.com/downloads/?q=build+tools) with Workloads: Desktop development with C++ (There are a few JIT torch cpp extension that builds in runtime)
|
|
- Alternatively, according to [@doctorpangloss](https://github.com/MrForExample/ComfyUI-3D-Pack/issues/5), you can setup the c++/cuda build environments in windows by using [chocolatey](https://chocolatey.org/) with following command:
|
|
|
|
```bash
|
|
# using git bash for the sake of simplicity
|
|
# enable developer mode
|
|
# google this: allow os.symlink on windows by adding your username to the local security policy entry for it.
|
|
# you will have to restart your computer
|
|
# install chocolatey using powershell, then install the prereqs for compilation on Windows
|
|
choco install -y visualstudio2022buildtools
|
|
choco install -y visualstudio2022-workload-vctools --package-parameters "--add Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset --add Microsoft.VisualStudio.Component.VC.Llvm.Clang"
|
|
```
|
|
|
|
Go to the Comfy3D root directory: *ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack* and run:
|
|
|
|
```bash
|
|
install_windows_portable_win_py311_cu121.bat
|
|
```
|
|
|
|
### Install Method 1: Using Miniconda(Works on Windows & Linux & Mac)
|
|
***Note: [In some edge cases Miniconda fails Anaconda could fix the issue](https://github.com/MrForExample/ComfyUI-3D-Pack/issues/49)***
|
|
|
|
#### Setup with Miniconda:
|
|
First download [Miniconda](https://docs.conda.io/projects/miniconda/en/latest/) (*One of the best way to manage a clean and separated python envirments*)
|
|
|
|
Then running following commands to setup the Miniconda environment for ComfyUI:
|
|
|
|
```bash
|
|
# Go to your Your ComfyUI root directory, for my example:
|
|
cd C:\Users\reall\Softwares\ComfyUI_windows_portable
|
|
|
|
conda create -p ./python_miniconda_env/ComfyUI python=3.11
|
|
|
|
# conda will tell what command to use to activate the env
|
|
conda activate C:\Users\reall\Softwares\ComfyUI_windows_portable\python_miniconda_env\ComfyUI
|
|
|
|
# update pip
|
|
python -m pip install --upgrade pip
|
|
|
|
# You can using following command to installing CUDA only in the miniconda environment you just created if you don't want to donwload and install it manually & globally:
|
|
# conda install -c "nvidia/label/cuda-12.1.0" cuda-toolkit
|
|
|
|
# Install the main packahes
|
|
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
|
|
|
|
pip install -r ./ComfyUI/requirements.txt
|
|
|
|
# Then go to ComfyUI-3D-Pack directory under the ComfyUI Root Directory\ComfyUI\custom_nodes for my example is:
|
|
cd C:\Users\reall\Softwares\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-3D-Pack
|
|
```
|
|
|
|
- Alternatively you can check this tutorial: [Installing ComfyUI with Miniconda On Windows and Mac](https://www.comflowy.com/preparation-for-study/install#step-two-download-comfyui)
|
|
|
|
#### Install with Miniconda:
|
|
Go to the Comfy3D root directory: *ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack* and run:
|
|
|
|
```bash
|
|
install_miniconda.bat
|
|
```
|
|
Just in case `install_miniconda.bat` may not working in your OS, you could also run the following commands under the same directory: (Works with Linux & macOS)
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
|
|
pip install -r requirements_post.txt
|
|
```
|
|
<br>
|
|
|
|
**Plus:**<br>
|
|
- For those who want to run it inside Google Colab, you can check the [install instruction from @lovisdotio](https://github.com/MrForExample/ComfyUI-3D-Pack/issues/13)
|
|
|
|
## Run:
|
|
Copy the files inside folder [__New_ComfyUI_Bats](./_New_ComfyUI_Bats/) to your ComfyUI root directory, and double click run_nvidia_gpu_miniconda.bat to start ComfyUI!
|
|
- Alternatively you can just activate the Conda env: `python_miniconda_env\ComfyUI`, and go to your ComfyUI root directory then run command `python ./ComfyUI/main.py`
|
|
|
|
## Tips
|
|
* OpenGL world & camera coordinate system:
|
|
```
|
|
World Camera
|
|
|
|
+y up target
|
|
| | /
|
|
| | /
|
|
|______+x |/______right
|
|
/ /
|
|
/ /
|
|
/ /
|
|
+z forward
|
|
|
|
elevation: in (-90, 90), from +y to -y is (-90, 90)
|
|
azimuth: in (-180, 180), from +z to +x is (0, 90)
|
|
```
|
|
|
|
* Wonder3D world & camera coordinate system:
|
|
|
|
|
|
|
|
* Three.js coordinate system: (z-axis is pointing towards you and is coming out of the screen)
|
|
|
|
|
|
|
|
* If you encounter OpenGL errors (e.g., `[F glutil.cpp:338] eglInitialize() failed`), then set `force_cuda_rasterize` to true on corresponding node
|
|
* If after the installation, your ComfyUI get stucked at starting or running, you could following the instruction in following link to solve the problem: [Code Hangs Indefinitely When Evaluating Neuron Models on GPU](https://github.com/lava-nc/lava-dl/discussions/211)
|