nodes/MrForExample-ComfyUI-3D-Pack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add 3D Visualization & Switch Axis for 3DGS & 3D Mesh & Gaussian Splatting Orbit Renderer

Browse Source
This commit is contained in:
MrForExample
2024-02-04 19:31:31 +01:00
parent d9a79e4039
commit 16c685d98f
32 changed files with 3952 additions and 2665 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+74 -24
View File
@@ -1,23 +1,23 @@
# ComfyUI-3D-Pack
An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, Differentiable Rendering, SDS/VSD Optimization, etc.)
<span style="font-size:1.5em;">
<a href=#currently-support>Features</a> &mdash;
<a href=#roadmap>Roadmap</a> &mdash;
<a href=#install>Install</a> &mdash;
<a href=#run>Run</a> &mdash;
<a href=#tips>Tips</a>
</span>
### Note: this project is still a WIP and not been released into ComFyUI package database yet
ps: I'll show some generated result soon
---
<br>
**[IMPORTANT!!!]** <br> Currently this package is only been tested in following setups:
- Windows 10/11
- Miniconda/Conda Python 3.11.7
- I tried install this package with ComfyUI embed python env first, but I can't find a way to build CUDA related libraries, e.g. diff-gaussian-rasterization, nvdiffrast, simple-knn.
- Torch version: 2.1.2+cu121/V.2.1.2+cu118
## Currently support:
- For use case please check [example workflows](./_Example_Workflows/)
- For use case please check [Example Workflows](./_Example_Workflows/). [**Last update: 04/02/2024**]
- **Note:** you need to put [Example Inputs Files & Folders](_Example_Workflows/_Example_Inputs_Files/) under ComfyUI Root Directory\ComfyUI\input folder before you can run the example workflow
- Triplane Gaussian Transformers:
- **Preview 3DGS and 3D Mesh**: 3D Visualization inside ComfyUI:
- Using [gsplat.js](https://github.com/huggingface/gsplat.js/tree/main) and [three.js](https://github.com/mrdoob/three.js/tree/dev) for 3DGS & 3D Mesh visualization respectively
- **Triplane Gaussian Transformers**: [VAST-AI-Research/TriplaneGaussian](https://github.com/VAST-AI-Research/TriplaneGaussian)
- Enable single image to 3D Gaussian, then you can use Deep Marching Tetrahedrons node to convert it to mesh
<table class="center">
@@ -32,45 +32,71 @@ ps: I'll show some generated result soon
</tr>
</table>
- Stack Orbit Camera Poses node to automatically generate all range of camera pose combinations
- **Stack Orbit Camera Poses**: Automatically generate all range of camera pose combinations
- You can use it to conditioning the [StableZero123 (You need to Download the checkpoint first)](https://comfyanonymous.github.io/ComfyUI_examples/3d/), with full range of camera poses in one prompt pass
- You can use it to generate the orbit camera poses and directly input to other 3D process node (e.g. GaussianSplatting and BakeTextureToMesh)
- Example usage:
<img src="_Example_Workflows/_Example_Outputs/Cammy_Cam_Rotate_Clockwise_Camposes.png" width="256"/> <img src="_Example_Workflows/_Example_Outputs/Cammy_Cam_Rotate_Counter_Clockwise_Camposes.png" width="256"/>
<img src="_Example_Workflows/_Example_Outputs/Cammy_Cam_Rotate_Clockwise.gif" width="256"/> <img src="_Example_Workflows/_Example_Outputs/Cammy_Cam_Rotate_Counter_Clockwise.gif" width="256"/>
- Load 3D file (.obj, .ply, .glb)
- Also have node that support save and load 3DGS .ply file
- 3D Gaussian Splatting, with:
- Coordinate system:
- Azimuth: In top view, from angle 0 rotate 360 degree with step -90 you get (0, -90, -180/180, 90, 0), in this case camera rotates clock-wise, vice versa.
- Elevation: 0 when camera points horizontally forward, pointing down to the ground is negitive angle, vice versa.
- **3D Gaussian Splatting**
- [Improved Differential Gaussian Rasterization](https://github.com/ashawkey/diff-gaussian-rasterization)
- Better Compactness-based Densification method from [Gsgen](https://gsgen3d.github.io/),
- Support initialize gaussians from given 3D mesh (Optional)
- Support mini-batch optimazation
- Multi-View images as inputs
- Export to .ply support
- Export to standard 3DGS .ply format supported
- **Gaussian Splatting Orbit Renderer**
- Render 3DGS to images sequences or video, given a 3DGS file and camera poses generated by **Stack Orbit Camera Poses** node
- Bake Multi-View images into UVTexture of given 3D mesh using [Nvdiffrast](https://github.com/NVlabs/nvdiffrast), supports:
- Export to .obj, .ply, .glb
- Deep Marching Tetrahedrons
- **Deep Marching Tetrahedrons**
- Allow convert 3DGS .ply file to 3D mesh <br>
*Note: I didn't spent time to turn the hyperprameters yet, the result will be improved in the future!*
- **Save & Load 3D file**
- .obj, .ply, .glb for 3D Mesh
- .ply for 3DGS
- **Switch Axis for 3DGS & 3D Mesh**
- Since different algorithms likely use different coordinate system, so the ability to re-mapping the axis of coordinate is crucial for passing generated result between differnt nodes.
## Roadmap:
- [x] Add DMTet algorithm to allow convertion from points cloud(Gaussian/.ply) to mesh (.obj, .ply, .glb)
- [x] Integrate [Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers](https://github.com/VAST-AI-Research/TriplaneGaussian)
- [ ] Add interactive 3D UI inside ComfuUI to visulaize training and generated results for 3D representations
- [x] Add interactive 3D UI inside ComfuUI to visulaize training and generated results for 3D representations
- [ ] Improve DMTet result & add support to training it with images(RGB, Alpha, Normal Map)
- [x] Add a new node to generate renderer image sequence given a 3D gaussians and orbit camera poses (So we can later feed it to the differentiable renderer to bake it onto a given mesh)
- [ ] Add a general SDS/VSD Optimization algorithm to allow training 3D representations with diffusion model, *The real fun begins here* ;)
- [ ] Improve 3DGS to Mesh:
- Support to training DMTet with images(RGB, Alpha, Normal Map)
- Find better methods to converts 3DGS or Points Cloud to Mesh (Normal maps reconstruction maybe?)
- [ ] Add Structure from motion (SfM) initialization for 3DGS (Better first guess -> Faster convergence & Better result)
- [ ] Add a few best Nerf algorithms (No idea yet, [instant-ngp](https://github.com/NVlabs/instant-ngp) maybe?)
## Install:
**[IMPORTANT!!!]** <br> Currently this package is only been tested in following setups:
- Windows 10/11
- Miniconda/Conda Python 3.11.7
- I tried install this package with ComfyUI embed python env first, but I can't find a way to build CUDA related libraries, e.g. diff-gaussian-rasterization, nvdiffrast, simple-knn.
- Torch version: 2.1.2+cu121/V.2.1.2+cu118
<br>
Assume you have already downloaded [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
First download [Miniconda](https://docs.conda.io/projects/miniconda/en/latest/) (*One of the best way to manage a clean and separated python envirments*)
@@ -90,7 +116,7 @@ pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r ./ComfyUI/requirements.txt
# Then go to ComfyUI-3D-Pack directory under the .\ComfyUI\custom_nodes for my example is:
# Then go to ComfyUI-3D-Pack directory under the ComfyUI Root Directory\ComfyUI\custom_nodes for my example is:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-3D-Pack
# Finally you can double click following .bat script or run it in CLI:
install.bat
@@ -101,7 +127,7 @@ install.bat
Just in case `install.bat` may not working in your PC, you could also run the following commands under this package's root directory:
```bash
# First make sure the Conda env: python_miniconda_env\ComfyUI is activated, then go to Go to ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack and:
# First make sure the Conda env: python_miniconda_env\ComfyUI is activated, then go to ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack and:
pip install -r requirements.txt
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
@@ -122,6 +148,30 @@ install.bat
pip install git+https://github.com/facebookresearch/pytorch3d.git@stable
```
- ***Install with ComfyUI's embed python env on Windows:***
If you have already installed ComfyUI on Windows with embed python environment and you don't want to switch to Miniconda/Conda and reinstall all your packages <br>
*(Which is ideal, but unfortunately so far it seems I'm not experienced enough in this matter, but I'll continue to looking for the better solutions and please give me some suggestions if ou know better, thanks :)*
According to [@doctorpangloss](https://github.com/MrForExample/ComfyUI-3D-Pack/issues/5), you can setup the c++/cuda build environments in windows by running following command:
```bash
# using git bash for the sake of simplicity
# enable developer mode
# google this: allow os.symlink on windows by adding your username to the local security policy entry for it.
# you will have to restart your computer
# install chocolatey using powershell, then install the prereqs for compilation on Windows
choco install -y visualstudio2022buildtools
choco install -y visualstudio2022-workload-vctools --package-parameters "--add Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset --add Microsoft.VisualStudio.Component.VC.Llvm.Clang"
# I couldn't find cuda version 12.1, you may need to install cudav12.1 manually
choco install -y cuda
```
Then just go to ComfyUI-3D-Pack directory under the ComfyUI Root Directory\ComfyUI\custom_nodes and running:
```bash
pip install -r ./ComfyUI/requirements.txt
install.bat
```
## Run:
Copy the files inside folder [__New_ComfyUI_Bats](./_New_ComfyUI_Bats/) to your ComfyUI root directory, and double click run_nvidia_gpu_miniconda.bat to start ComfyUI!
- Alternatively you can just activate the Conda env: python_miniconda_env\ComfyUI, and go to your ComfyUI root directory then run command `python ./ComfyUI/main.py`
_Example_Workflows/3DGS_Multi-View_Renders_to_ply.json
-463
View File
@@ -1,463 +0,0 @@
{
"last_node_id": 21,
"last_link_id": 26,
"nodes": [
{
"id": 10,
"type": "PreviewImage",
"pos": [
1410,
-320
],
"size": {
"0": 210,
"1": 246
},
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 4
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 9,
"type": "MaskToImage",
"pos": [
1110,
-320
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 8
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
4
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "MaskToImage"
}
},
{
"id": 13,
"type": "[ComfyUI-3D] Load 3D Mesh",
"pos": [
1053,
415
],
"size": {
"0": 315,
"1": 154
},
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "mesh",
"type": "MESH",
"links": [],
"shape": 3,
"slot_index": 0
}
],
"title": "Load 3D Mesh [Optional]",
"properties": {
"Node name for S&R": "[ComfyUI-3D] Load 3D Mesh"
},
"widgets_values": [
"Meshes\\Cammy_Uniform.obj",
false,
true,
false,
false
]
},
{
"id": 19,
"type": "PreviewImage",
"pos": [
420,
-320
],
"size": [
210,
246
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 22
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 7,
"type": "VHS_LoadImages",
"pos": [
113,
54
],
"size": [
243.60000610351562,
194
],
"flags": {},
"order": 1,
"mode": 0,
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
1,
22,
23
],
"shape": 3,
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": [
2
],
"shape": 3,
"slot_index": 1
},
{
"name": "INT",
"type": "INT",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "VHS_LoadImages"
},
"widgets_values": {
"directory": "Images_enhanced",
"image_load_cap": 0,
"skip_first_images": 0,
"select_every_nth": 1,
"choose folder to upload": "image",
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"frame_load_cap": 0,
"skip_first_images": 0,
"filename": "Images_enhanced",
"type": "input",
"format": "folder",
"select_every_nth": 1
}
}
}
},
{
"id": 8,
"type": "InvertMask",
"pos": [
746,
-14
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 2
}
],
"outputs": [
{
"name": "MASK",
"type": "MASK",
"links": [
8,
24
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "InvertMask"
}
},
{
"id": 4,
"type": "[ComfyUI-3D] Generate Orbit Camera Poses",
"pos": [
830,
132
],
"size": {
"0": 400,
"1": 200
},
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "reference_images",
"type": "IMAGE",
"link": 1
}
],
"outputs": [
{
"name": "orbit_camposes",
"type": "ORBIT_CAMPOSES",
"links": [
25
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Generate Orbit Camera Poses"
},
"widgets_values": [
"([0:120], 1.75, 0, 0, 360)"
]
},
{
"id": 20,
"type": "[ComfyUI-3D] Gaussian Splatting",
"pos": [
1381,
54
],
"size": {
"0": 443.4000244140625,
"1": 758
},
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "reference_images",
"type": "IMAGE",
"link": 23
},
{
"name": "reference_masks",
"type": "MASK",
"link": 24
},
{
"name": "reference_orbit_camera_poses",
"type": "ORBIT_CAMPOSES",
"link": 25
},
{
"name": "points_cloud",
"type": "POINTCLOUD",
"link": null
},
{
"name": "ply_to_initialize_gaussian",
"type": "GS_PLY",
"link": null
},
{
"name": "mesh_to_initialize_gaussian",
"type": "MESH",
"link": null
}
],
"outputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"links": [
26
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Gaussian Splatting"
},
"widgets_values": [
49.1,
1000,
3,
10000,
0.2,
3,
0,
0,
0.5,
0.01,
0.05,
0.005,
0.005,
0.001,
0.00002,
0.02,
500,
5000,
3,
0.01,
100,
100000,
100,
700,
0.01,
0
]
},
{
"id": 21,
"type": "[ComfyUI-3D] Save 3DGS",
"pos": [
1994,
55
],
"size": {
"0": 315,
"1": 58
},
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"link": 26
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Save 3DGS"
},
"widgets_values": [
"3DGS_[time(%Y-%m-%d)].ply"
]
}
],
"links": [
[
1,
7,
0,
4,
0,
"IMAGE"
],
[
2,
7,
1,
8,
0,
"MASK"
],
[
4,
9,
0,
10,
0,
"IMAGE"
],
[
8,
8,
0,
9,
0,
"MASK"
],
[
22,
7,
0,
19,
0,
"IMAGE"
],
[
23,
7,
0,
20,
0,
"IMAGE"
],
[
24,
8,
0,
20,
1,
"MASK"
],
[
25,
4,
0,
20,
2,
"ORBIT_CAMPOSES"
],
[
26,
20,
0,
21,
0,
"GS_PLY"
]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}
_Example_Workflows/3DGS_Multi-View_Renders_to_ply_with_Stack_Orbit_Cam_Poses_node.json
-500
View File
@@ -1,500 +0,0 @@
{
"last_node_id": 24,
"last_link_id": 29,
"nodes": [
{
"id": 10,
"type": "PreviewImage",
"pos": [
1410,
-320
],
"size": {
"0": 210,
"1": 246
},
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 4
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 9,
"type": "MaskToImage",
"pos": [
1110,
-320
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 8
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
4
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "MaskToImage"
}
},
{
"id": 19,
"type": "PreviewImage",
"pos": [
420,
-320
],
"size": {
"0": 210,
"1": 246
},
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 22
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 7,
"type": "VHS_LoadImages",
"pos": [
113,
54
],
"size": [
243.60000610351562,
194
],
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
22,
26
],
"shape": 3,
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": [
2
],
"shape": 3,
"slot_index": 1
},
{
"name": "INT",
"type": "INT",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "VHS_LoadImages"
},
"widgets_values": {
"directory": "Images_enhanced",
"image_load_cap": 0,
"skip_first_images": 0,
"select_every_nth": 1,
"choose folder to upload": "image",
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"frame_load_cap": 0,
"skip_first_images": 0,
"filename": "Images_enhanced",
"type": "input",
"format": "folder",
"select_every_nth": 1
}
}
}
},
{
"id": 8,
"type": "InvertMask",
"pos": [
746,
-14
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 2
}
],
"outputs": [
{
"name": "MASK",
"type": "MASK",
"links": [
8,
27
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "InvertMask"
}
},
{
"id": 22,
"type": "[ComfyUI-3D] Stack Orbit Camera Poses",
"pos": [
696,
174
],
"size": {
"0": 315,
"1": 586
},
"flags": {},
"order": 1,
"mode": 0,
"outputs": [
{
"name": "orbit_camposes",
"type": "ORBIT_CAMPOSES",
"links": [
28
],
"shape": 3,
"slot_index": 0
},
{
"name": "orbit_radius_list",
"type": "FLOAT",
"links": null,
"shape": 6
},
{
"name": "elevation_list",
"type": "FLOAT",
"links": null,
"shape": 6
},
{
"name": "azimuth_list",
"type": "FLOAT",
"links": null,
"shape": 6
},
{
"name": "orbit_center_X_list",
"type": "FLOAT",
"links": null,
"shape": 6
},
{
"name": "orbit_center_Y_list",
"type": "FLOAT",
"links": null,
"shape": 6
},
{
"name": "orbit_center_Z_list",
"type": "FLOAT",
"links": null,
"shape": 6
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Stack Orbit Camera Poses"
},
"widgets_values": [
1.75,
1.75,
0.1,
0,
0,
0,
0,
-0.1,
3,
0,
0,
0.1,
0,
0,
0.1,
0,
0,
0.1
]
},
{
"id": 23,
"type": "[ComfyUI-3D] Save 3DGS",
"pos": [
2178,
54
],
"size": {
"0": 315,
"1": 58
},
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"link": 29
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Save 3DGS"
},
"widgets_values": [
"MeshTest\\3DGS_01.ply"
]
},
{
"id": 13,
"type": "[ComfyUI-3D] Load 3D Mesh",
"pos": [
1109,
327
],
"size": {
"0": 315,
"1": 154
},
"flags": {},
"order": 2,
"mode": 0,
"outputs": [
{
"name": "mesh",
"type": "MESH",
"links": [],
"shape": 3,
"slot_index": 0
}
],
"title": "Load 3D Mesh [Optional]",
"properties": {
"Node name for S&R": "[ComfyUI-3D] Load 3D Mesh"
},
"widgets_values": [
"Meshes\\Cammy_Uniform.obj",
false,
true,
false,
false
]
},
{
"id": 24,
"type": "[ComfyUI-3D] Gaussian Splatting",
"pos": [
1516,
54
],
"size": {
"0": 443.4000244140625,
"1": 758
},
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "reference_images",
"type": "IMAGE",
"link": 26
},
{
"name": "reference_masks",
"type": "MASK",
"link": 27
},
{
"name": "reference_orbit_camera_poses",
"type": "ORBIT_CAMPOSES",
"link": 28
},
{
"name": "points_cloud",
"type": "POINTCLOUD",
"link": null
},
{
"name": "ply_to_initialize_gaussian",
"type": "GS_PLY",
"link": null
},
{
"name": "mesh_to_initialize_gaussian",
"type": "MESH",
"link": null
}
],
"outputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"links": [
29
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Gaussian Splatting"
},
"widgets_values": [
49.1,
1000,
3,
10000,
0.2,
3,
0,
0,
0.5,
0.01,
0.05,
0.005,
0.005,
0.001,
0.00002,
0.02,
500,
5000,
3,
0.01,
100,
100000,
100,
700,
0.01,
0
]
}
],
"links": [
[
2,
7,
1,
8,
0,
"MASK"
],
[
4,
9,
0,
10,
0,
"IMAGE"
],
[
8,
8,
0,
9,
0,
"MASK"
],
[
22,
7,
0,
19,
0,
"IMAGE"
],
[
26,
7,
0,
24,
0,
"IMAGE"
],
[
27,
8,
0,
24,
1,
"MASK"
],
[
28,
22,
0,
24,
2,
"ORBIT_CAMPOSES"
],
[
29,
24,
0,
23,
0,
"GS_PLY"
]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}
_Example_Workflows/Bake_Multi-View-Renders_into_UV_Texture.json
-486
View File
@@ -1,486 +0,0 @@
{
"last_node_id": 16,
"last_link_id": 12,
"nodes": [
{
"id": 10,
"type": "PreviewImage",
"pos": [
1410,
-320
],
"size": {
"0": 210,
"1": 246
},
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 4
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 8,
"type": "InvertMask",
"pos": [
746,
-14
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 2
}
],
"outputs": [
{
"name": "MASK",
"type": "MASK",
"links": [
7,
8
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "InvertMask"
}
},
{
"id": 9,
"type": "MaskToImage",
"pos": [
1110,
-320
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 8
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
4
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "MaskToImage"
}
},
{
"id": 14,
"type": "[ComfyUI-3D] Save 3D Mesh",
"pos": [
2132,
54
],
"size": {
"0": 315,
"1": 58
},
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "mesh",
"type": "MESH",
"link": 10
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Save 3D Mesh"
},
"widgets_values": [
"MeshTest\\Mesh_01.obj"
]
},
{
"id": 15,
"type": "PreviewImage",
"pos": [
2138,
349
],
"size": {
"0": 210,
"1": 246
},
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 11
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 4,
"type": "[ComfyUI-3D] Generate Orbit Camera Poses",
"pos": [
830,
132
],
"size": {
"0": 400,
"1": 200
},
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "reference_images",
"type": "IMAGE",
"link": 1
}
],
"outputs": [
{
"name": "orbit_camposes",
"type": "ORBIT_CAMPOSES",
"links": [
6
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Generate Orbit Camera Poses"
},
"widgets_values": [
"([0:120], 1.75, 0, 0, 360)"
]
},
{
"id": 13,
"type": "[ComfyUI-3D] Load 3D Mesh",
"pos": [
999,
412
],
"size": {
"0": 315,
"1": 154
},
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "mesh",
"type": "MESH",
"links": [
9
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Load 3D Mesh"
},
"widgets_values": [
"Meshes\\Cammy_Uniform.obj",
false,
true,
false,
true
]
},
{
"id": 7,
"type": "VHS_LoadImages",
"pos": [
212,
54
],
"size": [
243.60000610351562,
194
],
"flags": {},
"order": 1,
"mode": 0,
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
1,
5,
12
],
"shape": 3,
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": [
2
],
"shape": 3,
"slot_index": 1
},
{
"name": "INT",
"type": "INT",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "VHS_LoadImages"
},
"widgets_values": {
"directory": "Images_enhanced",
"image_load_cap": 0,
"skip_first_images": 0,
"select_every_nth": 1,
"choose folder to upload": "image",
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"frame_load_cap": 0,
"skip_first_images": 0,
"filename": "Images_enhanced",
"type": "input",
"format": "folder",
"select_every_nth": 1
}
}
}
},
{
"id": 16,
"type": "PreviewImage",
"pos": [
504,
-302
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 12
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 12,
"type": "[ComfyUI-3D] Bake Texture To Mesh",
"pos": [
1419,
55
],
"size": {
"0": 531.5999755859375,
"1": 286
},
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "reference_images",
"type": "IMAGE",
"link": 5
},
{
"name": "reference_masks",
"type": "MASK",
"link": 7
},
{
"name": "reference_orbit_camera_poses",
"type": "ORBIT_CAMPOSES",
"link": 6
},
{
"name": "mesh",
"type": "MESH",
"link": 9
}
],
"outputs": [
{
"name": "trained_mesh",
"type": "MESH",
"links": [
10
],
"shape": 3,
"slot_index": 0
},
{
"name": "baked_texture",
"type": "IMAGE",
"links": [
11
],
"shape": 3,
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Bake Texture To Mesh"
},
"widgets_values": [
49.1,
1000,
5,
0.1,
false,
0.01,
0.5,
false
]
}
],
"links": [
[
1,
7,
0,
4,
0,
"IMAGE"
],
[
2,
7,
1,
8,
0,
"MASK"
],
[
4,
9,
0,
10,
0,
"IMAGE"
],
[
5,
7,
0,
12,
0,
"IMAGE"
],
[
6,
4,
0,
12,
2,
"ORBIT_CAMPOSES"
],
[
7,
8,
0,
12,
1,
"MASK"
],
[
8,
8,
0,
9,
0,
"MASK"
],
[
9,
13,
0,
12,
3,
"MESH"
],
[
10,
12,
0,
14,
0,
"MESH"
],
[
11,
12,
1,
15,
0,
"IMAGE"
],
[
12,
7,
0,
16,
0,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}
_Example_Workflows/Images_to_3DGS_to_DMTetMesh.json
-640
View File
@@ -1,640 +0,0 @@
{
"last_node_id": 27,
"last_link_id": 36,
"nodes": [
{
"id": 10,
"type": "PreviewImage",
"pos": [
1410,
-320
],
"size": {
"0": 210,
"1": 246
},
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 4
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 9,
"type": "MaskToImage",
"pos": [
1110,
-320
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 8
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
4
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "MaskToImage"
}
},
{
"id": 19,
"type": "PreviewImage",
"pos": [
420,
-320
],
"size": {
"0": 210,
"1": 246
},
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 22
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 13,
"type": "[ComfyUI-3D] Load 3D Mesh",
"pos": [
1053,
415
],
"size": {
"0": 315,
"1": 154
},
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "mesh",
"type": "MESH",
"links": [],
"shape": 3,
"slot_index": 0
}
],
"title": "Load 3D Mesh [Optional]",
"properties": {
"Node name for S&R": "[ComfyUI-3D] Load 3D Mesh"
},
"widgets_values": [
"Meshes\\Cammy_Uniform.obj",
false,
true,
false,
false
]
},
{
"id": 7,
"type": "VHS_LoadImages",
"pos": [
113,
54
],
"size": [
243.60000610351562,
194
],
"flags": {},
"order": 1,
"mode": 0,
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
1,
22,
27
],
"shape": 3,
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": [
2
],
"shape": 3,
"slot_index": 1
},
{
"name": "INT",
"type": "INT",
"links": null,
"shape": 3
}
],
"properties": {
"Node name for S&R": "VHS_LoadImages"
},
"widgets_values": {
"directory": "Images_enhanced",
"image_load_cap": 0,
"skip_first_images": 0,
"select_every_nth": 1,
"choose folder to upload": "image",
"videopreview": {
"hidden": false,
"paused": false,
"params": {
"frame_load_cap": 0,
"skip_first_images": 0,
"filename": "Images_enhanced",
"type": "input",
"format": "folder",
"select_every_nth": 1
}
}
}
},
{
"id": 8,
"type": "InvertMask",
"pos": [
746,
-14
],
"size": {
"0": 210,
"1": 26
},
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 2
}
],
"outputs": [
{
"name": "MASK",
"type": "MASK",
"links": [
8,
28
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "InvertMask"
}
},
{
"id": 4,
"type": "[ComfyUI-3D] Generate Orbit Camera Poses",
"pos": [
830,
132
],
"size": {
"0": 400,
"1": 200
},
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "reference_images",
"type": "IMAGE",
"link": 1
}
],
"outputs": [
{
"name": "orbit_camposes",
"type": "ORBIT_CAMPOSES",
"links": [
29
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Generate Orbit Camera Poses"
},
"widgets_values": [
"([0:120], 1.75, 0, 0, 360)"
]
},
{
"id": 24,
"type": "[ComfyUI-3D] Save 3DGS",
"pos": [
2122,
-131
],
"size": {
"0": 315,
"1": 58
},
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"link": 30
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Save 3DGS"
},
"widgets_values": [
"MeshTest/3DGS_01.ply"
]
},
{
"id": 21,
"type": "[ComfyUI-3D] Deep Marching Tetrahedrons",
"pos": [
2963,
55
],
"size": {
"0": 443.4000244140625,
"1": 314
},
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "reference_points_cloud",
"type": "POINTCLOUD",
"link": 35
},
{
"name": "reference_images",
"type": "IMAGE",
"link": null
},
{
"name": "reference_masks",
"type": "MASK",
"link": null
}
],
"outputs": [
{
"name": "trained_mesh",
"type": "MESH",
"links": [
26
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Deep Marching Tetrahedrons"
},
"widgets_values": [
5000,
1,
0.1,
1,
1,
128,
0.0001,
2,
128,
5
]
},
{
"id": 22,
"type": "[ComfyUI-3D] Save 3D Mesh",
"pos": [
3534,
56
],
"size": {
"0": 353.38916015625,
"1": 61.879417419433594
},
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "mesh",
"type": "MESH",
"link": 26
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Save 3D Mesh"
},
"widgets_values": [
"MeshTest\\Cammy_Uniform_DMTet_01.obj"
]
},
{
"id": 26,
"type": "[ComfyUI-3D] Load 3DGS",
"pos": [
2107,
194
],
"size": {
"0": 315,
"1": 58
},
"flags": {},
"order": 2,
"mode": 0,
"outputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"links": [],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Load 3DGS"
},
"widgets_values": [
"Meshes/3DGS_01.ply"
]
},
{
"id": 23,
"type": "[ComfyUI-3D] Gaussian Splatting",
"pos": [
1501,
53
],
"size": {
"0": 443.4000244140625,
"1": 758
},
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "reference_images",
"type": "IMAGE",
"link": 27
},
{
"name": "reference_masks",
"type": "MASK",
"link": 28
},
{
"name": "reference_orbit_camera_poses",
"type": "ORBIT_CAMPOSES",
"link": 29
},
{
"name": "points_cloud",
"type": "POINTCLOUD",
"link": null
},
{
"name": "ply_to_initialize_gaussian",
"type": "GS_PLY",
"link": null
},
{
"name": "mesh_to_initialize_gaussian",
"type": "MESH",
"link": null
}
],
"outputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"links": [
30,
36
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Gaussian Splatting"
},
"widgets_values": [
49.1,
1000,
3,
10000,
0.2,
3,
0,
0,
0.5,
0.01,
0.05,
0.005,
0.005,
0.001,
0.00002,
0.02,
500,
5000,
3,
0.01,
100,
100000,
100,
700,
0.01,
0
]
},
{
"id": 27,
"type": "[ComfyUI-3D] Convert 3DGS To Pointcloud",
"pos": [
2555,
53
],
"size": {
"0": 241.79998779296875,
"1": 26
},
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"link": 36
}
],
"outputs": [
{
"name": "points_cloud",
"type": "POINTCLOUD",
"links": [
35
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Convert 3DGS To Pointcloud"
}
}
],
"links": [
[
1,
7,
0,
4,
0,
"IMAGE"
],
[
2,
7,
1,
8,
0,
"MASK"
],
[
4,
9,
0,
10,
0,
"IMAGE"
],
[
8,
8,
0,
9,
0,
"MASK"
],
[
22,
7,
0,
19,
0,
"IMAGE"
],
[
26,
21,
0,
22,
0,
"MESH"
],
[
27,
7,
0,
23,
0,
"IMAGE"
],
[
28,
8,
0,
23,
1,
"MASK"
],
[
29,
4,
0,
23,
2,
"ORBIT_CAMPOSES"
],
[
30,
23,
0,
24,
0,
"GS_PLY"
],
[
35,
27,
0,
21,
0,
"POINTCLOUD"
],
[
36,
23,
0,
27,
0,
"GS_PLY"
]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}
_Example_Workflows/Load_3DGS_to_DMTetMesh.json
-190
View File
@@ -1,190 +0,0 @@
{
"last_node_id": 28,
"last_link_id": 37,
"nodes": [
{
"id": 21,
"type": "[ComfyUI-3D] Deep Marching Tetrahedrons",
"pos": [
1450,
420
],
"size": {
"0": 443.4000244140625,
"1": 314
},
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "reference_points_cloud",
"type": "POINTCLOUD",
"link": 37
},
{
"name": "reference_images",
"type": "IMAGE",
"link": null
},
{
"name": "reference_masks",
"type": "MASK",
"link": null
}
],
"outputs": [
{
"name": "trained_mesh",
"type": "MESH",
"links": [
26
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Deep Marching Tetrahedrons"
},
"widgets_values": [
5000,
1,
0.1,
1,
1,
128,
0.0001,
2,
128,
5
]
},
{
"id": 22,
"type": "[ComfyUI-3D] Save 3D Mesh",
"pos": [
2020,
420
],
"size": {
"0": 353.38916015625,
"1": 61.879417419433594
},
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "mesh",
"type": "MESH",
"link": 26
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Save 3D Mesh"
},
"widgets_values": [
"MeshTest\\picacho.obj"
]
},
{
"id": 28,
"type": "[ComfyUI-3D] Convert 3DGS To Pointcloud",
"pos": [
1035,
422
],
"size": {
"0": 241.79998779296875,
"1": 26
},
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"link": 36
}
],
"outputs": [
{
"name": "points_cloud",
"type": "POINTCLOUD",
"links": [
37
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Convert 3DGS To Pointcloud"
}
},
{
"id": 26,
"type": "[ComfyUI-3D] Load 3DGS",
"pos": [
592,
423
],
"size": {
"0": 315,
"1": 58
},
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "gs_ply",
"type": "GS_PLY",
"links": [
36
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Load 3DGS"
},
"widgets_values": [
"Meshes/3DGS_TGS.ply"
]
}
],
"links": [
[
26,
21,
0,
22,
0,
"MESH"
],
[
36,
26,
0,
28,
0,
"GS_PLY"
],
[
37,
28,
0,
21,
0,
"POINTCLOUD"
]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}
_Example_Workflows/Multi-View-Images_to_3DGS_to_3DMesh(DMTet and DiffRast).json
+1544
View File
File diff suppressed because it is too large Load Diff
_Example_Workflows/Triplane_Gaussian_Transformers_to_3DGS.json
+1167 -180
View File
File diff suppressed because it is too large Load Diff
_Example_Workflows/Using_Stack_Orbit_Camera_Poses_to_conditioning_StableZero123.json
+2 -2
View File
@@ -402,7 +402,7 @@
},
{
"id": 41,
"type": "[ComfyUI-3D] Stack Orbit Camera Poses",
"type": "[Comfy3D] Stack Orbit Camera Poses",
"pos": [
-10,
540
@@ -465,7 +465,7 @@
}
],
"properties": {
"Node name for S&R": "[ComfyUI-3D] Stack Orbit Camera Poses"
"Node name for S&R": "[Comfy3D] Stack Orbit Camera Poses"
},
"widgets_values": [
0,
__init__.py
+24 -2
View File
@@ -1,5 +1,15 @@
import os
import sys
import folder_paths as comfy_paths
ROOT_PATH = os.path.join(comfy_paths.get_folder_paths("custom_nodes")[0], "ComfyUI-3D-Pack")
sys.path.append(ROOT_PATH)
import shutil
import __main__
import importlib
import inspect
from .webserver.server import server
NODE_CLASS_MAPPINGS = {}
NODE_DISPLAY_NAME_MAPPINGS = {}
@@ -10,10 +20,22 @@ for name, cls in inspect.getmembers(module, inspect.isclass):
if cls.__module__ == module.__name__:
name = name.replace("_", " ")
node = f"[ComfyUI-3D] {name}"
node = f"[Comfy3D] {name}"
disp = f"{name}"
NODE_CLASS_MAPPINGS[node] = cls
NODE_DISPLAY_NAME_MAPPINGS[node] = disp
__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']
WEB_DIRECTORY = "./web"
__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS', 'WEB_DIRECTORY']
# # Cleanup old extension folder
folder_web = os.path.join(os.path.dirname(os.path.realpath(__main__.__file__)), "web")
extensions_folder = os.path.join(folder_web, 'extensions', 'ComfyUI-3D-Pack')
def cleanup():
if os.path.exists(extensions_folder):
shutil.rmtree(extensions_folder)
print('\033[34mComfy3D: \033[92mRemoved old extension folder\033[0m')
cleanup()
algorithms/_Test.py
+5 -4
View File
@@ -1,8 +1,9 @@
import os
import numpy as np
import torch
grid_res = 128
p = os.path.join(os.path.dirname(__file__), f'../data/tets/{grid_res}_tets.npz')
print(p)
vector3_tensor = torch.tensor([[1, 2, 3], [3, 2, 1], [2, 3, 1]]) # shape (N, 3)
target_axis = (2, 0, 1) # or [2, 0, 1]
vector3_tensor[:, [0, 1, 2]] = vector3_tensor[:, target_axis]
print(vector3_tensor)
algorithms/diff_mesh_renderer.py
+1 -1
View File
@@ -37,7 +37,7 @@ def trunc_rev_sigmoid(x, eps=1e-6):
def make_divisible(x, m=8):
return int(math.ceil(x / m) * m)
class Renderer(nn.Module):
class DiffRastRenderer(nn.Module):
def __init__(self, mesh, force_cuda_rast):
super().__init__()
algorithms/diff_texturing.py
+5 -10
View File
@@ -8,8 +8,9 @@ from pytorch_msssim import SSIM, MS_SSIM
import comfy.utils
from .diff_mesh_renderer import Renderer
from .diff_mesh_renderer import DiffRastRenderer
from ..shared_utils.camera_utils import orbit_camera, OrbitCamera
from ..shared_utils.image_utils import prepare_torch_img
class DiffTextureBaker:
@@ -17,7 +18,7 @@ class DiffTextureBaker:
self.device = torch.device("cuda")
# prepare main components for optimization
self.renderer = Renderer(mesh, force_cuda_rasterize).to(self.device)
self.renderer = DiffRastRenderer(mesh, force_cuda_rasterize).to(self.device)
self.optimizer = torch.optim.Adam(self.renderer.get_params(texture_learning_rate, train_mesh_geometry, geometry_learning_rate))
#self.ssim_loss = SSIM(data_range=1, size_average=True, channel=3)
@@ -28,12 +29,6 @@ class DiffTextureBaker:
self.batch_size = batch_size
def prepare_img(self, img):
# (H, W, C) -> (C, H, W)
img_new = img.permute(2, 0, 1).unsqueeze(0).to(self.device)
img_new = F.interpolate(img_new, (self.ref_size_H, self.ref_size_W), mode="bilinear", align_corners=False).contiguous()
return img_new
def prepare_training(self, reference_images, reference_masks, reference_orbit_camera_poses, reference_orbit_camera_fovy):
self.ref_imgs_num = len(reference_images)
@@ -49,8 +44,8 @@ class DiffTextureBaker:
ref_imgs_torch_list = []
ref_masks_torch_list = []
for i in range(self.ref_imgs_num):
ref_imgs_torch_list.append(self.prepare_img(reference_images[i]))
ref_masks_torch_list.append(self.prepare_img(reference_masks[i].unsqueeze(2)))
ref_imgs_torch_list.append(prepare_torch_img(reference_images[i], self.ref_size_H, self.ref_size_W, self.device))
ref_masks_torch_list.append(prepare_torch_img(reference_masks[i].unsqueeze(2), self.ref_size_H, self.ref_size_W, self.device))
self.ref_imgs_torch = torch.cat(ref_imgs_torch_list, dim=0)
self.ref_masks_torch = torch.cat(ref_masks_torch_list, dim=0)
algorithms/dmtet.py
+1 -1
View File
@@ -229,7 +229,7 @@ class DMTetMesh:
def get_mesh(self):
verts, faces = self.pred_verts_faces()
mesh = Mesh(v=verts, f=faces)
mesh = Mesh(v=verts.detach(), f=faces.detach(), device=self.device)
mesh.auto_normal()
mesh.auto_uv()
return mesh
algorithms/main_3DGS.py
+52 -24
View File
@@ -8,8 +8,9 @@ from pytorch_msssim import SSIM, MS_SSIM
import comfy.utils
from .main_3DGS_renderer import Renderer
from .main_3DGS_renderer import GaussianSplattingRenderer
from ..shared_utils.camera_utils import orbit_camera, OrbitCamera, MiniCam, calculate_fovX, get_projection_matrix
from ..shared_utils.image_utils import prepare_torch_img
class GSParams:
def __init__(self,
@@ -72,6 +73,46 @@ class GSParams:
# other gaussian params
self.sh_degree = sh_degree
class GaussianSplattingCameraController:
def __init__(self, renderer, cam_size_W, cam_size_H, reference_orbit_camera_fovy, invert_bg_prob=1.0, static_bg=None, device='cuda'):
self.device = torch.device(device)
self.renderer = renderer
self.cam = OrbitCamera(cam_size_W, cam_size_H, fovy=reference_orbit_camera_fovy)
self.projection_matrix = get_projection_matrix(self.cam.near, self.cam.far, self.cam.fovx, self.cam.fovy).transpose(0, 1).cuda()
self.invert_bg_prob = invert_bg_prob
self.black_bg = torch.tensor([0, 0, 0], dtype=torch.float32, device=self.device)
self.white_bg = torch.tensor([1, 1, 1], dtype=torch.float32, device=self.device)
self.static_bg = None if static_bg is None else torch.tensor(static_bg, dtype=torch.float32, device=self.device)
def render_at_pose(self, cam_pose):
radius, elevation, azimuth, center_X, center_Y, center_Z = cam_pose
orbit_target = np.array([center_X, center_Y, center_Z], dtype=np.float32)
render_pose = orbit_camera(elevation, azimuth, radius, target=orbit_target)
render_cam = MiniCam(render_pose, self.cam.W, self.cam.H, self.cam.fovy, self.cam.fovx, self.cam.near, self.cam.far, self.projection_matrix)
if self.static_bg is None:
bg_color = self.white_bg if np.random.rand() > self.invert_bg_prob else self.black_bg
else:
bg_color = self.static_bg
return self.renderer.render(render_cam, bg_color=bg_color)
def render_all_pose(self, all_cam_poses):
all_rendered_images, all_rendered_masks = [], []
for cam_pose in all_cam_poses:
out = self.render_at_pose(cam_pose)
image = out["image"] # [3, H, W] in [0, 1]
mask = out["alpha"] # [1, H, W] in [0, 1]
all_rendered_images.append(image)
all_rendered_masks.append(mask)
# [Number of Poses, 3, H, W], [Number of Poses, 1, H, W] both in [0, 1]
return torch.stack(all_rendered_images, dim=0), torch.stack(all_rendered_masks, dim=0)
class GaussianSplatting:
@@ -82,8 +123,7 @@ class GaussianSplatting:
if gs_params is None:
gs_params = GSParams()
self.renderer = Renderer(sh_degree=gs_params.sh_degree)
self.gaussain_scale_factor = 1
self.renderer = GaussianSplattingRenderer(sh_degree=gs_params.sh_degree)
self.renderer.initialize(init_input, num_pts=gs_params.num_pts)
# setup training
@@ -95,31 +135,26 @@ class GaussianSplatting:
self.ms_ssim_loss = MS_SSIM(data_range=1, size_average=True, channel=3)
self.gs_params = gs_params
def prepare_img(self, img):
img_new = img.permute(2, 0, 1).unsqueeze(0).to(self.device)
img_new = F.interpolate(img_new, (self.ref_size_H, self.ref_size_W), mode="bilinear", align_corners=False).contiguous()
return img_new
def prepare_training(self, reference_images, reference_masks, reference_orbit_camera_poses, reference_orbit_camera_fovy):
self.ref_imgs_num = len(reference_images)
self.all_ref_cam_poses = reference_orbit_camera_poses
self.ref_cam_fovy = reference_orbit_camera_fovy
self.ref_size_H = reference_images[0].shape[0]
self.ref_size_W = reference_images[0].shape[1]
# default camera settings
self.cam = OrbitCamera(self.ref_size_W, self.ref_size_H, fovy=reference_orbit_camera_fovy)
self.projection_matrix = get_projection_matrix(self.cam.near, self.cam.far, self.cam.fovx, self.cam.fovy).transpose(0, 1).cuda()
self.cam_controller = GaussianSplattingCameraController(
self.renderer, self.ref_size_W, self.ref_size_H, reference_orbit_camera_fovy, self.gs_params.invert_bg_prob, None, self.device
)
self.all_ref_cam_poses = reference_orbit_camera_poses
# prepare reference images and masks
ref_imgs_torch_list = []
ref_masks_torch_list = []
for i in range(self.ref_imgs_num):
ref_imgs_torch_list.append(self.prepare_img(reference_images[i]))
ref_masks_torch_list.append(self.prepare_img(reference_masks[i].unsqueeze(2)))
ref_imgs_torch_list.append(prepare_torch_img(reference_images[i], self.ref_size_H, self.ref_size_W, self.device))
ref_masks_torch_list.append(prepare_torch_img(reference_masks[i].unsqueeze(2), self.ref_size_H, self.ref_size_W, self.device))
self.ref_imgs_torch = torch.cat(ref_imgs_torch_list, dim=0)
self.ref_masks_torch = torch.cat(ref_masks_torch_list, dim=0)
@@ -158,14 +193,7 @@ class GaussianSplatting:
i = random.randint(0, ref_imgs_num_minus_1)
radius, elevation, azimuth, center_X, center_Y, center_Z = self.all_ref_cam_poses[i]
orbit_target = np.array([center_X, center_Y, center_Z], dtype=np.float32)
ref_pose = orbit_camera(elevation, azimuth, radius, target=orbit_target)
ref_cam = MiniCam(ref_pose, self.ref_size_W, self.ref_size_H, self.cam.fovy, self.cam.fovx, self.cam.near, self.cam.far, self.projection_matrix)
bg_color = torch.tensor([1, 1, 1] if np.random.rand() > self.gs_params.invert_bg_prob else [0, 0, 0], dtype=torch.float32, device=self.device)
out = self.renderer.render(ref_cam, bg_color=bg_color)
out = self.cam_controller.render_at_pose(self.all_ref_cam_poses[i])
image = out["image"] # [3, H, W] in [0, 1]
mask = out["alpha"] # [1, H, W] in [0, 1]
algorithms/main_3DGS_renderer.py
+8 -31
View File
@@ -10,8 +10,6 @@ import torch
from torch import nn
import torch.nn.functional as F
from ..mesh_processer.mesh import PointCloud
pytorch3d_capable = True
try:
import pytorch3d
@@ -30,8 +28,8 @@ from diff_gaussian_rasterization import (
from simple_knn._C import distCUDA2
from ..shared_utils.sh_utils import eval_sh, SH2RGB, RGB2SH
from ..mesh_processer.mesh import Mesh
from ..mesh_processer.mesh_utils import read_gs_ply
from ..mesh_processer.mesh import Mesh, PointCloud
from ..mesh_processer.mesh_utils import construct_list_of_gs_attributes, write_gs_ply, read_gs_ply
def inverse_sigmoid(x):
return torch.log(x/(1-x))
@@ -285,7 +283,7 @@ class GaussianModel:
self._features_dc = torch.empty(0) # diffuse color
self._features_rest = torch.empty(0) # spherical harmonic coefficients
self._scaling = torch.empty(0)
self._rotation = torch.empty(0)
self._rotation = torch.empty(0) # (w, x, y, z)
self._opacity = torch.empty(0)
self.max_radii2D = torch.empty(0)
self.xyz_gradient_accum = torch.empty(0)
@@ -505,20 +503,6 @@ class GaussianModel:
opacities_new = inverse_sigmoid(torch.min(self.get_opacity, torch.ones_like(self.get_opacity)*0.01))
optimizable_tensors = self.replace_tensor_to_optimizer(opacities_new, "opacity")
self._opacity = optimizable_tensors["opacity"]
def construct_list_of_attributes(self):
l = ['x', 'y', 'z', 'nx', 'ny', 'nz']
# All channels except the 3 DC
for i in range(self._features_dc.shape[1]*self._features_dc.shape[2]):
l.append('f_dc_{}'.format(i))
for i in range(self._features_rest.shape[1]*self._features_rest.shape[2]):
l.append('f_rest_{}'.format(i))
l.append('opacity')
for i in range(self._scaling.shape[1]):
l.append('scale_{}'.format(i))
for i in range(self._rotation.shape[1]):
l.append('rot_{}'.format(i))
return l
def get_points_cloud(self):
xyz = self._xyz.detach().cpu().numpy()
@@ -526,9 +510,6 @@ class GaussianModel:
shs = self.get_features.detach().cpu().numpy()
pcd = PointCloud(points=xyz, colors=SH2RGB(shs), normals=normals)
return pcd
def get_mesh(self):
xyz = self._xyz.detach().cpu().numpy()
def to_ply(self):
xyz = self._xyz.detach().cpu().numpy()
@@ -538,24 +519,20 @@ class GaussianModel:
opacities = self._opacity.detach().cpu().numpy()
scale = self._scaling.detach().cpu().numpy()
rotation = self._rotation.detach().cpu().numpy()
dtype_full = [(attribute, 'f4') for attribute in self.construct_list_of_attributes()]
elements = np.empty(xyz.shape[0], dtype=dtype_full)
attributes = np.concatenate((xyz, normals, f_dc, f_rest, opacities, scale, rotation), axis=1)
elements[:] = list(map(tuple, attributes))
el = PlyElement.describe(elements, 'vertex')
return PlyData([el])
return write_gs_ply(xyz, normals, f_dc, f_rest, opacities, scale, rotation, construct_list_of_gs_attributes(self._features_dc, self._features_rest, self._scaling, self._rotation))
def create_from_ply(self, plydata):
xyz, features_dc, features_extra, opacities, scales, rots = read_gs_ply(plydata)
self._xyz = nn.Parameter(torch.tensor(xyz, dtype=torch.float, device="cuda").requires_grad_(True))
self.init_xyz = self._xyz.clone().detach()
self._features_dc = nn.Parameter(torch.tensor(features_dc, dtype=torch.float, device="cuda").transpose(1, 2).contiguous().requires_grad_(True))
self._features_rest = nn.Parameter(torch.tensor(features_extra, dtype=torch.float, device="cuda").transpose(1, 2).contiguous().requires_grad_(True))
self._opacity = nn.Parameter(torch.tensor(opacities, dtype=torch.float, device="cuda").requires_grad_(True))
self._scaling = nn.Parameter(torch.tensor(scales, dtype=torch.float, device="cuda").requires_grad_(True))
self._rotation = nn.Parameter(torch.tensor(rots, dtype=torch.float, device="cuda").requires_grad_(True))
self.max_radii2D = torch.zeros((self.get_xyz.shape[0]), device="cuda")
self.active_sh_degree = self.max_sh_degree
@@ -842,7 +819,7 @@ class GaussianModel:
torch.cuda.empty_cache()
class Renderer:
class GaussianSplattingRenderer:
def __init__(self, sh_degree=3, white_background=True, radius=1):
self.sh_degree = sh_degree
mesh_processer/mesh.py
+11 -4
View File
@@ -6,7 +6,8 @@ import trimesh
import numpy as np
from ..shared_utils.sh_utils import SH2RGB
from shared_utils.sh_utils import SH2RGB
from shared_utils.image_utils import prepare_torch_img
def dot(x, y):
return torch.sum(x * y, -1, keepdim=True)
@@ -285,8 +286,7 @@ class Mesh:
mesh.albedo = torch.tensor(texture, dtype=torch.float32, device=device)
print(f"[load_trimesh] load texture: {texture.shape}")
else:
texture = np.ones((1024, 1024, 3), dtype=np.float32) * np.array([0.5, 0.5, 0.5])
mesh.albedo = torch.tensor(texture, dtype=torch.float32, device=device)
mesh.set_new_albedo(1024, 1024)
print(f"[load_trimesh] failed to load texture.")
vertices = _mesh.vertices
@@ -330,6 +330,13 @@ class Mesh:
)
return mesh
def set_new_albedo(self, res_H, res_W):
if self.albedo is None:
texture = np.ones((res_H, res_W, 3), dtype=np.float32) * np.array([0.5, 0.5, 0.5])
self.albedo = torch.tensor(texture, dtype=torch.float32, device=self.device)
else:
self.albedo = prepare_torch_img(self.albedo, res_H, res_W, self.device).squeeze(0).permute(1, 2, 0).contiguous() # (1, 3, H, W) -> (H, W, 3)
# aabb
def aabb(self):
@@ -614,7 +621,7 @@ class Mesh:
fp.write(f"Ka 1 1 1 \n")
fp.write(f"Kd 1 1 1 \n")
fp.write(f"Ks 0 0 0 \n")
fp.write(f"Tr 1 \n")
#fp.write(f"Tr 1 \n") # will cause the mesh materials in three.js completely transparent
fp.write(f"illum 1 \n")
fp.write(f"Ns 0 \n")
if self.albedo is not None:
mesh_processer/mesh_utils.py
+122 -5
View File
@@ -1,9 +1,14 @@
import torch
import numpy as np
from kornia.geometry.conversions import (
quaternion_to_axis_angle,
axis_angle_to_quaternion,
)
import pymeshlab as pml
from plyfile import PlyData, PlyElement
from .mesh import PointCloud
from ..shared_utils.sh_utils import SH2RGB, RGB2SH
from shared_utils.sh_utils import SH2RGB, RGB2SH
def _base_face_areas(face_vertices_0, face_vertices_1, face_vertices_2):
"""Base function to compute the face areas."""
@@ -301,6 +306,34 @@ def clean_mesh(
return verts, faces
def construct_list_of_gs_attributes(features_dc, features_rest, scaling, rotation):
l = ['x', 'y', 'z', 'nx', 'ny', 'nz']
for i in range(features_dc.shape[1]*features_dc.shape[2]):
l.append('f_dc_{}'.format(i))
for i in range(features_rest.shape[1]*features_rest.shape[2]):
l.append('f_rest_{}'.format(i))
l.append('opacity')
for i in range(scaling.shape[1]):
l.append('scale_{}'.format(i))
for i in range(rotation.shape[1]):
l.append('rot_{}'.format(i))
return l
def calculate_max_sh_degree_from_gs_ply(plydata):
extra_f_names = [p.name for p in plydata.elements[0].properties if p.name.startswith("f_rest_")]
#assert len(extra_f_names)!=3*(max_sh_degree + 1) ** 2 - 3:
max_sh_degree = int(((len(extra_f_names) + 3) / 3) ** 0.5 - 1)
return max_sh_degree, extra_f_names
def write_gs_ply(xyz, normals, f_dc, f_rest, opacities, scale, rotation, list_of_attributes):
dtype_full = [(attribute, 'f4') for attribute in list_of_attributes]
elements = np.empty(xyz.shape[0], dtype=dtype_full)
attributes = np.concatenate((xyz, normals, f_dc, f_rest, opacities, scale, rotation), axis=1)
elements[:] = list(map(tuple, attributes))
el = PlyElement.describe(elements, 'vertex')
return PlyData([el])
def read_gs_ply(plydata):
xyz = np.stack((np.asarray(plydata.elements[0]["x"]),
np.asarray(plydata.elements[0]["y"]),
@@ -312,9 +345,8 @@ def read_gs_ply(plydata):
features_dc[:, 1, 0] = np.asarray(plydata.elements[0]["f_dc_1"])
features_dc[:, 2, 0] = np.asarray(plydata.elements[0]["f_dc_2"])
extra_f_names = [p.name for p in plydata.elements[0].properties if p.name.startswith("f_rest_")]
#assert len(extra_f_names)!=3*(max_sh_degree + 1) ** 2 - 3:
max_sh_degree = int(((len(extra_f_names) + 3) / 3) ** 0.5 - 1)
max_sh_degree, extra_f_names = calculate_max_sh_degree_from_gs_ply(plydata)
features_extra = np.zeros((xyz.shape[0], len(extra_f_names)))
for idx, attr_name in enumerate(extra_f_names):
features_extra[:, idx] = np.asarray(plydata.elements[0][attr_name])
@@ -335,9 +367,94 @@ def read_gs_ply(plydata):
def ply_to_points_cloud(plydata):
xyz, features_dc, features_extra, opacities, scales, rots = read_gs_ply(plydata)
features_dc = np.transpose(features_dc, (0, 2, 1)) # equivalent of torch.transpose(features_dc, 1, 2)
features_extra = np.transpose(features_extra, (0, 2, 1))
shs = np.concatenate((features_dc, features_extra), axis=1)
normals = np.zeros_like(xyz)
pcd = PointCloud(points=xyz, colors=SH2RGB(shs), normals=normals)
return pcd
return pcd
def get_target_axis_and_scale(axis_string, scale_value=1.0):
"""
Coordinate system inverts when:
1. Any of the axis inverts
2. Two of the axises switch
If coordinate system inverts twice in a row then it will not be inverted
"""
axis_names = ["x", "y", "z"]
target_axis, target_scale, coordinate_invert_count = [], [], 0
axis_switch_count = 0
for i in range(len(axis_names)):
s = axis_string[i]
if s[0] == "-":
target_scale.append(-scale_value)
coordinate_invert_count += 1
else:
target_scale.append(scale_value)
new_axis_i = axis_names.index(s[1])
if new_axis_i != i:
axis_switch_count += 1
target_axis.append(new_axis_i)
if axis_switch_count == 2:
coordinate_invert_count += 1
return target_axis, target_scale, coordinate_invert_count
def switch_vector_axis(vector3_tensor, target_axis):
"""
Example:
vector3_tensor = torch.tensor([[1, 2, 3], [3, 2, 1], [2, 3, 1]]) # shape (N, 3)
target_axis = (2, 0, 1) # or [2, 0, 1]
vector3_tensor[:, [0, 1, 2]] = vector3_tensor[:, target_axis]
# Result: tensor([[3, 1, 2], [1, 3, 2], [1, 2, 3]])
"""
vector3_tensor[:, [0, 1, 2]] = vector3_tensor[:, target_axis]
return vector3_tensor
def switch_ply_axis_and_scale(plydata, target_axis, target_scale, coordinate_invert_count):
"""
Args:
target_axis (array): shape (3)
target_scale (array): shape (3)
"""
xyz, features_dc, features_extra, opacities, scales, rots = read_gs_ply(plydata)
normals = np.zeros_like(xyz)
features_dc_2d = features_dc.reshape(features_dc.shape[0], features_dc.shape[1]*features_dc.shape[2])
features_extra_2d = features_extra.reshape(features_extra.shape[0], features_extra.shape[1]*features_extra.shape[2])
target_scale = torch.tensor(target_scale).float().cuda()
xyz = switch_vector_axis(torch.tensor(xyz).float().cuda() * target_scale, target_axis).detach().cpu().numpy()
scales = switch_vector_axis(torch.tensor(scales).float().cuda(), target_axis).detach().cpu().numpy()
# change rotation representation from quaternion (w, x, y, z) to axis angle vector (x, y, z) to make swich axis easier
rots_axis_angle = quaternion_to_axis_angle(torch.tensor(rots).float().cuda())
rots_axis_angle = switch_vector_axis(rots_axis_angle * target_scale, target_axis)
"""
Since axis–angle vector is composed of axis (unit vector/direction) and clockwise radians angle (vector magnitude),
so in order to invert the sign of angle when coordinate system inverts, we also need to invert the direction of axis–angle vector
"""
if coordinate_invert_count % 2 != 0:
rots_axis_angle = -rots_axis_angle
rots = axis_angle_to_quaternion(rots_axis_angle).detach().cpu().numpy()
return write_gs_ply(xyz, normals, features_dc_2d, features_extra_2d, opacities, scales, rots, construct_list_of_gs_attributes(features_dc, features_extra, scales, rots))
def switch_mesh_axis_and_scale(mesh, target_axis, target_scale):
"""
Args:
target_axis (array): shape (3)
target_scale (array): shape (3)
"""
target_scale = torch.tensor(target_scale).float().cuda()
mesh.v = switch_vector_axis(mesh.v * target_scale, target_axis)
mesh.vn = switch_vector_axis(mesh.vn * target_scale, target_axis)
return mesh
nodes.py
+272 -72
View File
@@ -4,7 +4,6 @@ import re
import math
import copy
from enum import Enum
import subprocess
import folder_paths as comfy_paths
import torch
@@ -13,10 +12,17 @@ import numpy as np
from plyfile import PlyData
from .shared_utils.common_utils import cstr
from .shared_utils.common_utils import cstr, parse_save_filename
from .mesh_processer.mesh import Mesh
from .mesh_processer.mesh_utils import ply_to_points_cloud
from .algorithms.main_3DGS import GaussianSplatting, GSParams
from .mesh_processer.mesh_utils import (
ply_to_points_cloud,
get_target_axis_and_scale,
switch_ply_axis_and_scale,
switch_mesh_axis_and_scale,
calculate_max_sh_degree_from_gs_ply,
)
from .algorithms.main_3DGS import GaussianSplatting, GaussianSplattingCameraController, GSParams
from .algorithms.main_3DGS_renderer import GaussianSplattingRenderer
from .algorithms.diff_texturing import DiffTextureBaker
from .algorithms.dmtet import DMTetMesh
from .algorithms.triplane_gaussian_transformers import TGS
@@ -25,11 +31,10 @@ from .tgs.data import CustomImageOrbitDataset
from .tgs.utils.misc import todevice, get_device
ROOT_PATH = os.path.join(comfy_paths.get_folder_paths("custom_nodes")[0], "ComfyUI-3D-Pack")
sys.path.append(ROOT_PATH)
MANIFEST = {
"name": "ComfyUI-3D-Pack",
"version": (0,0,1),
"version": (0,0,2),
"author": "Mr. For Example",
"project": "https://github.com/MrForExample/ComfyUI-3D-Pack",
"description": "An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, etc.)",
@@ -50,6 +55,72 @@ ELEVATION_MAX = 90.0
AZIMUTH_MIN = -180.0
AZIMUTH_MAX = 180.0
class Preview_3DGS:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"gs_file_path": ("STRING", {"default": '', "multiline": False}),
},
}
OUTPUT_NODE = True
RETURN_TYPES = ()
FUNCTION = "preview_gs"
CATEGORY = "Comfy3D/Visualize"
def preview_gs(self, gs_file_path):
gs_folder_path, filename = os.path.split(gs_file_path)
if not os.path.isabs(gs_file_path):
gs_file_path = os.path.join(comfy_paths.output_directory, gs_folder_path)
if not filename.lower().endswith(SUPPORTED_3DGS_EXTENSIONS):
cstr(f"[{self.__class__.__name__}] File name {filename} does not end with supported 3DGS file extensions: {SUPPORTED_3DGS_EXTENSIONS}").error.print()
gs_file_path = ""
previews = [
{
"filepath": gs_file_path,
}
]
return {"ui": {"previews": previews}, "result": ()}
class Preview_3DMesh:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"mesh_file_path": ("STRING", {"default": '', "multiline": False}),
},
}
OUTPUT_NODE = True
RETURN_TYPES = ()
FUNCTION = "preview_mesh"
CATEGORY = "Comfy3D/Visualize"
def preview_mesh(self, mesh_file_path):
mesh_folder_path, filename = os.path.split(mesh_file_path)
if not os.path.isabs(mesh_file_path):
mesh_file_path = os.path.join(comfy_paths.output_directory, mesh_folder_path)
if not filename.lower().endswith(SUPPORTED_3D_EXTENSIONS):
cstr(f"[{self.__class__.__name__}] File name {filename} does not end with supported 3D file extensions: {SUPPORTED_3D_EXTENSIONS}").error.print()
mesh_file_path = ""
previews = [
{
"filepath": mesh_file_path,
}
]
return {"ui": {"previews": previews}, "result": ()}
class Load_3D_Mesh:
@classmethod
@@ -71,7 +142,7 @@ class Load_3D_Mesh:
"mesh",
)
FUNCTION = "load_mesh"
CATEGORY = "ComfyUI3D/Import|Export"
CATEGORY = "Comfy3D/Import|Export"
def load_mesh(self, mesh_file_path, resize, renormal, retex, optimizable):
mesh = None
@@ -96,7 +167,7 @@ class Load_3DGS:
def INPUT_TYPES(cls):
return {
"required": {
"gs_ply_file_path": ("STRING", {"default": '', "multiline": False}),
"gs_file_path": ("STRING", {"default": '', "multiline": False}),
},
}
@@ -107,22 +178,22 @@ class Load_3DGS:
"gs_ply",
)
FUNCTION = "load_gs"
CATEGORY = "ComfyUI3D/Import|Export"
CATEGORY = "Comfy3D/Import|Export"
def load_gs(self, gs_ply_file_path):
def load_gs(self, gs_file_path):
gs_ply = None
if not os.path.isabs(gs_ply_file_path):
gs_ply_file_path = os.path.join(comfy_paths.input_directory, gs_ply_file_path)
if not os.path.isabs(gs_file_path):
gs_file_path = os.path.join(comfy_paths.input_directory, gs_file_path)
if os.path.exists(gs_ply_file_path):
folder, filename = os.path.split(gs_ply_file_path)
if os.path.exists(gs_file_path):
folder, filename = os.path.split(gs_file_path)
if filename.lower().endswith(SUPPORTED_3DGS_EXTENSIONS):
gs_ply = PlyData.read(gs_ply_file_path)
gs_ply = PlyData.read(gs_file_path)
else:
cstr(f"[{self.__class__.__name__}] File name {filename} does not end with supported 3DGS file extensions: {SUPPORTED_3DGS_EXTENSIONS}").error.print()
else:
cstr(f"[{self.__class__.__name__}] File {gs_ply_file_path} does not exist").error.print()
cstr(f"[{self.__class__.__name__}] File {gs_file_path} does not exist").error.print()
return (gs_ply, )
class Save_3D_Mesh:
@@ -132,32 +203,27 @@ class Save_3D_Mesh:
return {
"required": {
"mesh": ("MESH",),
"mesh_save_path": ("STRING", {"default": 'Mesh_[time(%Y-%m-%d)].obj', "multiline": False}),
"save_path": ("STRING", {"default": 'Mesh_%Y-%m-%d-%M-%S-%f.obj', "multiline": False}),
},
}
OUTPUT_NODE = True
RETURN_TYPES = ()
RETURN_TYPES = (
"STRING",
)
RETURN_NAMES = (
"save_path",
)
FUNCTION = "save_mesh"
CATEGORY = "ComfyUI3D/Import|Export"
CATEGORY = "Comfy3D/Import|Export"
def save_mesh(self, mesh, mesh_save_path):
def save_mesh(self, mesh, save_path):
save_path = parse_save_filename(save_path, comfy_paths.output_directory, SUPPORTED_3D_EXTENSIONS, self.__class__.__name__)
mesh_folder_path, filename = os.path.split(mesh_save_path)
if not os.path.isabs(mesh_save_path):
mesh_folder_path = os.path.join(comfy_paths.output_directory, mesh_folder_path)
mesh_save_path = os.path.join(mesh_folder_path, filename)
os.makedirs(mesh_folder_path, exist_ok=True)
if filename.lower().endswith(SUPPORTED_3D_EXTENSIONS):
mesh.write(mesh_save_path)
cstr(f"[{self.__class__.__name__}] saved model to {mesh_save_path}").msg.print()
else:
cstr(f"File name {filename} does not end with supported 3D file extensions: {SUPPORTED_3D_EXTENSIONS}").error.print()
return ()
if save_path is not None:
mesh.write(save_path)
return (save_path, )
class Save_3DGS:
@@ -166,31 +232,92 @@ class Save_3DGS:
return {
"required": {
"gs_ply": ("GS_PLY",),
"save_path": ("STRING", {"default": '3DGS_[time(%Y-%m-%d)].ply', "multiline": False}),
"save_path": ("STRING", {"default": '3DGS_%Y-%m-%d-%M-%S-%f.ply', "multiline": False}),
},
}
OUTPUT_NODE = True
RETURN_TYPES = ()
FUNCTION = "save_3DGS"
CATEGORY = "ComfyUI3D/Import|Export"
RETURN_TYPES = (
"STRING",
)
RETURN_NAMES = (
"save_path",
)
FUNCTION = "save_gs"
CATEGORY = "Comfy3D/Import|Export"
def save_3DGS(self, gs_ply, save_path):
def save_gs(self, gs_ply, save_path):
folder_path, filename = os.path.split(save_path)
save_path = parse_save_filename(save_path, comfy_paths.output_directory, SUPPORTED_3DGS_EXTENSIONS, self.__class__.__name__)
if filename.lower().endswith(SUPPORTED_3DGS_EXTENSIONS):
if not os.path.isabs(save_path):
folder_path = os.path.join(comfy_paths.output_directory, folder_path)
save_path = os.path.join(folder_path, filename)
os.makedirs(folder_path, exist_ok=True)
if save_path is not None:
gs_ply.write(save_path)
else:
cstr(f"[{self.__class__.__name__}] File name {filename} does not end with supported 3DGS file extensions: {SUPPORTED_3DGS_EXTENSIONS}").error.print()
return ()
return (save_path, )
class Switch_3DGS_Axis:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"gs_ply": ("GS_PLY",),
"axis_x_to": (["+x", "-x", "+y", "-y", "+z", "-z"],),
"axis_y_to": (["+y", "-y", "+z", "-z", "+x", "-x"],),
"axis_z_to": (["+z", "-z", "+x", "-x", "+y", "-y"],),
},
}
RETURN_TYPES = (
"GS_PLY",
)
RETURN_NAMES = (
"switched_gs_ply",
)
FUNCTION = "switch_axis_and_scale"
CATEGORY = "Comfy3D/Preprocessor"
def switch_axis_and_scale(self, gs_ply, axis_x_to, axis_y_to, axis_z_to):
switched_gs_ply = None
if axis_x_to[1] != axis_y_to[1] and axis_x_to[1] != axis_z_to[1] and axis_y_to[1] != axis_z_to[1]:
target_axis, target_scale, coordinate_invert_count = get_target_axis_and_scale([axis_x_to, axis_y_to, axis_z_to])
switched_gs_ply = switch_ply_axis_and_scale(gs_ply, target_axis, target_scale, coordinate_invert_count)
else:
cstr(f"[{self.__class__.__name__}] axis_x_to: {axis_x_to}, axis_y_to: {axis_y_to}, axis_z_to: {axis_z_to} have to be on separated axis").error.print()
return (switched_gs_ply, )
class Switch_Mesh_Axis:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"mesh": ("MESH",),
"axis_x_to": (["+x", "-x", "+y", "-y", "+z", "-z"],),
"axis_y_to": (["+y", "-y", "+z", "-z", "+x", "-x"],),
"axis_z_to": (["+z", "-z", "+x", "-x", "+y", "-y"],),
},
}
RETURN_TYPES = (
"MESH",
)
RETURN_NAMES = (
"switched_mesh",
)
FUNCTION = "switch_axis_and_scale"
CATEGORY = "Comfy3D/Preprocessor"
def switch_axis_and_scale(self, mesh, axis_x_to, axis_y_to, axis_z_to):
switched_mesh = None
if axis_x_to[1] != axis_y_to[1] and axis_x_to[1] != axis_z_to[1] and axis_y_to[1] != axis_z_to[1]:
target_axis, target_scale, coordinate_invert_count = get_target_axis_and_scale([axis_x_to, axis_y_to, axis_z_to])
switched_mesh = switch_mesh_axis_and_scale(mesh, target_axis, target_scale)
else:
cstr(f"[{self.__class__.__name__}] axis_x_to: {axis_x_to}, axis_y_to: {axis_y_to}, axis_z_to: {axis_z_to} have to be on separated axis").error.print()
return (switched_mesh, )
class Convert_3DGS_To_Pointcloud:
@@ -209,7 +336,7 @@ class Convert_3DGS_To_Pointcloud:
"points_cloud",
)
FUNCTION = "convert_gs_ply"
CATEGORY = "ComfyUI3D/Preprocessor"
CATEGORY = "Comfy3D/Preprocessor"
def convert_gs_ply(self, gs_ply):
@@ -234,7 +361,7 @@ class Convert_Mesh_To_Pointcloud:
"points_cloud",
)
FUNCTION = "convert_mesh"
CATEGORY = "ComfyUI3D/Preprocessor"
CATEGORY = "Comfy3D/Preprocessor"
def convert_mesh(self, mesh):
@@ -298,7 +425,7 @@ class Stack_Orbit_Camera_Poses:
)
FUNCTION = "get_camposes"
CATEGORY = "ComfyUI3D/Preprocessor"
CATEGORY = "Comfy3D/Preprocessor"
class Pose_Config(Enum):
STOP_LARGER_STEP_POS = 0
@@ -451,7 +578,7 @@ class Generate_Orbit_Camera_Poses:
"orbit_camposes",
)
FUNCTION = "get_camposes"
CATEGORY = "ComfyUI3D/Preprocessor"
CATEGORY = "Comfy3D/Preprocessor"
class Slice_Camposes:
def __init__(self, start_reference_image_index, end_reference_image_index, camposes_start_to_end):
@@ -506,6 +633,64 @@ class Generate_Orbit_Camera_Poses:
cstr(f"[{self.__class__.__name__}] Last end_reference_image_index: {end_reference_image_index} plus 1 must equal to current start_reference_image_index: {start_reference_image_index}").error.print()
return (orbit_camposes, )
class Gaussian_Splatting_Orbit_Renderer:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"gs_ply": ("GS_PLY",),
"render_image_width": ("INT", {"default": 1024, "min": 128, "max": 8192}),
"render_image_height": ("INT", {"default": 1024, "min": 128, "max": 8192}),
"render_orbit_camera_poses": ("ORBIT_CAMPOSES",), # (orbit radius, elevation, azimuth, orbit center X, orbit center Y, orbit center Z)
"render_orbit_camera_fovy": ("FLOAT", {"default": 49.1, "min": 0.0, "max": 180.0, "step": 0.1}),
"render_background_color_r": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
"render_background_color_g": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
"render_background_color_b": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.001}),
}
}
RETURN_TYPES = (
"IMAGE",
"MASK",
)
RETURN_NAMES = (
"rendered_gs_images", # [Number of Poses, H, W, 3]
"rendered_gs_masks", # [Number of Poses, H, W, 1]
)
FUNCTION = "render_gs"
CATEGORY = "Comfy3D/Preprocessor"
def render_gs(
self,
gs_ply,
render_image_width,
render_image_height,
render_orbit_camera_poses,
render_orbit_camera_fovy,
render_background_color_r,
render_background_color_g,
render_background_color_b,
):
sh_degree, _ = calculate_max_sh_degree_from_gs_ply(gs_ply)
renderer = GaussianSplattingRenderer(sh_degree=sh_degree)
renderer.initialize(gs_ply)
cam_controller = GaussianSplattingCameraController(
renderer,
render_image_width,
render_image_height,
render_orbit_camera_fovy,
static_bg=[render_background_color_r, render_background_color_g, render_background_color_b]
)
all_rendered_images, all_rendered_masks = cam_controller.render_all_pose(render_orbit_camera_poses)
all_rendered_images = all_rendered_images.permute(0, 2, 3, 1) # [N, 3, H, W] -> [N, H, W, 3]
all_rendered_masks = all_rendered_masks.permute(0, 2, 3, 1).squeeze(3) # [N, 1, H, W] -> [N, H, W]
return (all_rendered_images, all_rendered_masks)
class Gaussian_Splatting:
@@ -545,7 +730,7 @@ class Gaussian_Splatting:
},
"optional": {
"points_cloud": ("POINTCLOUD",),
"points_cloud_to_initialize_gaussian": ("POINTCLOUD",),
"ply_to_initialize_gaussian": ("GS_PLY",),
"mesh_to_initialize_gaussian": ("MESH",),
}
@@ -557,10 +742,10 @@ class Gaussian_Splatting:
RETURN_NAMES = (
"gs_ply",
)
FUNCTION = "run_3DGS"
CATEGORY = "ComfyUI3D/Algorithm"
FUNCTION = "run_gs"
CATEGORY = "Comfy3D/Algorithm"
def run_3DGS(
def run_gs(
self,
reference_images,
reference_masks,
@@ -591,7 +776,7 @@ class Gaussian_Splatting:
opacity_reset_interval,
densify_grad_threshold,
gaussian_sh_degree,
points_cloud=None,
points_cloud_to_initialize_gaussian=None,
ply_to_initialize_gaussian=None,
mesh_to_initialize_gaussian=None,
):
@@ -638,8 +823,8 @@ class Gaussian_Splatting:
gaussian_sh_degree)
if points_cloud is not None:
gs_init_input = points_cloud
if points_cloud_to_initialize_gaussian is not None:
gs_init_input = points_cloud_to_initialize_gaussian
elif ply_to_initialize_gaussian is not None:
gs_init_input = ply_to_initialize_gaussian
else:
@@ -672,6 +857,8 @@ class Bake_Texture_To_Mesh:
"reference_orbit_camera_poses": ("ORBIT_CAMPOSES",), # (orbit radius, elevation, azimuth, orbit center X, orbit center Y, orbit center Z)
"reference_orbit_camera_fovy": ("FLOAT", {"default": 49.1, "min": 0.0, "max": 180.0, "step": 0.1}),
"mesh": ("MESH",),
"mesh_albedo_width": ("INT", {"default": 1024, "min": 128, "max": 8192}),
"mesh_albedo_height": ("INT", {"default": 1024, "min": 128, "max": 8192}),
"training_iterations": ("INT", {"default": 1000, "min": 1, "max": 100000}),
"batch_size": ("INT", {"default": 3, "min": 1, "max": 0xffffffffffffffff}),
"texture_learning_rate": ("FLOAT", {"default": 0.1, "min": 0.00001, "step": 0.00001}),
@@ -691,10 +878,25 @@ class Bake_Texture_To_Mesh:
"baked_texture", # [1, H, W, 3]
)
FUNCTION = "bake_texture"
CATEGORY = "ComfyUI3D/Algorithm"
CATEGORY = "Comfy3D/Algorithm"
def bake_texture(self, reference_images, reference_masks, reference_orbit_camera_poses, reference_orbit_camera_fovy, mesh,
training_iterations, batch_size, texture_learning_rate, train_mesh_geometry, geometry_learning_rate, ms_ssim_loss_weight, force_cuda_rasterize):
def bake_texture(self,
reference_images,
reference_masks,
reference_orbit_camera_poses,
reference_orbit_camera_fovy,
mesh,
mesh_albedo_width,
mesh_albedo_height,
training_iterations,
batch_size,
texture_learning_rate,
train_mesh_geometry,
geometry_learning_rate,
ms_ssim_loss_weight,
force_cuda_rasterize):
mesh.set_new_albedo(mesh_albedo_width, mesh_albedo_height)
trained_mesh = None
baked_texture = None
@@ -758,7 +960,7 @@ class Deep_Marching_Tetrahedrons:
"trained_mesh",
)
FUNCTION = "run_dmtet"
CATEGORY = "ComfyUI3D/Algorithm"
CATEGORY = "Comfy3D/Algorithm"
def run_dmtet(self, training_iterations, points_cloud_fitting_weight, mesh_smoothing_weight, chamfer_faces_sample_scale, mesh_scale, grid_resolution, geometry_learning_rate,
positional_encoding_multires, mlp_internal_dims, mlp_hidden_layer_num,
@@ -805,7 +1007,7 @@ class Load_Triplane_Gaussian_Transformers:
"tgs_model",
)
FUNCTION = "load_TGS"
CATEGORY = "ComfyUI3D/Import|Export"
CATEGORY = "Comfy3D/Import|Export"
def load_TGS(self):
@@ -831,7 +1033,7 @@ class Load_Triplane_Gaussian_Transformers:
return (tgs_model, )
class Run_Triplane_Gaussian_Transformers:
class Triplane_Gaussian_Transformers:
@classmethod
def INPUT_TYPES(cls):
@@ -847,14 +1049,12 @@ class Run_Triplane_Gaussian_Transformers:
OUTPUT_NODE = True
RETURN_TYPES = (
"GS_PLY",
"FLOAT"
)
RETURN_NAMES = (
"gs_ply",
"gaussian_sh_degree"
)
FUNCTION = "run_TGS"
CATEGORY = "ComfyUI3D/Algorithm"
CATEGORY = "Comfy3D/Algorithm"
def run_TGS(self, reference_image, reference_mask, tgs_model, cam_dist):
config_path = os.path.join(ROOT_PATH, TGS_CONFIG_PATH)
@@ -889,5 +1089,5 @@ class Run_Triplane_Gaussian_Transformers:
)
"""
return (gs_ply[0], tgs_model.renderer.cfg.sh_degree)
return (gs_ply[0],)
shared_utils/common_utils.py
+30 -4
View File
@@ -1,5 +1,6 @@
import os
import sys
from datetime import datetime
class cstr(str):
# Modified from: WAS Node Suite
@@ -76,9 +77,9 @@ class cstr(str):
print(self, **kwargs)
#! MESSAGE TEMPLATES
cstr.color.add_code("msg", f"{cstr.color.BLUE}[ComfyUI-3D] {cstr.color.END}")
cstr.color.add_code("warning", f"{cstr.color.LIGHTYELLOW}[ComfyUI-3D] [WARNING] {cstr.color.END}")
cstr.color.add_code("error", f"{cstr.color.RED}[ComfyUI-3D] [ERROR] {cstr.color.END}")
cstr.color.add_code("msg", f"{cstr.color.BLUE}[Comfy3D] {cstr.color.END}")
cstr.color.add_code("warning", f"{cstr.color.LIGHTYELLOW}[Comfy3D] [WARNING] {cstr.color.END}")
cstr.color.add_code("error", f"{cstr.color.RED}[Comfy3D] [ERROR] {cstr.color.END}")
def get_persistent_directory(folder_name):
if sys.platform == "win32":
@@ -87,4 +88,29 @@ def get_persistent_directory(folder_name):
folder = os.path.join(os.path.expanduser("~"), "." + folder_name)
os.makedirs(folder, exist_ok=True)
return folder
return folder
def parse_save_filename(save_path, output_directory, supported_extensions, class_name):
folder_path, filename = os.path.split(save_path)
filename, file_extension = os.path.splitext(filename)
if file_extension.lower() in supported_extensions:
if not os.path.isabs(save_path):
folder_path = os.path.join(output_directory, folder_path)
os.makedirs(folder_path, exist_ok=True)
# replace time date format to current time
now = datetime.now() # current date and time
all_date_format = ["%Y", "%m", "%d", "%M", "%S", "%f"]
for date_format in all_date_format:
if date_format in filename:
filename = filename.replace(date_format, now.strftime(date_format))
save_path = os.path.join(folder_path, filename) + file_extension
cstr(f"[{class_name}] Saving model to {save_path}").msg.print()
return save_path
else:
cstr(f"[{class_name}] File name {filename} does not end with supported file extensions: {supported_extensions}").error.print()
return None
shared_utils/image_utils.py
+9
View File
@@ -0,0 +1,9 @@
import torch
import torch.nn.functional as F
import numpy as np
def prepare_torch_img(img, size_H, size_W, device="cuda"):
# [H, W, C] -> [1, C, H, W]
img_new = img.permute(2, 0, 1).unsqueeze(0).to(device)
img_new = F.interpolate(img_new, (size_H, size_W), mode="bilinear", align_corners=False).contiguous()
return img_new
tgs/models/renderer.py
+3 -22
View File
@@ -15,6 +15,8 @@ from ..utils.ops import scale_tensor
from .networks import MLP
from einops import rearrange, reduce
from mesh_processer.mesh_utils import construct_list_of_gs_attributes, write_gs_ply
inverse_sigmoid = lambda x: np.log(x / (1 - x))
def getWorld2View2(R, t, translate=np.array([.0, .0, .0]), scale=1.0):
@@ -90,21 +92,6 @@ class GaussianModel(NamedTuple):
scaling: Tensor
shs: Tensor
def construct_list_of_attributes(self):
l = ['x', 'y', 'z', 'nx', 'ny', 'nz']
features_dc = self.shs[:, :1]
features_rest = self.shs[:, 1:]
for i in range(features_dc.shape[1]*features_dc.shape[2]):
l.append('f_dc_{}'.format(i))
for i in range(features_rest.shape[1]*features_rest.shape[2]):
l.append('f_rest_{}'.format(i))
l.append('opacity')
for i in range(self.scaling.shape[1]):
l.append('scale_{}'.format(i))
for i in range(self.rotation.shape[1]):
l.append('rot_{}'.format(i))
return l
def to_ply(self):
xyz = self.xyz.detach().cpu().numpy()
@@ -117,13 +104,7 @@ class GaussianModel(NamedTuple):
scale = np.log(self.scaling.detach().cpu().numpy())
rotation = self.rotation.detach().cpu().numpy()
dtype_full = [(attribute, 'f4') for attribute in self.construct_list_of_attributes()]
elements = np.empty(xyz.shape[0], dtype=dtype_full)
attributes = np.concatenate((xyz, normals, f_dc, f_rest, opacities, scale, rotation), axis=1)
elements[:] = list(map(tuple, attributes))
el = PlyElement.describe(elements, 'vertex')
return PlyData([el])
return write_gs_ply(xyz, normals, f_dc, f_rest, opacities, scale, rotation, construct_list_of_gs_attributes(features_dc, features_rest, self.scaling, self.rotation))
class GSLayer(BaseModule):
@dataclass
web/html/gsVisualizer.html
+31
View File
@@ -0,0 +1,31 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="stylesheet" href="/extensions/ComfyUI-3D-Pack/style/gsStyle.css">
<link type="text/css" rel="stylesheet" href="/extensions/ComfyUI-3D-Pack/style/progressStyle.css">
</head>
<body>
<div id="root">
<div id="progress-container">
<dialog open id="progress-dialog">
<p>
<label for="progress-indicator">Loading scene...</label>
</p>
<progress max="100" id="progress-indicator"></progress>
</dialog>
</div>
<canvas id="canvas"></canvas>
<script type="importmap">
{
"imports": {
"gsplat": "https://unpkg.com/gsplat@latest/dist/index.js"
}
}
</script>
<script id="visualizer" type="module" filepath="" crossorigin src="/extensions/ComfyUI-3D-Pack/js/gsVisualizer.js"></script>
</div>
</body>
</html>
web/html/threeVisualizer.html
+32
View File
@@ -0,0 +1,32 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0">
<link type="text/css" rel="stylesheet" href="/extensions/ComfyUI-3D-Pack/style/threeStyle.css">
<link type="text/css" rel="stylesheet" href="/extensions/ComfyUI-3D-Pack/style/progressStyle.css">
</head>
<body>
<div id="progress-container">
<dialog open id="progress-dialog">
<p>
<label for="progress-indicator">Loading scene...</label>
</p>
<progress max="100" id="progress-indicator"></progress>
</dialog>
</div>
<div id="container"></div>
<script type="importmap">
{
"imports": {
"three": "https://unpkg.com/three@latest/build/three.module.js",
"three/addons/": "https://unpkg.com/three@latest/examples/jsm/"
}
}
</script>
<script id="visualizer" type="module" filepath="" crossorigin src="/extensions/ComfyUI-3D-Pack/js/threeVisualizer.js"></script>
</body>
</html>
web/js/LittlestTokyo.glb
BIN
View File
Binary file not shown.
web/js/gsVisualizer.js
+74
View File
@@ -0,0 +1,74 @@
import * as SPLAT from 'gsplat';
import { api } from '/scripts/api.js'
const visualizer = document.getElementById("visualizer");
const canvas = document.getElementById("canvas");
const progressDialog = document.getElementById("progress-dialog");
const progressIndicator = document.getElementById("progress-indicator");
const renderer = new SPLAT.WebGLRenderer(canvas);
const scene = new SPLAT.Scene();
const camera = new SPLAT.Camera();
const controls = new SPLAT.OrbitControls(camera, canvas);
controls.orbitSpeed = 3.0;
// Handle window reseize event
const handleResize = () => {
renderer.setSize(window.innerWidth, window.innerHeight);
};
handleResize();
window.addEventListener("resize", handleResize);
var lastFilepath = "";
var needUpdate = false;
function frameUpdate() {
var filepath = visualizer.getAttribute("filepath");
if (filepath == lastFilepath){
if (needUpdate){
controls.update();
renderer.render( scene, camera );
}
requestAnimationFrame( frameUpdate );
} else {
needUpdate = false;
scene.reset();
progressDialog.open = true;
lastFilepath = filepath;
main(lastFilepath);
}
};
const onProgress = function ( progress ) {
progressIndicator.value = progress * 100;
};
async function main(filepath="") {
// Check if file name is valid
if (/^.+\.[a-zA-Z]+$/.test(filepath)){
let params = {"filepath": filepath};
const url = api.apiURL('/viewfile?' + new URLSearchParams(params));
var splat = null;
var fileExt = filepath.split('.').pop().toLowerCase();
if (fileExt == "ply"){
splat = await SPLAT.PLYLoader.LoadAsync(url, scene, onProgress);
} else if (fileExt == "splat") {
splat = await SPLAT.Loader.LoadAsync(url, scene, onProgress);
} else {
throw new Error(`File extension name has to be either .ply or .splat, got .${fileExt}`);
}
needUpdate = true;
}
progressDialog.close();
frameUpdate();
}
//main("C:/Users/reall/Softwares/ComfyUI_windows_portable/ComfyUI/output/bonsai.splat");
main();
web/js/threeVisualizer.js
+147
View File
@@ -0,0 +1,147 @@
import * as THREE from 'three';
import { api } from '/scripts/api.js'
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';
import { RoomEnvironment } from 'three/addons/environments/RoomEnvironment.js';
import { MTLLoader } from 'three/addons/loaders/MTLLoader.js';
import { OBJLoader } from 'three/addons/loaders/OBJLoader.js';
import { GLTFLoader } from 'three/addons/loaders/GLTFLoader.js';
import { DRACOLoader } from 'three/addons/loaders/DRACOLoader.js';
const visualizer = document.getElementById("visualizer");
const container = document.getElementById( 'container' );
const progressDialog = document.getElementById("progress-dialog");
const progressIndicator = document.getElementById("progress-indicator");
const renderer = new THREE.WebGLRenderer( { antialias: true } );
renderer.setPixelRatio( window.devicePixelRatio );
renderer.setSize( window.innerWidth, window.innerHeight );
container.appendChild( renderer.domElement );
const pmremGenerator = new THREE.PMREMGenerator( renderer );
// scene
const scene = new THREE.Scene();
scene.background = new THREE.Color( 0x000000 );
scene.environment = pmremGenerator.fromScene( new RoomEnvironment( renderer ), 0.04 ).texture;
const ambientLight = new THREE.AmbientLight( 0xffffff );
const camera = new THREE.PerspectiveCamera( 40, window.innerWidth / window.innerHeight, 1, 100 );
camera.position.set( 5, 2, 8 );
const pointLight = new THREE.PointLight( 0xffffff, 15 );
camera.add( pointLight );
const controls = new OrbitControls( camera, renderer.domElement );
controls.target.set( 0, 0.5, 0 );
controls.update();
controls.enablePan = true;
controls.enableDamping = true;
// Handle window reseize event
window.onresize = function () {
camera.aspect = window.innerWidth / window.innerHeight;
camera.updateProjectionMatrix();
renderer.setSize( window.innerWidth, window.innerHeight );
};
var lastFilepath = "";
var needUpdate = false;
function frameUpdate() {
var filepath = visualizer.getAttribute("filepath");
if (filepath == lastFilepath){
if (needUpdate){
controls.update();
renderer.render( scene, camera );
}
requestAnimationFrame( frameUpdate );
} else {
needUpdate = false;
scene.clear();
progressDialog.open = true;
lastFilepath = filepath;
main(lastFilepath);
}
}
const onProgress = function ( xhr ) {
if ( xhr.lengthComputable ) {
progressIndicator.value = xhr.loaded / xhr.total * 100;
}
};
const onError = function ( e ) {
console.error( e );
};
async function main(filepath="") {
// Check if file name is valid
if (/^.+\.[a-zA-Z]+$/.test(filepath)){
let params = {"filepath": filepath};
const url = api.apiURL('/viewfile?' + new URLSearchParams(params));
var filepathSplit = filepath.split('.');
var fileExt = filepathSplit.pop().toLowerCase();
var filepathNoExt = filepathSplit.join(".");
if (fileExt == "obj"){
const loader = new OBJLoader();
var mtlFolderpath = filepath.substring(0, Math.max(filepath.lastIndexOf("/"), filepath.lastIndexOf("\\"))) + "/";
var mtlFilepath = filepathNoExt.replace(/^.*[\\\/]/, '') + ".mtl";
const mtlLoader = new MTLLoader();
mtlLoader.setPath(api.apiURL('/viewfile?' + new URLSearchParams({"filepath": mtlFolderpath})));
mtlLoader.load( mtlFilepath, function ( mtl ) {
mtl.preload();
loader.setMaterials( mtl );
}, onProgress, onError );
loader.load( url, function ( obj ) {
obj.scale.setScalar( 5 );
scene.add( obj );
}, onProgress, onError );
} else if (fileExt == "glb") {
const dracoLoader = new DRACOLoader();
dracoLoader.setDecoderPath( 'https://unpkg.com/three@latest/examples/jsm/libs/draco/gltf/' );
const loader = new GLTFLoader();
loader.setDRACOLoader( dracoLoader );
loader.load( url, function ( gltf ) {
const model = gltf.scene;
model.position.set( 1, 1, 0 );
model.scale.set( 0.01, 0.01, 0.01 );
scene.add( model );
}, onProgress, onError );
} else if (fileExt == "ply") {
} else {
throw new Error(`File extension name has to be either .ply or .splat, got .${fileExt}`);
}
needUpdate = true;
}
scene.add( ambientLight );
scene.add( camera );
progressDialog.close();
frameUpdate();
}
//main("C:/Users/reall/Softwares/ComfyUI_windows_portable/ComfyUI/output/MeshTest/Mesh_01.obj");
main();
web/style/gsStyle.css
+12
View File
@@ -0,0 +1,12 @@
body,
html {
margin: 0;
padding: 0;
overflow: hidden;
background-color: #000;
}
canvas {
width: 100vw;
height: 100vh;
}
web/style/progressStyle.css
+36
View File
@@ -0,0 +1,36 @@
dialog {
width: 100%;
text-align: center;
max-width: 20em;
color: white;
background-color: #000;
border: none;
position: relative;
transform: translate(-50%, -50%);
}
#progress-container {
position: absolute;
top: 50%;
left: 50%;
}
progress {
width: 100%;
height: 1em;
border: none;
background-color: #fff;
color: #eee;
}
progress::-webkit-progress-bar {
background-color: #333;
}
progress::-webkit-progress-value {
background-color: #eee;
}
progress::-moz-progress-bar {
background-color: #eee;
}
web/style/threeStyle.css
+91
View File
@@ -0,0 +1,91 @@
body {
margin: 0;
background-color: #000;
color: #fff;
font-family: Monospace;
font-size: 13px;
line-height: 24px;
overscroll-behavior: none;
}
a {
color: #ff0;
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
button {
cursor: pointer;
text-transform: uppercase;
}
#info {
position: absolute;
top: 0px;
width: 100%;
padding: 10px;
box-sizing: border-box;
text-align: center;
-moz-user-select: none;
-webkit-user-select: none;
-ms-user-select: none;
user-select: none;
pointer-events: none;
z-index: 1; /* TODO Solve this in HTML */
}
a, button, input, select {
pointer-events: auto;
}
.lil-gui {
z-index: 2 !important; /* TODO Solve this in HTML */
}
@media all and ( max-width: 640px ) {
.lil-gui.root {
right: auto;
top: auto;
max-height: 50%;
max-width: 80%;
bottom: 0;
left: 0;
}
}
#overlay {
position: absolute;
font-size: 16px;
z-index: 2;
top: 0;
left: 0;
width: 100%;
height: 100%;
display: flex;
align-items: center;
justify-content: center;
flex-direction: column;
background: rgba(0,0,0,0.7);
}
#overlay button {
background: transparent;
border: 0;
border: 1px solid rgb(255, 255, 255);
border-radius: 4px;
color: #ffffff;
padding: 12px 18px;
text-transform: uppercase;
cursor: pointer;
}
#notSupported {
width: 50%;
margin: auto;
background-color: #f00;
margin-top: 20px;
padding: 10px;
}
web/visualization.js
+165
View File
@@ -0,0 +1,165 @@
import { app } from "/scripts/app.js"
class Visualizer {
constructor(node, container, visualSrc) {
this.node = node
this.iframe = document.createElement('iframe')
Object.assign(this.iframe, {
scrolling: "no",
overflow: "hidden",
})
this.iframe.src = "/extensions/ComfyUI-3D-Pack/html/" + visualSrc + ".html"
container.appendChild(this.iframe)
}
updateVisual(filepath) {
const iframeDocument = this.iframe.contentWindow.document
const previewScript = iframeDocument.getElementById('visualizer')
previewScript.setAttribute("filepath", filepath)
}
remove() {
this.container.remove()
}
}
function createVisualizer(node, inputName, typeName, inputData, app) {
node.name = inputName
const widget = {
type: typeName,
name: "preview3d",
callback: () => {},
draw : function(ctx, node, widgetWidth, widgetY, widgetHeight) {
const margin = 10
const top_offset = 5
const visible = app.canvas.ds.scale > 0.5 && this.type === typeName
const w = widgetWidth - margin * 4
const clientRectBound = ctx.canvas.getBoundingClientRect()
const transform = new DOMMatrix()
.scaleSelf(
clientRectBound.width / ctx.canvas.width,
clientRectBound.height / ctx.canvas.height
)
.multiplySelf(ctx.getTransform())
.translateSelf(margin, margin + widgetY)
Object.assign(this.visualizer.style, {
left: `${transform.a * margin + transform.e}px`,
top: `${transform.d + transform.f + top_offset}px`,
width: `${(w * transform.a)}px`,
height: `${(w * transform.d - widgetHeight - (margin * 15) * transform.d)}px`,
position: "absolute",
overflow: "hidden",
zIndex: app.graph._nodes.indexOf(node),
})
Object.assign(this.visualizer.children[0].style, {
transformOrigin: "50% 50%",
width: '100%',
height: '100%',
border: '0 none',
})
this.visualizer.hidden = !visible
},
}
const container = document.createElement('div')
container.id = `Comfy3D_${inputName}`
node.visualizer = new Visualizer(node, container, typeName)
widget.visualizer = container
widget.parent = node
document.body.appendChild(widget.visualizer)
node.addCustomWidget(widget)
node.updateParameters = (params) => {
node.visualizer.updateVisual(params.filepath)
}
// Events for drawing backgound
node.onDrawBackground = function (ctx) {
if (!this.flags.collapsed) {
node.visualizer.iframe.hidden = false
} else {
node.visualizer.iframe.hidden = true
}
}
// Make sure visualization iframe is always inside the node when resize the node
node.onResize = function () {
let [w, h] = this.size
if (w <= 600) w = 600
if (h <= 500) h = 500
if (w > 600) {
h = w - 100
}
this.size = [w, h]
}
// Events for remove nodes
node.onRemoved = () => {
for (let w in node.widgets) {
if (node.widgets[w].visualizer) {
node.widgets[w].visualizer.remove()
}
}
}
return {
widget: widget,
}
}
function registerVisualizer(nodeType, nodeData, nodeClassName, typeName){
if (nodeData.name == nodeClassName) {
console.log("[3D Visualizer] Registering node: " + nodeData.name)
const onNodeCreated = nodeType.prototype.onNodeCreated
nodeType.prototype.onNodeCreated = async function() {
const r = onNodeCreated
? onNodeCreated.apply(this, arguments)
: undefined
let Preview3DNode = app.graph._nodes.filter(
(wi) => wi.type == nodeClassName
)
let nodeName = `Preview3DNode_${Preview3DNode.length}`
console.log(`[Comfy3D] Create: ${nodeName}`)
const result = await createVisualizer.apply(this, [this, nodeName, typeName, {}, app])
this.setSize([600, 500])
return r
}
nodeType.prototype.onExecuted = async function(message) {
if (message?.previews) {
this.updateParameters(message.previews[0])
}
}
}
}
app.registerExtension({
name: "Mr.ForExample.Visualizer.GS",
async init (app) {
},
async beforeRegisterNodeDef(nodeType, nodeData, app) {
registerVisualizer(nodeType, nodeData, "[Comfy3D] Preview 3DGS", "gsVisualizer")
registerVisualizer(nodeType, nodeData, "[Comfy3D] Preview 3DMesh", "threeVisualizer")
},
})
webserver/server.py
+34
View File
@@ -0,0 +1,34 @@
import server
import folder_paths as comfy_paths
import os
import time
import subprocess
from shared_utils.common_utils import cstr
web = server.web
SUPPORTED_VIEW_EXTENSIONS = (
'.png',
'.jpg',
'.jpeg ',
'.mtl',
'.obj',
'.glb',
'.ply',
'.splat'
)
@server.PromptServer.instance.routes.get("/extensions/ComfyUI-3D-Pack/html/viewfile")
async def view_file(request):
query = request.rel_url.query
# Security check to see if query client is local
if request.remote == "127.0.0.1" and "filepath" in query:
filepath = query["filepath"]
cstr(f"[Server Query view_file] Get file {filepath}").msg.print()
if filepath.lower().endswith(SUPPORTED_VIEW_EXTENSIONS) and os.path.exists(filepath):
return web.FileResponse(filepath)
return web.Response(status=404)